India Sitecore: November 2013

27 November 2013

Search with Sitecore - Article - 5 - Auto Complete For Search

Dear all friends, this is fifth and final article in this series and following are the list of articles for reference :

1) Search with Sitecore - Article-1 - Introduction
2) Search with Sitecore - Article - 2 - Configure Advance Database Crawler and More
3) Search with Sitecore - Article - 3 - Crawler to Crawl Media File like PDF, Document
4) Search with Sitecore - Article - 4 - Create Search API
5) Search with Sitecore - Article - 5 - Auto Complete For Search

Every implementer of search functionality has a fantasy where it can provide an true auto complete feature for there search text box or similar to AKA Google search. Basic class or inspiration i get out from the following stack overflow answer
http://stackoverflow.com/questions/120180/how-to-do-query-auto-completion-suggestions-in-lucene

So let's analyze our problem, "we need to provide legitimate search keywords during typing in our search text box. and it should be pretty fast so that end user has true experience. Following are three pin point issues from above problem statement".

a) We need to find the source for repository.
b) We need to build the repository once we found that source and it should be easily accessible and organised.
c) Query that repository and show with text box as user types.

Let's look closer to the problem statement, I said legitimate because it should be come from the content we already have in our Sitecore content repository. Content could be come from routine content like blog, news, content pages, media file names, meta data, item name anything but within the Sitecore Content universe. Off-course it should be also valid content means it should be not that content which is restricted by the business.
So my first problem is from where to find such content which is easily accessible rather than going through whole Sitecore Content.

Answer: We have such content already indexed in our existing Indexes like 'SeacrhItems', 'Documents' and 'Products'(Please read previous articles). They are perfect places to look as they already have indexed all keyword necessary and still compliant with the business rules as they are made keeping that point of view. So answer is i have that repository with me now. But it is not organised/refined and there are three different indexes to look for data.

Next issue is I need to build such repository.
Answer: Best way is to create new Lucene Index by reading existing index which will store all list of keywords. Reading all index and creating new index out of it I need full Lucene .NET library and probably few other third party libraries. This task can be done through Sitecore Scheduler task also, but I prefer to do it through Console Application which can be scheduled as Task Scheduler in deployment servers. We will name this index as 'AutoUpdate'.

The main class which is responsible to read index and build index is follow rest all classes are mere helper and initiator. So let's get into Code.

namespace AutoUpdateIndexer
{
/// <summary>
/// Search term auto-completer, works for single terms (so use on the last term of the query).
/// Returns more popular terms first.
/// </summary>
public class SearchAutoComplete
{
public static ILog log = log4net.LogManager.GetLogger(typeof(SearchAutoComplete));
public int MaxResults { get; set; }

private static readonly Lucene.Net.Util.Version kLuceneVersion = Lucene.Net.Util.Version.LUCENE_CURRENT;

private static readonly String kGrammedWordsField = "words";

private static readonly String kSourceWordField = "sourceWord";

private static readonly String kCountField = "count";

private static readonly String[] kEnglishStopWords = {
"a", "an", "and", "are", "as", "at", "be", "but", "by",
"for", "i", "if", "in", "into", "is",
"no", "not", "of", "on", "or", "s", "such",
"t", "that", "the", "their", "then", "there", "these",
"they", "this", "to", "was", "will", "with"
};

public bool IsFirstTime { get; set; }

private readonly Directory m_directory;

private IndexReader m_reader;

private IndexSearcher m_searcher;

public SearchAutoComplete(string autoCompleteDir) :
this(FSDirectory.Open(new System.IO.DirectoryInfo(autoCompleteDir)))
{
}

public SearchAutoComplete(Directory autoCompleteDir, int maxResults = 8)
{
this.m_directory = autoCompleteDir;
MaxResults = maxResults;

ReplaceSearcher();
}

/// <summary>
/// Find terms matching the given partial word that appear in the highest number of documents.</summary>
/// <param name="term">A word or part of a word</param>
/// <returns>A list of suggested completions</returns>
public string[] SuggestTermsFor(string term)
{
if (m_searcher == null)
return new string[] { };

// get the top terms for query
Query query = new TermQuery(new Term(kGrammedWordsField, term.ToLower()));

Sort sort = new Sort(new SortField(kCountField, SortField.INT));

TopDocs docs = m_searcher.Search(query, null, MaxResults, sort);
string[] suggestions = docs.ScoreDocs.Select(doc =>
m_reader.Document(doc.doc).Get(kSourceWordField)).ToArray();

return suggestions;
}

/// <summary>
/// Open the index in the given directory and create a new index of word frequency for the
/// given index.</summary>
/// <param name="sourceDirectory">Directory containing the index to count words in.</param>
/// <param name="fieldToAutocomplete">The field in the index that should be analyzed.</param>
public void BuildAutoCompleteIndex(Directory sourceDirectory, Directory TargetDirectory, bool verbose)
{
// build a dictionary (from the spell package)
using (IndexReader sourceReader = IndexReader.Open(sourceDirectory, true))
{

string[] fieldNames = sourceReader.GetFieldNames(IndexReader.FieldOption.ALL).ToArray();
foreach (string fieldToAutocomplete in fieldNames)
{
if (fieldToAutocomplete.Contains("__display name") || fieldToAutocomplete.Contains("_name") || !fieldToAutocomplete.Contains('_') || !fieldToAutocomplete.Contains("date") || !fieldToAutocomplete.Contains("threshold"))
{
LuceneDictionary dict = new LuceneDictionary(sourceReader, fieldToAutocomplete);

// code from
// org.apache.lucene.search.spell.SpellChecker.indexDictionary(
// Dictionary)
//IndexWriter.Unlock(m_directory);

// use a custom analyzer so we can do EdgeNGramFiltering
AutoCompleteAnalyzer analyzer = new AutoCompleteAnalyzer();
using (var writer = new IndexWriter(TargetDirectory, analyzer, IsFirstTime, IndexWriter.MaxFieldLength.UNLIMITED))
{
writer.SetMergeFactor(300);
writer.SetMaxBufferedDocs(150);

// go through every word, storing the original word (incl. n-grams)
// and the number of times it occurs
System.Collections.IEnumerator ie = dict.GetWordsIterator();
double num;
Guid guid;
foreach (string word in dict)
{
if (word.Length < UtilitySettings.AllowedMinimumWordLengthToBeIndexed)
continue; // too short we bail but
if (word.Length > UtilitySettings.AllowedMaxWordLengthToBeIndexed)
continue; //too long also we bail out

if (!word.Contains('<') && !word.Contains('>') && !word.Contains('/') && !word.Contains('\\') && !isNotFile(word) && !word.Contains('@') && !word.Contains('&') && !double.TryParse(word, out num) && !Guid.TryParse(word, out guid))
{
// ok index the word
// use the number of documents this word appears in
int freq = sourceReader.DocFreq(new Term(fieldToAutocomplete, word));
if (verbose)
{
log.Info(string.Format("Frequency {0} of this word {1}", freq, word));
}
var doc = MakeDocument(fieldToAutocomplete, word, freq);
writer.AddDocument(doc);
}
}
writer.Optimize();
}
}
}
}

// re-open our reader
//ReplaceSearcher();
}

private static Document MakeDocument(String fieldToAutocomplete, string word, int frequency)
{
var doc = new Document();
doc.Add(new Field(kSourceWordField, word, Field.Store.YES, Field.Index.NOT_ANALYZED)); // orig term
doc.Add(new Field(kGrammedWordsField, word, Field.Store.YES, Field.Index.ANALYZED)); // grammed
doc.Add(new Field(kCountField, frequency.ToString(), Field.Store.NO, Field.Index.NOT_ANALYZED)); // count
return doc;
}

private void ReplaceSearcher()
{
if (IndexReader.IndexExists(m_directory))
{
if (m_reader == null)
m_reader = IndexReader.Open(m_directory, true);
else
m_reader.Reopen();

m_searcher = new IndexSearcher(m_reader);
}
else
{
m_searcher = null;
}
}

private bool isNotFile(string word)
{
bool result = false;
if (word == null || word.Length < 1)
return result;

if (word.Contains(".png") || word.Contains(".jpeg") || word.Contains(".jpg") || word.Contains(".gif") || word.Contains(".tif") || word.Contains(".ico") || word.Contains(".bmp") || word.Contains(".aspx") || word.Contains("&amp"))
{
result = true;
}
return result;
}
}

public class AutoCompleteAnalyzer : Analyzer
{
private static readonly Lucene.Net.Util.Version kLuceneVersion = Lucene.Net.Util.Version.LUCENE_24;
private static readonly String[] kEnglishStopWords = {
"a", "an", "and", "are", "as", "at", "be", "but", "by",
"for", "i", "if", "in", "into", "is",
"no", "not", "of", "on", "or", "s", "such",
"t", "that", "the", "their", "then", "there", "these",
"they", "this", "to", "was", "will", "with"
};

public override TokenStream TokenStream(string fieldName, System.IO.TextReader reader)
{
TokenStream result = new StandardTokenizer(kLuceneVersion, reader);
result = new StandardFilter(result);
result = new LowerCaseFilter(result);
result = new ASCIIFoldingFilter(result);
result = new StopFilter(false, result, StopFilter.MakeStopSet(kEnglishStopWords));
result = new EdgeNGramTokenFilter(
result, Lucene.Net.Analysis.NGram.EdgeNGramTokenFilter.Side.FRONT, 1, 20);
return result;
}
}
}

The main function is 'BuildAutoCompleteIndex' function which reads index through the source directory, loop through each valid field and write found terms into target directory after due filtration which is our new Index. This class also have function which can be used query the target directory for the terms.
Full project is shared on GitHub.

Figure 5.1 Console Screen of Search

Next Issue is to query this Index in such a way that it is feasible and give true experience to the user. First we will need to create a new class which will provide us functions to query above new index.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Lucene.Net.Store;
using Lucene.Net.Index;
using Lucene.Net.Search;
using SpellChecker.Net.Search.Spell;
using Lucene.Net.Analysis;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.Analysis.NGram;
using Lucene.Net.Documents;
using Portal.AppServices.Utilities;

namespace Portal.EShopServices.Search.Searcher
{
/// <summary>
/// Search term auto-completer, works for single terms (so use on the last term of the query).
/// Returns more popular terms first.
/// </summary>
///
public class SearchAutoComplete
{
public int MaxResults { get; set; }

private static readonly String kGrammedWordsField = "words";

private static readonly String kSourceWordField = "sourceWord";

private static readonly String kCountField = "count";

private readonly Directory m_directory;

private IndexReader m_reader;

private IndexSearcher m_searcher;

public SearchAutoComplete(string autoCompleteDir) :
this(FSDirectory.Open(new System.IO.DirectoryInfo(autoCompleteDir)))
{
}

public SearchAutoComplete(Directory autoCompleteDir, int maxResults = 100)
{
this.m_directory = autoCompleteDir;
MaxResults = maxResults;
ReplaceSearcher();
}

/// <summary>
/// Find terms matching the given partial word that appear in the highest number of documents.</summary>
/// <param name="term">A word or part of a word</param>
/// <returns>A list of suggested completions</returns>
public string[] SuggestTermsFor(string term)
{
if (m_searcher == null)
return new string[] { };

// get the top terms for query
Query query = new TermQuery(new Term(kGrammedWordsField, term.ToLower()));
Sort sort = new Sort(new SortField(kCountField, SortField.INT));
TopDocs docs = m_searcher.Search(query, null, MaxResults, sort);
string[] suggestions = docs.ScoreDocs.Select(doc =>
m_reader.Document(doc.doc).Get(kSourceWordField)).ToArray();

return suggestions;
}

private void ReplaceSearcher()
{
if (IndexReader.IndexExists(m_directory))
{
if (m_reader == null)
m_reader = IndexReader.Open(m_directory, true);
else
m_reader.Reopen();

m_searcher = new IndexSearcher(m_reader);
}
else
{
m_searcher = null;
}
}
}
}

The main function in this class is 'SuggestTermsFor' which take string as argument to search and revert back with possible suggestions in string Array.The class is initialized with the argument of directory path where AutoUpdate Index is located.
To call 'SuggestTermsFor' this function we need and ASPX .NET page which will work as resource which can answer our Ajax queries initiated from the jQuery getJson method. The HTML/designer side of the ASPX page is almost blank with just following declaration

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Ajax.aspx.cs" Inherits="Portal.Web.Services.Ajax" %>

The code page of the above page has following class:

using Portal.AppServices.Utilities;
using Portal.EShopServices.Search.Searcher;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;

namespace Portal.Web.Services
{
public partial class Ajax : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
if (Request.QueryString["term"] != null)
{
string searchTerm = ValidationHelper.ValidateToString(Request.QueryString["term"], "");
SearchTerm(searchTerm);
}
}

protected void SearchTerm(string searchTerm)
{
string response = string.Empty;
string[] suggestions = null;
List<MyCustomData> mcd = null;
if (searchTerm != null && searchTerm.Length > 1)
{
SearchAutoComplete searchIndex1 = new SearchAutoComplete(Searcher.GetAutoUpdateIndexDirectory(), Portal.AppServices.Constants.UtilitySettings.MaximumResultinAutoUpdate);
suggestions = searchIndex1.SuggestTermsFor(searchTerm);
if (suggestions != null && suggestions.Length > 1)
suggestions = suggestions.Distinct().ToArray<string>();

mcd = new List<MyCustomData>();
if (suggestions != null && suggestions.Length > 0)
{
int count = 0;
foreach (string st in suggestions)
{
if (!st.Contains("<") && !st.Contains(">") && !st.Contains('&') && count <= Portal.AppServices.Constants.UtilitySettings.MaximumResultinAutoUpdate)
{
MyCustomData mc = new MyCustomData();
mc.value = st;
mc.id = count.ToString();
mcd.Add(mc);
count++;
}
}
}
}
System.Web.Script.Serialization.JavaScriptSerializer jss = new System.Web.Script.Serialization.JavaScriptSerializer();
response = jss.Serialize(mcd);
Response.ContentType = "application/json";
Response.ClearContent();
Response.Write(response);
Response.End();
}

/// <summary>
/// This custom type is used to return the data for the Search Term
/// </summary>
public class MyCustomData
{
public string id { get; set; }
public string value { get; set; }
}
}
}

In above code on page load event we check does Request variable has QueryString value called "term", if yes then call the internal function 'SearchTerm' and pass the value which we get from the query string as parameter to it. That function intern will call our above 'SearchAutoComplete' class and get the result as string array. Once we get the array we convert that array into strongly type entity collection which has only two attributes like ID and Value and then finally parse that collection object into JSON string and send back as the response.
Following is the client side code which finally give life to the text box we are looking to build.

Figure 5.2 Auto Search Feature

We used jQueryUI plugin to get the affect of getting menu just below of textbox with the suggestions. Our suggestion will be add to the div named 'ulsuggestion'. Code is simple which jquery on function load bind and key press event to the text box which allows user to get the suggestion by making background ajax request to our Ajax.aspx page. Following is the example from working live website.

So the summary of this article is we are successful to build a new repository based on existing index and then with help of jQuery we are able to pull the suggestion keyword from the new repository on real time basis. I know the way i use this concept may not be perfect like i can use Windows Service which can build the AutoUpdate Index and i can use Signal to get new data from the server or i can use Language Suggestion etc., but whatever the way you choose to implement above the end result will be the same with some plus or minus of performance.

Summary of this whole series is, we started from understanding the Lucene Index and learned how we create/configure new index, how to crawl media files and in end how to provide auto update suggestion for our search text box. With few more features and bits you similarly you also may make the enterprise search end to end in Sitecore 6.0 to 6.6 Version.

Sitecore 7.0 is new and latest and have altogether different search capabilities where we need to write less code and use more. My next project is in Sitecore 7.0 where i may implement search through new way, whatever I do I will keep sharing my learning and I request you all to give praise or curse but please provide the feedback.

25 November 2013

Search with Sitecore - Article - 4 - Create Search API

Dear all friends, this is fourth article in this series and following are the list articles for reference:

1) Search with Sitecore - Article-1 - Introduction
2) Search with Sitecore - Article - 2 - Configure Advance Database Crawler and More
3) Search with Sitecore - Article - 3 - Crawler to Crawl Media File like PDF, Document
4) Search with Sitecore - Article - 4 - Create Search API
5) Search with Sitecore - Article - 5 - Auto Complete For Search

Today we will create some methods and class which can help us to search our new indexes created so far.
Now without taking your much time let me put straight the full code i have in front of you then talk about it.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Sitecore.Caching;
using Sitecore.Data.Items;
using Sitecore.Data.Managers;
using scSearchContrib.Searcher;
using scSearchContrib.Searcher.Utilities;
using scSearchContrib.Searcher.Parameters;
using System.IO;
using Sitecore.Data;
using Sitecore.Collections;
using Sitecore.Search;
using Portal.EShopServices.Enumerations;
using Sitecore.Links;
using Lucene.Net.Search;
using Portal.AppServices.Constants;

namespace Portal.EShopServices.Search.Searcher
{
/// <summary>
/// This class provide the helper method to query the advance Search Crawler
/// This class is based on http://sitecorian.github.io/SitecoreSearchContrib/
/// </summary>
public static class MySearcher
{
#region All Required Properties
/// <summary>
/// Required to Set before any function get called in this Class
/// This will set the index to search
/// </summary>
public static string Index { get; set; }

/// <summary>
/// This property lets you know the result count after the search function is executed.
/// </summary>
public static int ResultCount { get; set; }

/// <summary>
/// This property will allow you to sort the result based on some field
/// Need to set before function is about to execute.
/// </summary>
public static string SortFieldName { get; set; }

/// <summary>
/// This property will allow you to tell query did you want all version of the items or not.
/// Need to set before function is about to execute.
/// </summary>
public static bool IsShowAllVersions { get; set; }

/// <summary>
/// If you have provided the sort property earlier then this property can arrange the order in descending.. as latest to last
/// Need to set before function is about to execute
/// </summary>
public static bool IsReverseTrue { get; set; }

/// <summary>
/// For example, a search resulting in 50 items, page size 10 would go like this:
///Page 1 (start=0, end=10) => items 1-10 (ok)
///Page 2 (start=10, end=20) => items 10-30 (wrong)
///Page 3 (start=20, end=30) => items 20-50 (wrong)
///Page 4 (start=30, end=40) => items 30-50 (wrong)
///Page 5 (start=40, end=50) => items 40-50 (ok)
///Here 'start' is StartRange.
///Need to set before function is about to execute
/// </summary>
public static int StartRange { get; set; }

/// <summary>
/// For example, a search resulting in 50 items, page size 10 would go like this:
///Page 1 (start=0, end=10) => items 1-10 (ok)
///Page 2 (start=10, end=20) => items 10-30 (wrong)
///Page 3 (start=20, end=30) => items 20-50 (wrong)
///Page 4 (start=30, end=40) => items 30-50 (wrong)
///Page 5 (start=40, end=50) => items 40-50 (ok)
///Here 'end' is StartRange.
///Need to set before function is about to execute
/// </summary>
public static int EndRange { get; set; }

/// <summary>
/// This property specifies the query condition like AND/OR/NOT clause
/// Need to set before function is about to execute
/// </summary>
public static Sitecore.Search.QueryOccurance queryOccurence { get; set; }

/// <summary>
/// This property allows to search in base template
/// Need to set before function is about to execute
/// </summary>
public static bool IsSearchBaseTemplate { get; set; }
#endregion

#region Getting Skinny Items - All Raw Search

/// <summary>
/// This will do the search by default for current context database and current context language
/// Make sure to set properties before calling this function
/// </summary>
/// <param name="fullTextQuery">text to be searched</param>
/// <param name="locationFilters">Multiple IDS can be given</param>
/// <param name="templateFilters">Multiple templateIDS can be given</param>
/// <param name="relationFilters">Multiple relationIDS can be given</param>
/// <returns></returns>
public static List<SkinnyItem> GetItems(string fullTextQuery, string locationFilters, string templateFilters, string relationFilters)
{
return GetItems(Sitecore.Context.Database.Name, Sitecore.Context.Language.Name, templateFilters, locationFilters, relationFilters, fullTextQuery, IsSearchBaseTemplate);
}

/// <summary>
/// This will do the search by default for current context database and current context langauge, template filters and relation filters will be send as blank
/// Make sure to set properties before calling this function
/// </summary>
/// <param name="fullTextQuery">text to be searched</param>
/// <param name="locationFilters">Multiple locationIDS can be given</param>
/// <returns></returns>
public static List<SkinnyItem> GetItems(string fullTextQuery, string locationFilters)
{
return GetItems(fullTextQuery, locationFilters, "", "");
}

/// <summary>
/// This will do the search by default for current context database and current context langauge, template filters, location filters and relation filters will be send as blank
/// Make sure to set properties before calling this function
/// </summary>
/// <param name="fullTextQuery">text to be searched</param>
/// <param name="locationFilters">Multiple locationIDS can be given</param>
/// <returns></returns>
public static List<SkinnyItem> GetItems(string fullTextQuery)
{
return GetItems(fullTextQuery, "");
}

public static List<SkinnyItem> GetItems(string databaseName, string language, string templateFilters, string locationFilters, string relationFilters, string fullTextQuery, bool DoSearchBaseTemplate)
{
List<SkinnyItem> result = new List<SkinnyItem>();
var searchParam = new SearchParam();
searchParam = new SearchParam
{
Database = databaseName,
Language = language,
TemplateIds = templateFilters,
LocationIds = locationFilters,
FullTextQuery = fullTextQuery,
RelatedIds = relationFilters,
SearchBaseTemplates = DoSearchBaseTemplate
};
int count = 0;
//Here Portal.AppServices.Constants.UtilitySettings.FileIndexName is name of our earlier created FileIndex
if (Index.Equals(Portal.AppServices.Constants.UtilitySettings.FileIndexName, StringComparison.InvariantCultureIgnoreCase))
{
Sort sort = null;
if (!string.IsNullOrEmpty(SortFieldName))
sort = new Sort(new SortField(SortFieldName.ToLowerInvariant(), SortField.STRING, IsReverseTrue));

Query qr = searchParam.ProcessQuery(queryOccurence, SearchManager.GetIndex(Portal.AppServices.Constants.UtilitySettings.FileIndexName), true);
result = RunQuery(qr, IsShowAllVersions, sort, StartRange, EndRange, out count);
}
else
{
using (var runner = new QueryRunner(Index, true))
{
Query qr = searchParam.ProcessQuery(queryOccurence, SearchManager.GetIndex(Portal.AppServices.Constants.UtilitySettings.FileIndexName), true);
result = runner.RunQuery(qr, false, SortFieldName, true, StartRange, EndRange);
}
}
ResultCount = count;
return result;
}

public static List<SkinnyItem> RunQuery(Query query, bool showAllVersions, Sort sorter, int start, int end, out int totalResults)
{
var items = new List<SkinnyItem>();
if (query == null || string.IsNullOrEmpty(query.ToString()))
{
totalResults = 0;
return items;
}
using (var context = new IndexSearchContext(SearchManager.GetIndex(Index)))
{
SearchHits searchhits;
if (sorter != null)
{
var hits = context.Searcher.Search(query, sorter);
searchhits = new SearchHits(hits);
}
else
{
searchhits = context.Search(query);
}
if (searchhits == null)
{
totalResults = 0;
return null;
}

totalResults = searchhits.Length;
if (end == 0 || end > searchhits.Length)
{
end = totalResults;
}
var resultCollection = searchhits.FetchResults(start, end - start);
GetItemsFromSearchResult(resultCollection, items, showAllVersions);
}
return items;
}

public static void GetItemsFromSearchResult(IEnumerable<SearchResult> searchResults, List<SkinnyItem> items, bool showAllVersions)
{
foreach (var result in searchResults)
{
var guid = result.Document.GetField(BuiltinFields.Tags).StringValue();
if (guid != null && !string.IsNullOrEmpty(guid))
{
var itemId = new ID(guid);
var db = Sitecore.Context.Database;
var item = db.GetItem(itemId);
var itemInfo = new SkinnyItem(item.Uri);
foreach (Lucene.Net.Documents.Field field in result.Document.GetFields())
{
itemInfo.Fields.Add(field.Name(), field.StringValue());
}

items.Add(itemInfo);
}
if (showAllVersions)
{
GetItemsFromSearchResult(result.Subresults, items, true);
}
}
}
//This function is used in the Auto Search Functionality for the text box
public static Lucene.Net.Store.FSDirectory GetAutoUpdateIndexDirectory()
{
string autoUpdate = Portal.AppServices.Constants.UtilitySettings.OuterIndexFolder + @"\" + Portal.AppServices.Constants.UtilitySettings.AutoUpdateIndexName;
try
{
if (!System.IO.Directory.Exists(autoUpdate))
System.IO.Directory.CreateDirectory(autoUpdate);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message + System.Environment.NewLine);
Console.WriteLine(ex.StackTrace);
}
Lucene.Net.Store.FSDirectory fsd = null;
if (System.IO.Directory.Exists(autoUpdate))
{
fsd = Lucene.Net.Store.FSDirectory.Open(new System.IO.DirectoryInfo(autoUpdate));
}
return fsd;
}
#endregion

#region Header Search Functions

/// <summary>
/// Handle non-pages results to remove them from search results (prevents error pages)
/// </summary>
/// <param name="_items">List of Search Results</param>
/// <returns>List of Filtered Search Results</returns>
public static List<WrapUpItem> HandleNonPagesItem(List<Item> _items)
{
List<WrapUpItem> processList = new List<WrapUpItem>();
foreach (Item itm in _items)
{
if (DoesSitecoreItemHavePeresentation(itm))
{
WrapUpItem _wui = new WrapUpItem();
_wui.ItemType = WrapUpItem.type.Default;
_wui.Item = itm;
processList.Add(_wui);
}
}
return processList;
}

public static List<WrapUpItem> HandleMediaItems(List<Item> _items)
{
List<WrapUpItem> processList = new List<WrapUpItem>();
foreach (MediaItem itm in _items)
{
WrapUpItem _wui = null;
try
{
if (Sitecore.Resources.Media.MediaManager.HasMediaContent(itm))
{
_wui = new WrapUpItem();
_wui.MediaUrl = Sitecore.Resources.Media.MediaManager.GetMediaUrl(itm);
_wui.Item = itm;
if (itm.MimeType.ToLower().Contains("pdf"))
{
_wui.MediaType = WrapUpItem.mediatype.PDF;
}
else if (itm.MimeType.ToLower().Contains("word"))
{
_wui.MediaType = WrapUpItem.mediatype.Word;
}
else if (itm.MimeType.ToLower().Contains("text"))
{
_wui.MediaType = WrapUpItem.mediatype.Text;
}
else if (itm.MimeType.ToLower().Contains("html"))
{
_wui.MediaType = WrapUpItem.mediatype.HTML;
}
_wui.ItemType = WrapUpItem.type.Media;
}
}
catch { }
finally { if (_wui != null) { processList.Add(_wui); } }
}
return processList;
}

public static List<WrapUpItem> HandleProductItems(List<Item> _items)
{
List<WrapUpItem> processList = new List<WrapUpItem>();
foreach (Item itm in _items)
{
WrapUpItem _wui = null;
bool isActiveProduct = false;
try
{
if (isProductItem(itm))
{
_wui = new WrapUpItem();
_wui.ItemType = WrapUpItem.type.Product;
_wui.Item = itm;
//Write your logic to assign further properties of the WrapUpItem class
}
}
catch { }
finally { if (_wui != null && isActiveProduct) { processList.Add(_wui); } }
}
return processList;
}

public static bool isProductItem(Item itm)
{
bool result = false;
// Write your own logic to distinguish specific Item like product
return result;
}

public static bool DoesSitecoreItemHavePeresentation(Item item)
{
return item.Fields[Sitecore.FieldIDs.LayoutField] != null
&& item.Fields[Sitecore.FieldIDs.LayoutField].Value
!= String.Empty;
}

/// <summary>
/// Check if an item URL contains "-w-", which is an invalid link
/// </summary>
/// <param name="_item"></param>
/// <returns>true if contains "-w-"; otherwise, false</returns>
private static bool CheckForParameters(Item _item)
{
string itemUrl = LinkManager.GetItemUrl(_item);
if (itemUrl.Contains("-w-"))
return true;
else
return false;
}
#endregion
}
/// <summary>
/// This Class Provide an Wrap Class which has additional properties required during search.
/// </summary>
public class WrapUpItem
{
public enum type
{
Product = 0,
Media = 1,
Default = 2
}

public enum mediatype
{
PDF = 0,
Word = 1,
Text = 2,
HTML = 3
}
public Item Item { get; set; }
public type ItemType { get; set; }
public string MediaUrl { get; set; }
public string ProductUrl { get; set; }
public mediatype MediaType { get; set; }
}
}

First few facts about above code I have three main index and separate index which doesn't build from the Sitecore data.

a) SeacrhItems (This index is for routine content items and its root is pointing to the Home Node which includes it all child also).
b) Product (This index is for specific items which are product items and located under the product catalog obviously out of Home Node).
c) Documents (This index is specifically for the media items stored under the media library path).
d) AutoUpdate (This index is not based on the Sitecore Items and not used for routine search queries, but stored in the same location where other Sitecore Indexes are stored, we will discuss about this index in later post).

Similarly you may have as many index you want targeted to your specific set of contents. I created above class keeping multiple index in mind so that i can query whatever index i like and club them as collection and show them on front-end.
Why i need to club the collection ?
--Well it depend upon requirement, in our project we have just one global textbox sitting on Header where user can type any query and then result should be shown on relevant tabs on an page. So first tab results in collection of all results find through all indexes except fourth one. Here results are arranged in the precedence given to Products >> Normal Search Items >> Files. Well other tabs have particular location targeted like business section, FAQ section etc., under Home node (Using above API we can search based on the location from our SearchItems Index).

Now To quickly explain what we have done above, is first we have an class which provide us properties, flag variables and methods which assist us to make search and differentiate between items. Items could be media files, normal content item or could be some specific business item like product for which you may target in search. Above class also provide some routine functions like knowing whether an Item is an brows-able item or not.
From purpose of making binding easy at the front-end side in .net user control we have created another class called 'WrapUpItem' which as name predict is just a wrapper class which contain additional properties relevant to an searched Item. These properties can be more or less depend upon your business context / requirements. I am not suggesting that above is the best approach of doing it, i am just saying that it is one of the approach available while you do search in Sitecore.
Further below are the initiator functions(How to use above class) for each of the index from which you want to make search, these function can sit within .NET User control or wherever you like.

//This Property store the searched keyword in session you may store wherever you like
public string SearchText
{
get
{
return ValidationHelper.ValidateToString(Session[Portal.EShopServices.Constants.Sessions.SearchText], "");
}
set
{
Session[Portal.EShopServices.Constants.Sessions.SearchText] = value;
}
}

protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getProductSearch()
{
List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
List<Item> _liSitecoreItems = null;
List<SkinnyItem> ski = null;
Portal.EShopServices.Search.Searcher.Searcher.Index = Portal.AppServices.Constants.UtilitySettings.ProductIndexName;
Portal.EShopServices.Search.Searcher.Searcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
//We can concatenate the GUID of all templates required using | sign and get from some constant value
string templateIDs = Portal.EShopServices.Constants.TemplateID.AllProducts;
//We can store the location of products stored under product catalog in app settings etc.,
string locationId = Sitecore.Context.Database.GetItem(Portal.AppServices.Constants.UtilitySettings.eShop).ID.ToString();
// We are using Advance Database crawler functions need to get them referenced
ski = Portal.EShopServices.Search.Searcher.MySearcher.GetItems(SearchText, locationId, templateIDs, "");
//Get Item List collection from the skinny collection
if (ski != null && ski.Count > 0)
{
_liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
}
//Clean the non product Items
if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
{
liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleProductItems(_liSitecoreItems);
}
return liItems;
}

protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getMySearchItem(string locationId)
{
List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
List<Item> _liSitecoreItems = null;
List<SkinnyItem> ski = null;
Portal.EShopServices.Search.Searcher.MySearcher.Index = Portal.AppServices.Constants.UtilitySettings.ContentIndexName;
Portal.EShopServices.Search.Searcher.MySearcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
if (locationId.Length > 0)
{
ski = Portal.EShopServices.Search.Searcher.Searcher.GetItems(SearchText, locationId);
}
else
{
ski = Portal.EShopServices.Search.Searcher.Searcher.GetItems(SearchText);
}
//Get Item List collection from the skinny collection
if (ski != null && ski.Count > 0)
{
_liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
}
//Clean the non-browsable items
if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
{
liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleNonPagesItem(_liSitecoreItems);
}
return liItems;
}

protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getFileSearch()
{

List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
List<Item> _liSitecoreItems = null;
List<SkinnyItem> ski = null;
Portal.EShopServices.Search.Searcher.MySearcher.Index = Portal.AppServices.Constants.UtilitySettings.FileIndexName;
Portal.EShopServices.Search.Searcher.MySearcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
ski = Portal.EShopServices.Search.Searcher.MySearcher.GetItems(SearchText);
//Get Item List collection from the skinny collection
if (ski != null && ski.Count > 0)
{
_liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
}
//Clean the non-media items
if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
{
liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleMediaItems(_liSitecoreItems);
}
return liItems;
}

I hope above article is able to complete the picture of how to search from the custom indexes created using advance database crawler. ADC(advance database crawler) itself gives few function from which you can directly search but i prefer to create my own single class which can use ADC and become centralized approach for querying any index.

In next article we will see how to provide the auto update feature in the text box where keyword should come real from our indexed data, that would be final article in this series. Please let me know feedback so far in comments.

Pages

27 November 2013

Search with Sitecore - Article - 5 - Auto Complete For Search

25 November 2013

Search with Sitecore - Article - 4 - Create Search API

Labels

Followers

Total Pageviews

Pages

27 November 2013

Search with Sitecore - Article - 5 - Auto Complete For Search

25 November 2013

Search with Sitecore - Article - 4 - Create Search API

Be in Touch Allways

Total Pageviews