Search with Sitecore - Article - 4 - Create Search API

Dear all friends, this is fourth article in this series and following are the list articles for reference:


1) Search with Sitecore - Article-1 - Introduction 
2) Search with Sitecore - Article - 2 - Configure Advance Database Crawler and More
3) Search with Sitecore - Article - 3 - Crawler to Crawl Media File like PDF, Document  
4) Search with Sitecore - Article - 4 - Create Search API
5) Search with Sitecore - Article - 5 - Auto Complete For Search

Today we will create some methods and class which can help us to search our new indexes created so far.
Now without taking your much time let me put straight the full code i have in front of you then talk about it.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Sitecore.Caching;
using Sitecore.Data.Items;
using Sitecore.Data.Managers;
using scSearchContrib.Searcher;
using scSearchContrib.Searcher.Utilities;
using scSearchContrib.Searcher.Parameters;
using System.IO;
using Sitecore.Data;
using Sitecore.Collections;
using Sitecore.Search;
using Portal.EShopServices.Enumerations;
using Sitecore.Links;
using Lucene.Net.Search;
using Portal.AppServices.Constants;

namespace Portal.EShopServices.Search.Searcher
{
    /// <summary>
    /// This class provide the helper method to query the advance Search Crawler 
    /// This class is based on http://sitecorian.github.io/SitecoreSearchContrib/
    /// </summary>
    public static class MySearcher
    {
        #region All Required Properties
        /// <summary>
        /// Required to Set before any function get called in this Class
        /// This will set the index to search
        /// </summary>
        public static string Index { get; set; }

        /// <summary>
        /// This property lets you know the result count after the search function is executed.
        /// </summary>
        public static int ResultCount { get; set; }

        /// <summary>
        /// This property will allow you to sort the result based on some field
        /// Need to set before function is about to execute.
        /// </summary>
        public static string SortFieldName { get; set; }

        /// <summary>
        /// This property will allow you to tell query did you want all version of the items or not.
        /// Need to set before function is about to execute.
        /// </summary>
        public static bool IsShowAllVersions { get; set; }

        /// <summary>
        /// If you have provided the sort property earlier then this property can arrange the order in descending.. as latest to last
        /// Need to set before function is about to execute
        /// </summary>
        public static bool IsReverseTrue { get; set; }

        /// <summary>
        /// For example, a search resulting in 50 items, page size 10 would go like this:
        ///Page 1 (start=0, end=10) => items 1-10 (ok)
        ///Page 2 (start=10, end=20) => items 10-30 (wrong)
        ///Page 3 (start=20, end=30) => items 20-50 (wrong)
        ///Page 4 (start=30, end=40) => items 30-50 (wrong)
        ///Page 5 (start=40, end=50) => items 40-50 (ok)
        ///Here 'start' is StartRange.
        ///Need to set before function is about to execute
        /// </summary>
        public static int StartRange { get; set; }

        /// <summary>
        /// For example, a search resulting in 50 items, page size 10 would go like this:
        ///Page 1 (start=0, end=10) => items 1-10 (ok)
        ///Page 2 (start=10, end=20) => items 10-30 (wrong)
        ///Page 3 (start=20, end=30) => items 20-50 (wrong)
        ///Page 4 (start=30, end=40) => items 30-50 (wrong)
        ///Page 5 (start=40, end=50) => items 40-50 (ok)
        ///Here 'end' is StartRange.
        ///Need to set before function is about to execute
        /// </summary>
        public static int EndRange { get; set; }

        /// <summary>
        /// This property specifies the query condition like AND/OR/NOT clause
        /// Need to set before function is about to execute
        /// </summary>
        public static Sitecore.Search.QueryOccurance queryOccurence { get; set; }

        /// <summary>
        /// This property allows to search in base template
        /// Need to set before function is about to execute
        /// </summary>
        public static bool IsSearchBaseTemplate { get; set; }
        #endregion

        #region Getting Skinny Items - All Raw Search

        /// <summary>
        /// This will do the search by default for current context database and current context language
        /// Make sure to set properties before calling this function
        /// </summary>
        /// <param name="fullTextQuery">text to be searched</param>
        /// <param name="locationFilters">Multiple IDS can be given</param>
        /// <param name="templateFilters">Multiple templateIDS can be given</param>
        /// <param name="relationFilters">Multiple relationIDS can be given</param>
        /// <returns></returns>
        public static List<SkinnyItem> GetItems(string fullTextQuery, string locationFilters, string templateFilters, string relationFilters)
        {
            return GetItems(Sitecore.Context.Database.Name, Sitecore.Context.Language.Name, templateFilters, locationFilters, relationFilters, fullTextQuery, IsSearchBaseTemplate);
        }

        /// <summary>
        /// This will do the search by default for current context database and current context langauge, template filters and relation filters will be send as blank
        /// Make sure to set properties before calling this function
        /// </summary>
        /// <param name="fullTextQuery">text to be searched</param>
        /// <param name="locationFilters">Multiple locationIDS can be given</param>
        /// <returns></returns>
        public static List<SkinnyItem> GetItems(string fullTextQuery, string locationFilters)
        {
            return GetItems(fullTextQuery, locationFilters, "", "");
        }

        /// <summary>
        /// This will do the search by default for current context database and current context langauge, template filters, location filters and relation filters will be send as blank
        /// Make sure to set properties before calling this function
        /// </summary>
        /// <param name="fullTextQuery">text to be searched</param>
        /// <param name="locationFilters">Multiple locationIDS can be given</param>
        /// <returns></returns>
        public static List<SkinnyItem> GetItems(string fullTextQuery)
        {
            return GetItems(fullTextQuery, "");
        }

        public static List<SkinnyItem> GetItems(string databaseName, string language, string templateFilters, string locationFilters, string relationFilters, string fullTextQuery, bool DoSearchBaseTemplate)
        {
            List<SkinnyItem> result = new List<SkinnyItem>();
            var searchParam = new SearchParam();
            searchParam = new SearchParam
            {
                Database = databaseName,
                Language = language,
                TemplateIds = templateFilters,
                LocationIds = locationFilters,
                FullTextQuery = fullTextQuery,
                RelatedIds = relationFilters,
                SearchBaseTemplates = DoSearchBaseTemplate
            };
            int count = 0;
             //Here Portal.AppServices.Constants.UtilitySettings.FileIndexName is name of our earlier created FileIndex
            if (Index.Equals(Portal.AppServices.Constants.UtilitySettings.FileIndexName, StringComparison.InvariantCultureIgnoreCase))
            {
                Sort sort = null;
                if (!string.IsNullOrEmpty(SortFieldName))
                    sort = new Sort(new SortField(SortFieldName.ToLowerInvariant(), SortField.STRING, IsReverseTrue));

                Query qr = searchParam.ProcessQuery(queryOccurence, SearchManager.GetIndex(Portal.AppServices.Constants.UtilitySettings.FileIndexName), true);
                result = RunQuery(qr, IsShowAllVersions, sort, StartRange, EndRange, out count);
            }
            else
            {
                using (var runner = new QueryRunner(Index, true))
                {
                    Query qr = searchParam.ProcessQuery(queryOccurence, SearchManager.GetIndex(Portal.AppServices.Constants.UtilitySettings.FileIndexName), true);
                    result = runner.RunQuery(qr, false, SortFieldName, true, StartRange, EndRange);
                }
            }
            ResultCount = count;
            return result;
        }

        public static List<SkinnyItem> RunQuery(Query query, bool showAllVersions, Sort sorter, int start, int end, out int totalResults)
        {
            var items = new List<SkinnyItem>();
            if (query == null || string.IsNullOrEmpty(query.ToString()))
            {
                totalResults = 0;
                return items;
            }
            using (var context = new IndexSearchContext(SearchManager.GetIndex(Index)))
            {
                SearchHits searchhits;
                if (sorter != null)
                {
                    var hits = context.Searcher.Search(query, sorter);
                    searchhits = new SearchHits(hits);
                }
                else
                {
                    searchhits = context.Search(query);
                }
                if (searchhits == null)
                {
                    totalResults = 0;
                    return null;
                }

                totalResults = searchhits.Length;
                if (end == 0 || end > searchhits.Length)
                {
                    end = totalResults;
                }
                var resultCollection = searchhits.FetchResults(start, end - start);
                GetItemsFromSearchResult(resultCollection, items, showAllVersions);
            }
            return items;
        }

        public static void GetItemsFromSearchResult(IEnumerable<SearchResult> searchResults, List<SkinnyItem> items, bool showAllVersions)
        {
            foreach (var result in searchResults)
            {
                var guid = result.Document.GetField(BuiltinFields.Tags).StringValue();
                if (guid != null && !string.IsNullOrEmpty(guid))
                {
                    var itemId = new ID(guid);
                    var db = Sitecore.Context.Database;
                    var item = db.GetItem(itemId);
                    var itemInfo = new SkinnyItem(item.Uri);
                    foreach (Lucene.Net.Documents.Field field in result.Document.GetFields())
                    {
                        itemInfo.Fields.Add(field.Name(), field.StringValue());
                    }

                    items.Add(itemInfo);
                }
                if (showAllVersions)
                {
                    GetItemsFromSearchResult(result.Subresults, items, true);
                }
            }
        }
       //This function is used in the Auto Search Functionality for the text box        
 public static Lucene.Net.Store.FSDirectory GetAutoUpdateIndexDirectory()
        {
            string autoUpdate = Portal.AppServices.Constants.UtilitySettings.OuterIndexFolder + @"\" + Portal.AppServices.Constants.UtilitySettings.AutoUpdateIndexName;
            try
            {
                if (!System.IO.Directory.Exists(autoUpdate))
                    System.IO.Directory.CreateDirectory(autoUpdate);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message + System.Environment.NewLine);
                Console.WriteLine(ex.StackTrace);
            }
            Lucene.Net.Store.FSDirectory fsd = null;
            if (System.IO.Directory.Exists(autoUpdate))
            {
                fsd = Lucene.Net.Store.FSDirectory.Open(new System.IO.DirectoryInfo(autoUpdate));
            }
            return fsd;
        }
        #endregion

        #region Header Search Functions

        /// <summary>
        /// Handle non-pages results to remove them from search results (prevents error pages)
        /// </summary>
        /// <param name="_items">List of Search Results</param>
        /// <returns>List of Filtered Search Results</returns>
        public static List<WrapUpItem> HandleNonPagesItem(List<Item> _items)
        {
            List<WrapUpItem> processList = new List<WrapUpItem>();
            foreach (Item itm in _items)
            {
                if (DoesSitecoreItemHavePeresentation(itm))
                {
                    WrapUpItem _wui = new WrapUpItem();
                    _wui.ItemType = WrapUpItem.type.Default;
                    _wui.Item = itm;
                    processList.Add(_wui);
                }
            }
            return processList;
        }

        public static List<WrapUpItem> HandleMediaItems(List<Item> _items)
        {
            List<WrapUpItem> processList = new List<WrapUpItem>();
            foreach (MediaItem itm in _items)
            {
                WrapUpItem _wui = null;
                try
                {
                    if (Sitecore.Resources.Media.MediaManager.HasMediaContent(itm))
                    {
                        _wui = new WrapUpItem();
                        _wui.MediaUrl = Sitecore.Resources.Media.MediaManager.GetMediaUrl(itm);
                        _wui.Item = itm;
                        if (itm.MimeType.ToLower().Contains("pdf"))
                        {
                            _wui.MediaType = WrapUpItem.mediatype.PDF;
                        }
                        else if (itm.MimeType.ToLower().Contains("word"))
                        {
                            _wui.MediaType = WrapUpItem.mediatype.Word;
                        }
                        else if (itm.MimeType.ToLower().Contains("text"))
                        {
                            _wui.MediaType = WrapUpItem.mediatype.Text;
                        }
                        else if (itm.MimeType.ToLower().Contains("html"))
                        {
                            _wui.MediaType = WrapUpItem.mediatype.HTML;
                        }
                        _wui.ItemType = WrapUpItem.type.Media;
                    }
                }
                catch { }
                finally { if (_wui != null) { processList.Add(_wui); } }
            }
            return processList;
        }

        public static List<WrapUpItem> HandleProductItems(List<Item> _items)
        {
            List<WrapUpItem> processList = new List<WrapUpItem>();
            foreach (Item itm in _items)
            {
                WrapUpItem _wui = null;
                bool isActiveProduct = false;
                try
                {
                    if (isProductItem(itm))
                    {
                        _wui = new WrapUpItem();
                        _wui.ItemType = WrapUpItem.type.Product;
                        _wui.Item = itm;
                       //Write your logic to assign further properties of the WrapUpItem class
                    }
                }
                catch { }
                finally { if (_wui != null && isActiveProduct) { processList.Add(_wui); } }
            }
            return processList;
        }

        public static bool isProductItem(Item itm)
        {
            bool result = false;
            // Write your own logic to distinguish specific Item like product 
            return result;
        }

        public static bool DoesSitecoreItemHavePeresentation(Item item)
        {
            return item.Fields[Sitecore.FieldIDs.LayoutField] != null
                   && item.Fields[Sitecore.FieldIDs.LayoutField].Value
                   != String.Empty;
        }

        /// <summary>
        /// Check if an item URL contains "-w-", which is an invalid link
        /// </summary>
        /// <param name="_item"></param>
        /// <returns>true if contains "-w-"; otherwise, false</returns>
        private static bool CheckForParameters(Item _item)
        {
            string itemUrl = LinkManager.GetItemUrl(_item);
            if (itemUrl.Contains("-w-"))
                return true;
            else
                return false;
        }
        #endregion
    }
    /// <summary>
    /// This Class Provide an Wrap Class which has additional properties required during search. 
    /// </summary>
    public class WrapUpItem
    {
        public enum type
        {
            Product = 0,
            Media = 1,
            Default = 2
        }

        public enum mediatype
        {
            PDF = 0,
            Word = 1,
            Text = 2,
            HTML = 3
        }
        public Item Item { get; set; }
        public type ItemType { get; set; }
        public string MediaUrl { get; set; }
        public string ProductUrl { get; set; }
        public mediatype MediaType { get; set; }
    }
}

First few facts about above code I have three main index and separate index which doesn't build from the Sitecore data.

a) SeacrhItems (This index is for routine content items and its root is pointing to the Home Node which includes it all child also).
b) Product (This index is for specific items which are product items and located under the product catalog obviously out of Home Node).
c) Documents (This index is specifically for the media items stored under the media library path).
d) AutoUpdate (This index is not based on the Sitecore Items and not used for routine search queries, but stored in the same location where other Sitecore Indexes are stored, we will discuss about this index in later post).

Similarly you may have as many index you want targeted to your specific set of contents. I created above class keeping multiple index in mind so that i can query whatever index i like and club them as collection and show them on front-end.
Why i need to club the collection ?
--Well it depend upon requirement, in our project we have just one global textbox sitting on Header where user can type any query and then result should be shown on relevant tabs on an page. So first tab results in collection of all results find through all indexes except fourth one. Here results are arranged in the precedence given to Products >> Normal Search Items >> Files. Well other tabs have particular location targeted like business section, FAQ section etc., under Home node (Using above API we can search based on the location from our SearchItems Index).

Now To quickly explain what we have done above, is first we have an class which provide us properties, flag variables and methods which assist us to make search and differentiate between items. Items could be media files, normal content item or could be some specific business item like product for which you may target in search. Above class also provide some routine functions like knowing whether an Item is an brows-able item or not.
From purpose of making binding easy at the front-end side in .net user control we have created another class called 'WrapUpItem' which as name predict is just a wrapper class which contain additional properties relevant to an searched Item. These properties can be more or less depend upon your business context / requirements. I am not suggesting that above is the best approach of doing it, i am just saying that it is one of the approach available while you do search in Sitecore.
Further below are the initiator functions(How to use above class) for each of the index from which you want to make search, these function can sit within .NET User control or wherever you like.

//This Property store the searched keyword in session you may store wherever you like
public string SearchText
        {
            get
            {
                return ValidationHelper.ValidateToString(Session[Portal.EShopServices.Constants.Sessions.SearchText], "");
            }
            set
            {
                Session[Portal.EShopServices.Constants.Sessions.SearchText] = value;
            }
        }

protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getProductSearch()
        {
            List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
            List<Item> _liSitecoreItems = null;
            List<SkinnyItem> ski = null;
            Portal.EShopServices.Search.Searcher.Searcher.Index = Portal.AppServices.Constants.UtilitySettings.ProductIndexName;
            Portal.EShopServices.Search.Searcher.Searcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
            //We can concatenate the GUID of all templates required using | sign and get from some constant value
            string templateIDs = Portal.EShopServices.Constants.TemplateID.AllProducts;
           //We can store the location of products stored under product catalog in app settings etc.,
            string locationId = Sitecore.Context.Database.GetItem(Portal.AppServices.Constants.UtilitySettings.eShop).ID.ToString();
            // We are using Advance Database crawler functions need to get them referenced
            ski = Portal.EShopServices.Search.Searcher.MySearcher.GetItems(SearchText, locationId, templateIDs, "");
            //Get Item List collection from the skinny collection
            if (ski != null && ski.Count > 0)
            {
                _liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
            }
            //Clean the non product Items
            if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
            {
                liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleProductItems(_liSitecoreItems);
            }
            return liItems;
        }

        protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getMySearchItem(string locationId)
        {
            List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
            List<Item> _liSitecoreItems = null;
            List<SkinnyItem> ski = null;
            Portal.EShopServices.Search.Searcher.MySearcher.Index = Portal.AppServices.Constants.UtilitySettings.ContentIndexName;
            Portal.EShopServices.Search.Searcher.MySearcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
            if (locationId.Length > 0)
            {
                ski = Portal.EShopServices.Search.Searcher.Searcher.GetItems(SearchText, locationId);
            }
            else
            {
                ski = Portal.EShopServices.Search.Searcher.Searcher.GetItems(SearchText);
            }
            //Get Item List collection from the skinny collection
            if (ski != null && ski.Count > 0)
            {
                _liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
            }
            //Clean the non-browsable items
            if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
            {
                liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleNonPagesItem(_liSitecoreItems);
            }
            return liItems;
        }

        protected List<Portal.EShopServices.Search.Searcher.WrapUpItem> getFileSearch()
        {

            List<Portal.EShopServices.Search.Searcher.WrapUpItem> liItems = null;
            List<Item> _liSitecoreItems = null;
            List<SkinnyItem> ski = null;
            Portal.EShopServices.Search.Searcher.MySearcher.Index = Portal.AppServices.Constants.UtilitySettings.FileIndexName;
            Portal.EShopServices.Search.Searcher.MySearcher.queryOccurence = Sitecore.Search.QueryOccurance.Must;
            ski = Portal.EShopServices.Search.Searcher.MySearcher.GetItems(SearchText);
            //Get Item List collection from the skinny collection
            if (ski != null && ski.Count > 0)
            {
                _liSitecoreItems = SearchHelper.GetItemListFromInformationCollection(ski);
            }
            //Clean the non-media items
            if (_liSitecoreItems != null && _liSitecoreItems.Count > 0)
            {
                liItems = Portal.EShopServices.Search.Searcher.MySearcher.HandleMediaItems(_liSitecoreItems);
            }
            return liItems;
        }

I hope above article is able to complete the picture of how to search from the custom indexes created using advance database crawler. ADC(advance database crawler) itself gives few function from which you can directly search but i prefer to create my own single class which can use ADC and become centralized approach for querying any index.

In next article we will see how to provide the auto update feature in the text box where keyword should come real from our indexed data, that would be final article in this series. Please let me know feedback so far in comments.