Metadata Property Mappings

In order for me to make full use the FullTextSqlQuery SharePoint object which I spoke about in my previous blog here I needed to understand how metadata property mapping worked. This is my "lamens term" explanation: There are "Crawled Properties" which as a SQL developer you could think of as an indexed field. Then you get a "Managed Property" which you can map to multiple "Crawled Properties". The "Managed Property" is the field that will be available to use. You could think of this property as an interface to your indexed field(s).

So hopefully that doesn't make things more complicated, moving forward this is how you would go about creating Metadata Property Mapping:

To navigate to the Property Mapping Page go to Shared Services Provider. On the home page click on the "Search settings" option under "Search". The navigation menu should have changed now. Click on the "Metadata properties" item in the quick launch menu under "Queries and Results". On this screen you can see all the managed properties and their crawled property mappings. From this page you are able to see all meta data properties. 

If you navigate to the "Title" propery and click on it you will see the following screen:

Let me explain what all these properties mean:

A. The type of information in this property. The available types are (Text, Integer, Decimal, Date and Time, Yes/No). Note that you can only link crawled properties that are the same type as your managed property.

B. The number of items found with this property. This is a very useful indication of whether your managed property will actually return any results. If this value is 0 is means one of two things.
1. Your mapped crawled properties do not contain any values
2. You haven't run a crawl yet. After you have created a managed property always run a crawl in order for SharePoint to index it.

C. Option "Include values from all crawled properties mapped" means that all mapped crawled properties will be returned in a deliminated fashion.
Option "Include values from a single crawled property based on the order specified" means that it will only return the first value that is not null depending on their order in the list box.

D. Crawled properties mapping to this managed property.
On the title example you will see that the first mapped property is "Mail:5(Text)". The keyword "Mail:" means that this crawled property will be retrieved out of a mail object. This full name is not very descriptive to the user but you can just google a field or test it out to find out what value it contains.

If you click on the "Add Mapping" Button you will be able to search for your desired property. If you are looking for a column inside a list or library you will find them under the "SharePoint" category. All custom column's crawled property names will begin with "ows_" and spaces will be converted to "_x0020_". Note: if you cannot find the property you are looking for check whether they are of the same type as your managed property. This has got me before ;)

E. Allow this property to be used in scopes.
This boolean value can be misleading in it's description. If this checkbox is selected it means that you will be able to create a rule in your scope that can use this property. If this property is unchecked you can still use scopes in conjunction with this field in your searches.

So one last thing to remember is ALWAYS RUN A CRAWL after any updates!!!

SQL Query in SharePoint

The "FullTextSqlQuery" object's constructor requires an object that has context but don't be fooled. This context will not be the context in which your query will search on. Scopes are used for that are I will be explaining scopes a little later on.

Query Text

The "FullTextSqlQuery" object has a "QueryText" property which takes a query very similar to a generic SQL query. Here are a list of minor differences for those who are interested:

• There is only one table to search on - "Scope" and this table is suffixed by parenthesis.
• You cannot search on all columns using *. You have to specify the column names you require.
• There are no aggregate functions eg. 'Count'
• I found that "LIKE" doesn't work. I have found examples that use "LIKE" and state that it works. I myself never got the results I was expecting so I use "CONTAINS(Title, 'SharePoint').
• The comparable commands are Equals "=", Not Equals "!=", Contains "CONTAINS(ColumnName, 'Value')" and FREETEXT(*, 'value') which allows you to search on all columns. NOTE: You cannot put spaces in any of these value objects. Replace all spaces with the '+' character to search multiple words. And use the '*' character in the values as a wildcard - WILDCARDS ONLY WORK IN SP 2010!.

So here is an example of what your QueryText could look like:

SELECT Title, Description, Path FROM SCOPE() WHERE CONTAINS(Title, 'SharePoint') AND IsDocument = 1 ORDER BY Title

Query Properties

There are some properties on the "FullTextSqlQuery" object that you will need to set.

• ResultTypes. This is an enum of type Microsoft.Office.Server.Search.Query.ResultType. I've only ever used the "RelevantResults" value. I've tested the rest but I don't get any results using them so not sure what they are for.
• TrimDuplicates. As the name states.
• StartRow. Used for paging. The index of the first item to return. NOTE: If you're using the API this index starts at 0 but if you are using the webservice to call a Query the index starts at 1.
• RowLimit. Used for paging. The amount of results the query must return.

There are some more properties but I'm not sure exactly what they are used for yet.

Columns

If you want to know what columns you can select on you can see this in your Shared Services Provider. On the home page click on the "Search settings" option under "Search". The navigation menu should have changed now. Click on the "Metadata properties" item in the quick launch menu under "Queries and Results". On this screen you can see all the managed properties and their crawled property mappings. Check about my other post "Metadata Property Mapping" for more information.

Scopes

Scopes allow us to search in specific areas of sharepoint. I haven't worked extensively with this yet but from what I understand at the moment you can only create a scope inside a web application and a site collection. This means that if you have multiple web applications and multiple site collections you can make your search specific to one of multiple of these. If you don't specify a scope the search will return results from all web application and all site collections. This optomises your search greatly as the scopes index all your results. NOTE: In your for your scope to be visible from your "FullTextSqlQuery" it needs to be Shared.

Indexing

All "Managed properties" and "Scopes" are indexed by SharePoint. When a new object is added to SharePoint is will not be indexed yet and will not be returned by your search query until this has happened. A crawl needs to be started in order for any new objects or object changes to appear in your results. If you are busy testing your search you can manually start a full crawl inside your Shared Services Site. On the home page click on the "Search settings" option under "Search". The navigation menu should have changed now. Click on the "Content sources" item in the quick launch menu under "Crawling". You can start a full crawl on your content source on this page. Once the crawl is complete run your search to see if your results are being returned. If you want to deploy a custom search feature you will need to setup an incremental crawl for this content source so that it executes every so often.

Results

The results of a "FullTextSqlQuery" are returned in the form of a "ResultTableCollection" from the "Execute" method. In order to get your results from this collection you need to the index the collection with your ResultType like so "fullTextSqlQuery.Execute()[ResultType.RelevantResults];".
The ResultTable is not very straight forward and I struggled to get my results out. First you need to call the Read() function inside a while loop and then use a for loop to get the index of each result item. You then use two methods on the table: "GetName()" and "GetValue()". Each of these methods take an index parameter. I then need to compare the name to the name im looking for and set the value for the object. Here's a code sample:

while (resultTable.Read())
{
  for (int i = 0; i < resultTable.FieldCount; i++)
  {
      string name = resultTable.GetName(i);
      object objValue = resultTable.GetValue(i);
      string value = string.Empty;
      if (objValue != null)
      {
        value = objValue.ToString();
      }

      if (name == SearchProperties.Title.ToString().ToUpper(CultureInfo.CurrentCulture))
      {
        sr.Title = value;
      }                       
   }
}

Bugs
Yup, like most complex software out there, even Microsoft software, there are bugs. So far I have picked up two.

1. Sort by Author does not return desired results. The Author managed property is linked to a Mail and Office category field. For some reason if you sort by this property it adds a null row after every result so your count will double. As far as I'm aware this is a bug in 2007 and 2010.

2. ModifedBy and CreatedBy are empty. Although these fields exist inside the SPItem, you will not be able to retrieve them using the FullTextSqlQuery object. It seems that these fields are not indexable and they will always return null.

Resources
1. http://community.bamboosolutions.com/blogs/bambooteamblog/archive/2009/04/24/wss-custom-search.aspx
2. http://blogs.msdn.com/b/varun_malhotra/archive/2008/08/16/moss-search-with-order-by-clause-doesn-t-return-all-results.aspx