This is the second in a series of articles on how we’ve built JobServe Labs and some of the projects that are available.  In the first article we described how we’re using Windows Live Writer to manage our content, and how we’ve written an XML RPC backend for it to talk to, using WCF in C#.

This time, we’re going to look at the suggestions service that we did for our Web Browser Search Provider, and how you can set up your own.

Before you start reading, please feel free to download the source code that is built by following this walkthrough – it also shows how to output the images in the suggestions that are supported by IE8.

What is OpenSearch?

Originally, OpenSearch enabled a website to expose to interested consumers (be they web browsers, search engines or anything else) any of the search capabilities that they might have.  The notion of extracting this out to a standard meant that a consumer could be automatically compatible with lots of search engines once the code had been written to interpret one.

The search facility of a website typically takes the form of a URL that can accept a URL-encoded search phrase in the query string – for example, many people will be familiar with Google’s query string format, since each search results page is represented by a different URL:

http://www.google.co.uk/search?hl=en&q=jobserve – first page of a Google Search for ‘jobserve’

http://www.google.co.uk/search?q=jobserve&hl=en&start=10&sa=N – second page – note the additional ‘start’ parameter, which tells the search engine which result should come first.

If you’re looking to implement OpenSearch for your website, then you should, at a minimum, have a URL that can also ‘seed’ a search in this way.

The interactive suggestions-part of OpenSearch is an extension that was suggested after the original spec was drawn up, and is, primarily, a convenient way that a search service can help users narrow down their search without having to first run multiple searches.  For example, a lookup into a known keywords list, a list of popular searches, both of these, or more besides.

Our search suggestions service, for example, responds with actual job results which (in IE8 only, unfortunately) take you directly to the job page.

Before you start

Assuming you do have a URL on your site that you can use to launch your search – you should then take a look at the OpenSearch spec, which is the standard around which your suggestions service is going to be built, and which is one of the easier to understand standards.  In particular, you should review the OpenSearch Suggestions Extension specification, which details the URL format that your service should ideally support and format of the data that you should be returning.  One thing to note here is that the specification describes the response data you should return in terms of JSON-formatted objects; however this is not your only option.  Microsoft have adapted and extended the OpenSearch Suggestions Extension into XML format for use with Internet Explorer 8, and with this you have additional capabilities in IE8, such as separators and images in your suggestions list:

In the above screenshot, you can see the top result has an associated image, and the search service is also using a separator to embed a link to an information page about Microsoft’s newly branded Bing search engine.

When it comes to the choice between the JSON or XML suggestions format, you don’t necessarily have to chose either/or – IE8 supports both, whereas Firefox, for example, only supports the JSON format.  However you can expose two services in one OpenSearch Description Document, and both browsers can hang off the one document quite happily (IE8 will favour the XML service when faced with both).

Project Setup

We use Visual Studio 2008 here, so these steps are based on you using that environment.

• Create a new C# Asp.Net Web Application
• Leave the project’s web properties so that it runs from the Visual Studio Development Web Server – it’s considerably easier to do testing with this - however you will want to manually specify the port so that later on your code knows the base location of the URLs to be generated.
• Now add a new WCF service – call it ‘OpenSearch.svc’ (Hint: the ‘WCF Service’ item template is found in the ‘C#’ root folder on the add dialog).
• When you create WCF services, the wizard automatically modifies the web.config to add WS-HTTP and Metadata Exchange (MEX) endpoints – you can remove these from the configuration as we’re going to be working with the zero-configuration WebServiceHostFactory which is great for creating RESTful services for XML and JSON.
• Also, the wizard abstracts the service implementation from it’s operation contracts by creating an interface, IOpenSearch in this case – which you can delete.
• Open the code-behind for the service (OpenSearch.svc.cs) and remove the IOpenSearch interface from the class, and get rid of the ‘DoWork’ method declaration from the class body.
• Finally, you need to add a reference to the System.ServiceModel.Web assembly – this brings in the aforementioned WebServiceHostFactory class, as well as the WebGetAttribute and UriTemplate classes, which we need in order to expose our service on the URLs of our choice.

Results Classes

In order to support both the JSON OpenSearch suggestions format (supported by Firefox) and XML OpenSearch suggestions format (used by Internet Explorer 8), we unfortunately have to split our class definitions between the two uses.  This is because the JSON format is based on a parallel array approach, with the Completions, Descriptions and QueryURLs being specified in three separate arrays, whereas the XML format is object-oriented, with all three pieces of data being specified as child values of a common suggestion element.

These two approaches can’t easily be represented in one set of classes for WCF serialization purposes, so to solve this problem in the way most convenient for us, our primary result classes on the server are going to be the ones that’ll produce the XML results for IE8 (to provide class->entity mapping for the XML), and we’ll create another standalone class for our JSON result format that can be constructed from those.

The first type you need is for each individual suggestion itself:

[DataContract(Namespace = "http://schemas.microsoft.com/Search/2008/suggestions")]
public class Suggestion
{
  [DataMember(EmitDefaultValue = false)]
  public string Url { get; set; }
  [DataMember]
  public string Text { get; set; }
  [DataMember(EmitDefaultValue = false)]
  public string Description { get; set; }
  [DataMember(Name = "Image", EmitDefaultValue = false)]
  public SuggestionImage Image { get; set; }
}

The keen eye will notice that we’ve also got a dependency on the ‘SuggestionImage’ class, which should be specified as follows:

public class SuggestionImage : IXmlSerializable
{
  public string Source { get; set; }
  public string Alt { get; set; }
  public int Height { get; set; }
  public int Width { get; set; }
  public XmlSchema GetSchema()
  {
    return null;
  }
  public void ReadXml(XmlReader reader)
  {
    //we never actually load our results from XML
  }
  public void WriteXml(XmlWriter writer)
  {
    writer.WriteAttributeString("source", Source);
    writer.WriteAttributeString("alt", Alt);
    if (string.IsNullOrEmpty(Height.ToString()) == false)
      writer.WriteAttributeString("height", Height.ToString());
    if (string.IsNullOrEmpty(Width.ToString()) == false)
      writer.WriteAttributeString("width", Width.ToString());
  }
}

This class might seem a little odd – since it doesn’t use any DataContract or DataMember attribute declarations.  The reason for this is because the XML OpenSearch format requires that the image data is specified in XML attributes; this requirement is initially a total show-stopper, because the WCF DataContractSerializer doesn’t natively support XML attribute serialization via its attributes.  However, you can take over the XML serialization yourself and write attribute strings for a given type if you simply implement the IXmlSerializable interface.

Now, we need to define the outer types in which we’re going to place our suggestions.  If you take a look at the aforementioned XML suggestions format, you’ll see that a root-node is expected, called ‘SearchSuggestion’, inside which we have an element called ‘Query’, and then one called ‘Section’, inside which each of our suggestion items should be written.  In order to achieve the latter correctly, we have to use the CollectionDataContract attribute, which allows us to define the inner element that’ll go within the <Section/> element, and give our list the name ‘Section’ at the same time:

[CollectionDataContract(Name = "Section", 
   Namespace = "http://schemas.microsoft.com/Search/2008/suggestions", 
   ItemName = "Item")]
public class SuggestionsSection : List<Suggestion>
{
}

Note the use of the List<> generic here.

And then, finally, we define our outermost result, that our search suggestions service is actually going to return to our callers:

[DataContract(Name = "SearchSuggestion",
  Namespace = "http://schemas.microsoft.com/Search/2008/suggestions")]
public class Suggestions
{
  [DataMember]
  public string Query { get; set; }
  [DataMember]
  public SuggestionsSection Section { get; set; }
}

One of the things that might be confusing about all our [DataContract] classes here is that we repeatedly specify the namespace.  We have to do this, otherwise the DataContractSerializer will, (un)helpfully take our class’ namespace and use that.  Being XML, the namespace is very important, and IE8 will refuse to parse the XML results you throw back if there are errant namespaces in your returned data.  By specifying the same namespace on every class, the DataContractSerializer will only output the namespace on the root node, as it will be satisfied that no namespace changes occur as it drills down through the data that it serializes.

We don’t have to worry about this on our SuggestionImage class, because it is our responsibility to write the namespace attribute out - because we’ve implemented IXmlSerializable – as a result, we simply don’t bother.

Adding support for the Separator element

You will notice that we’re not supporting the <Separator /> element of the XML format - this oversight is for brevity in this article.  If you wanted to support this you would need to introduce a common interface, ‘ISuggestionItem’, which the Suggestion object could implement.  Then you’d introduce another class, called ‘SuggestionSeparator’, which also implements the same interface.  Note here that to support titles on your Separators you’d also need to implement IXmlSerializable again, because the title is expected to be an attribute and, as we saw with the SuggestionImage class, we have to take over serialization when we want to do this.

Finally, you’d change your declaration of the SuggestionsSection class so that it inherits from List<ISuggestionItem>, and add two [KnownType] attribute declarations to that class so as to support Suggestion and SuggestionSeparator.

Moving on to the JSON Result object

As previously mentioned – and as is outlined in the aforementioned specification – the JSON results format is, curiously, not a structure, but an array of arrays.  This poses an interesting problem for the DataContractSerializer, because it’s JSON capabilities, provided by the DataContractJsonSerializer, are geared to produce JSON structures with fields mapped to the properties of the .Net type being serialized.  The answer?  To have your JSON object masquerade as a JSON array, and to add what you would normally think of as fields to that array on construction:

[KnownType(typeof(string))]
[KnownType(typeof(string[]))]
public class JSONSuggestions : List<object>
{
  private string Query;
  private string[] Completions;
  private string[] Descriptions;
  private string[] QueryURLs;
  public JSONSuggestions() { /*default Ctor required by WCF*/ }
  public JSONSuggestions(Suggestions source)
  {
    //build the individual members
    Query = source.Query;
    Completions = new string[source.Section.Count];
    Descriptions = new string[source.Section.Count];
    QueryURLs = new string[source.Section.Count];
    int current = 0;
    foreach (var result in source.Section)
    {
      Completions[current] = result.Text;
      QueryURLs[current] = result.Url;
      Descriptions[current] = result.Description;
      ++current;
    }
    //then add them to the inner list of objects in the 
    //order that the JSON suggestions format decrees.
    this.Add(Query);
    this.Add(Completions);
    this.Add(Descriptions);
    this.Add(QueryURLs);
  }
}

Note that we’re not using the DataContract/DataMember attributes on this class – instead we’re simply leveraging WCF’s POCO support introduced in .Net 3.5 SP1.  The solution presented by this class is quite subtle, and worthy of study.  First, the class inherits from List<object> – this is because we need to be able to serialize both a string (for the ‘query’ data) and arrays of strings (for the completions, descriptions and URLs data).  As a result, we also have to lend the DataContractSerializer a helping hand by using the KnownType attribute, so that it knows in advance that the list could contain either of these two types.

Our only constructor takes in our Suggestions object that we’ve prototyped in the previous section, and simply leeches all the data out of it into temporary holding variables.  Finally, these holding variables are added to the list in the order that the OpenSearch suggestions format specifies.  When this object is serialized by the JSON serializer, it will be written out as an array, because the outermost type is a collection type.

The Query Page and the Search Service

Now, we’re not going to write an actual search service here – what you’ll be left with is the skeleton code in which you can easily plug your search engine code.  We’re going to focus instead on the sprint finish to getting something returned to your browser.  We also need a few pages in our website project:

• A search page that accepts a query string search parameter, as explained in the opening section
• A few additional pages that we can present as actual search results

Our demo service is going to be query-string agnostic; so, although it will accept a query string, the results it returns are not going to be influenced by that input string – that’s for you to do.  The final step after this is to craft our OpenSearch description document so that web browsers can pick up our search provider.

So, quickly:

• Add three content pages to your web project – we’re going to call ours ‘Result1.aspx’, ‘Result2.aspx’ and ’Result3.aspx’.  Put some basic content on there that you can use to discriminate the three from each other.
• Add another page, and call it ‘Search.aspx’.  Modify the code-behind as follows:

public partial class Search : System.Web.UI.Page
{
  private string _searchPhrase;
  public string SearchPhrase
  {
    get
    {
      if(_searchPhrase == null)
      {
        if (Request.QueryString.Count == 0 ||
          (_searchPhrase = Request.QueryString["q"]) == null)
          _searchPhrase = "";
      }
      return _searchPhrase;
    }
  }
}

Now, open the mark-up for the page, and replace the body with:

<body runat="server">
  <div>
    <h1>
      You searched for '<%= SearchPhrase %>'</h1>
  </div>
  <div>
    Results:</div>
  <div>
    <ul>
      <li>
        <h2>
          <a href="Result1.aspx">Result 1</a>
        </h2>
        The first result page
      </li>
      <li>
        <h2>
          <a href="Result2.aspx">Result 2</a>
        </h2>
        The second result page
      </li>
      <li>
        <h2>
          <a href="Result3.aspx">Result 3</a>
        </h2>
        The third result page
      </li>
    </ul>
  </div>
</body>

With all that in place, it’s now time to go back to our OpenSearch.svc and expose our XML and JSON endpoints.

The first step is to open the .svc mark-up itself and add the highlighted code to it, to enable the WebServiceHostFactory:

<%@ ServiceHost Language="C#" Debug="true" Service="OpenSearchDemo.OpenSearch" CodeBehind="OpenSearch.svc.cs"
Factory="System.ServiceModel.Activation.WebServiceHostFactory" 
%>

Now we open the code-behind file and write our two service methods – one for XML and one for JSON:

[ServiceContract()]
public class OpenSearch
{
  [OperationContract]
  [WebGet(UriTemplate = "suggestions?q={query}",
    RequestFormat = WebMessageFormat.Xml,
    ResponseFormat = WebMessageFormat.Xml)]
  public Suggestions GetSuggestions(string query)
  {
    //TODO: replace with your real search engine code.
    Suggestions toReturn = new Suggestions() 
      { Query = query, Section = new SuggestionsSection() };
    //for brevity, we've fixed our dev web server's port so that we know exactly where
    //these URLs need to point.  In reality you'll either use auto-discovery by taking
    //a look at the WCF WebOperationContext, or via a configuration file.
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 1 Description",
      Text = "Result 1",
      Url = "http://localhost:37082/Result1.aspx"
    });
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 2 Description",
      Text = "Result 2",
      Url = "http://localhost:37082/Result2.aspx"
    });
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 3 Description",
      Text = "Result 3",
      Url = "http://localhost:37082/Result3.aspx"
    });
    return toReturn;
  }
  [OperationContract]
  [WebGet(UriTemplate = "json/suggestions?q={query}",
    RequestFormat = WebMessageFormat.Json,
    ResponseFormat = WebMessageFormat.Json)]
  public JSONSuggestions GetJSONSuggestions(string query)
  {
    return new JSONSuggestions(GetSuggestions(query));
  }
}

Note how the implementation of the JSON service is simply a piggy-back off the XML one.  When you’re writing XML and JSON services in this way, typically all the code will either go in one half of the implementation, or in another common block of code that both can call.  The contents of the WebGet attribute are very important at this point – and you should familiarise yourself with RESTful programming in WCF if you’re going to be writing more of these kinds of services.

NOTE: In the downloadable source code, we’ve included the ‘Image’ property for each of these results, so you can see the attribute-based image data in action as well.

Testing the services

At this point you should be able to build and run without any errors.  Once done, you should be able to open the url http://localhost[:devwebport]/suggestions.svc?q=Hello+World and, although you might not actually see any output in the browser, if you open the source for the current page you’ll see the XML response in the format the IE8 will be expecting.

Also, if you instead open http://localhost[:devwebport]/json/suggestions.svc?q=Hello+World, you should be prompted to initiate a file download which, when you open it in notepad, will be a JSON array of arrays as per the actual OpenSearch suggestions spec.

If in doubt, use Fiddler2 to debug the web traffic.  Note that you cannot get other people to call this yet, if you’re using the development web server that is, because it only responds to requests from the local machine.

Writing the OpenSearch Description document

Again, we’re not giving a full description of the OpenSearch spec here – everything you need to do is available from the OpenSearch site, we’re just giving you the XML you need to get up and running.  Add a new XML file to your web project, call it OpenSearch.xml, paste this in after the <?xml> directive:

<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" 
                       xmlns:moz="http://www.mozilla.org/2006/browser/search">
  <ShortName>JSLabs OpenSearchDemo</ShortName>
  <Description>Expose OpenSearch on your Website, with suggestions!</Description>
  <InputEncoding>UTF-8</InputEncoding>
  <!-- used by all opensearch compatible browsers -->
  <Url type="text/html"
       template="http://localhost:37082/search.aspx?q={searchTerms}"
       method="Get"/>
  <!-- suggestions - used by IE8 only -->
  <Url type="application/x-suggestions+xml"
       template="http://localhost:37082/opensearch.svc/suggestions?q={searchTerms}"
       method="Get" />
  <!-- suggestions - used by Firefox -->
  <Url type="application/x-suggestions+json"
       template="http://localhost:37082/opensearch.svc/json/suggestions?q={searchTerms}"
       method="Get" />
  <!-- best practise to include a link to self -->
  <Url type="application/opensearchdescription+xml" 
       rel="self" 
       template="http://lostlhost:37082/opensearch.xml" />
  <!-- not required, but another extension supported by Firefox -->
  <moz:SearchForm>http://localhost:37082</moz:SearchForm>
</OpenSearchDescription>

Final Step – Wiring the OpenSearch document up for the browser

As this excellent article on MSDN describes – there are a few ways to promote your OpenSearch provider.  One is auto-discovery, which is the recommended standards-based way to do it, and is supported by both IE7/8 and Mozilla Firefox 2/3.  To enable auto-discovery for our demo site, open up your Search.aspx page, and add this to the <head> region:

<link title="JSLabs OpenSearch Demo Provider" rel="search"
   type="application/opensearchdescription+xml"
   href="http://localhost:37082/opensearch.xml" />

Save the mark-up – and now, open IE 7/8.  Browse to /Search.aspx (no query string required) – and you’ll see the quick search box change colour:

Click on the down-arrow and expand the ‘Add Search Providers’ pop-out menu.

Confirm the dialog box that comes up (in IE8 make sure that the ‘Use suggestions from this provider’ box is ticked).  Once done, expand the list again, and select the newly added ‘JSLabs OpenSearch Demo’ provider (there’ll be two very similar, it’s the one with the magnifying glass) and start typing, you should see your three suggestions appear:

If you select either of the three results, notice that they take you straight through to the Result[1-3].aspx pages, rather than simply launch the search page again with that suggestion (this is how our Job Search suggestions end up taking you straight through to the job page).

For Firefox, the process is nigh-on identical, browse to the page, the search box will highlight and you can select the provide from a dropdown list:

Again, you’ll be asked to confirm whether you want to use suggestions from this provider, which you do.

At this point you can immediately start typing in the quick-search box, and our suggestions will appear:

This is where Firefox is a bit of a let-down, however (and why our job search provider does not provide suggestions in Firefox), if you select any of these results, you’ll notice that instead of browsing directly to the URL we’re passed back in the results (like IE8), Firefox simply launches the search page with the string ‘Result 1’, ‘Result 2’ or ‘Result 3’ (it’s whichever Completion string is selected).  This is a bug in the Mozilla browser, and was raised quite a while ago – we await to see if it’ll be fixed.

The other way to get your provider added to a browser is to use the non-standard AddSearchProvider method of the window.external property – and have a hyperlink or button that users can click to add the provider manually to the browser:

<a href="#"
   onclick="window.external.AddSearchProvider('http://localhost:37082/opensearch.xml')"
   >Add JSLabs OpenSearch Demo provider</a>

This is the technique that we use for our Web Browser OpenSearch Provider, primarily because we are hosting our provider on a different sub-domain to the main site.

As is mentioned in the launch article for the Web Browser OpenSearch Provider, Chrome and Safari do not support OpenSearch.  Opera 9 should, according to the documentation, support the auto-discovery method of supplying OpenSearch to the web browser.  Support doesn’t just have to stop at the browser, though - remember that there are websites out there which could leverage your OpenSearch capability (not the suggestions, just the basic search url) in order to better your content to a wider audience.  All in all, installing OpenSearch into your website is a worthwhile effort – and with WCF, supporting the suggestions extension is surprisingly simple.

Download the source code for this article now!