JobServe Labs - Articles in Technical

We help Microsoft fix Visual 2010 Help Manager

2010-02-10T11:33:56Z

In a previous post we described a bug in the VS2010 Beta 2 MSDN product which prevents you from getting hold of help collections etc when you sit behind an Http proxy.

In that post we provided a solution which involved leveraging Fiddler’s ability to chain together proxy authentication on an application’s behalf.

We also spotted this post on MSDN Forums and posted an answer indicating a possible solution for the person who opened the issue.

As you’ll notice from the bottom post – a request was made by an employee of Microsoft to see if we could help them test a fix that they had in mind. We did; the fix didn’t work completely, but we were then able to diagnose exactly why the fix didn’t work and have subsequently discovered how to make the Beta 2 MSDN 2010 Help Library Manager without having to use Fiddler. Based on our investigation, the team at Microsoft are now working on a fix – and might already have applied it in the Release Candidate of VS2010 – which released in the last week.

Use at your own risk!

Before we go into the details, it must be stressed that this fix is not offered officially by Microsoft and therefore carries no guarantees from them. We also make no guarantees about this fix either – it will involve changing registry settings and if this turns out to break anything else on your machine, then we will accept no responsibility for that. It’s your choice to follow our directions here!

Step 1

Modify the HelpLibManager.exe.config file found in Program Files [(x86) if on a 64 bit box]\Microsoft Help\v3.0 and add the following xml inside the <configuration /> node:

<system.net>
  <defaultProxy useDefaultCredentials="true" >
  </defaultProxy>
</system.net>

Save the config file.

On its own, this fix allows the help manager to ‘see’ the collections that are available for download – but it does not actually fix the problem of the download itself.

Step 2

This is the messy bit.

Open up the Registry Editor and make sure that registry value HKLM\Software\Microsoft\Windows\CurrentVersion\BITS\UseLMCompat is set to zero (the reason why is explained in a moment). If you need to change it, you should make a note of the previous value – or better yet, create a System Restore Point.

Restart the Background Intelligent Transfer Service, and then start up the help library manager. It should all now work without any problems – assuming you have internet connectivity at all; and that your user name and password allow you through the proxy in the first place :)

Why does it work?

We ‘discovered’ that Help Library Manager uses BITS (Background Intelligent Transfer Service) to initiate and manage the actual downloads of the content. Presumably this is because the files are typically rather large. This runs in a separate process and is communicated with over a COM interface, therefore the configuration file fix has no effect on that.

After some poking and prodding, we discovered that when a BITS transfer is initiated, the caller has the opportunity to specify the credentials to be used for the network connection. Working on the basis that the code in the Help Library can’t be setting these credentials (otherwise it would work), we looked for a way to get BITS to use default credentials. The article from the last link above provides the answer to that – which yields the registry hack.

Be aware that modifying this registry value affects all software that uses BITS – and therefore it’s just possible that other things might stop working as a result.

As we mentioned earlier – we don’t know if the bug has been redressed in the RC of Visual Studio 2010 – our findings, including this fix, were submitted to Microsoft on the 27th January – so it’s possible it was too late.

Hopefully, however, it’ll be fixed by the final release date!

That’s it for now, but stay tuned: We have some very exciting news on the way in the next couple of weeks; but we’re not allowed to talk about it yet ;)

Link Tag in Content Control not created as HtmlLink

2009-11-25T12:31:46Z

The Asp.Net System.Web.UI.WebControls.HtmlHead class is clever, especially when it comes to stylesheet <link> tags in master pages, since it magically rebases them (using Control.ResolveClientUrl) to be relative to the path of the page that is executing instead of the master page. This is especially necessary when a content page is declared in a different folder to its master page, because any CSS references are unlikely to work except by pure fluke.

With the advent of Visual Studio 2008, it has become common practise to have a content placeholder in a master page’s head section, which allows the page designer to add extra meta data, script references and CSS links on a content page as well. To be fair, this process started almost as soon as master pages came into being – except Visual Studio 2005 did not have design-time support for it, a deficit that was much bemoaned by K. Scott Allen at the time.

This website uses that ability so that our master page contains our common CSS references, but some of our content pages then have their own script references and styles. In order to get around the url-rebasing issue, we use a <base /> tag at the top of every page, so that we can use root-relative links for our urls, css and javascript references, then the browser will happily get them from the correct place. A recent article detailing a fix for a small bug with the Atom Feed link on one of our pages details this process a bit more.

Now we’re working on a new site, that will be using the Asp.Net MVC 1.0 Framework (and possibly 2.0 if it goes live before we’re finished). This brings even more potential problems with relative URL links, since every page URL is actually a folder. Although the <base/> tag is a solution, it’s not perfect since you then have to define a way to actually format out that base url. Many people use configuration, but it royally sucks for this, as it means that every time you move the site to a different machine (different developer boxes, QA box, staging box etc) or from a virtual directory to a website, you have to update the configuration.

Magic URL rebasing – the good

As mentioned in the introduction, any <link> tag that is placed directly inside the Head control of an Asp.Net page will be turned into an HtmlLink control – the magic for this is done in the HtmlHeadBuilder’s GetChildControlType method:

public override Type GetChildControlType(string tagName, IDictionary attribs)
{
    if (string.Equals(tagName, "title", StringComparison.OrdinalIgnoreCase))
    {
        return typeof(HtmlTitle);
    }
    if (string.Equals(tagName, "link", StringComparison.OrdinalIgnoreCase))
    {
        return typeof(HtmlLink);
    }
    if (string.Equals(tagName, "meta", StringComparison.OrdinalIgnoreCase))
    {
        return typeof(HtmlMeta);
    }
    return null;
}

Very simple indeed. As you can see, in order to get an HtmlLink created in place of a standard <link> tag within the head, you have to do, well, nothing at all (not even make it runat=server). When the HtmlLink control renders it’s output, instead of dumbly rendering the attributes (i.e. the href, rel etc) it does this:

protected override void RenderAttributes(HtmlTextWriter writer)
{
    if (!string.IsNullOrEmpty(this.Href))
    {
        base.Attributes["href"] = base.ResolveClientUrl(this.Href);
    }
    base.RenderAttributes(writer);
}

The ResolveClientUrl method is used to calculate the actual relative path to a resource, given a template-relative path (e.g. “~/Pages/page.aspx”, “~/Controls/control.ascx”, a master page virtual path or whatever) – which is retrieved from the first control up the parent-child hierarchy of the page that has one of these paths.

Magic URL rebasing – the bad

<script> tags don’t get rebased - ever

There is one tag in particular that is missing from the logic in the above code - “script”. Yes, that’s right, whilst <link> tags will be magically rebased, <script> references will not – which is indeed a stark omission from the Asp.Net framework. From the master page point of view, this means that you cannot really have <script> references in your master page’s header, unless all content pages will be in the same relative location from the target javascript file.

This problem can be solved in a couple of different ways. The first would be to pre-process the page content before it finally gets rendered, searching for any <script> references, parsing them manually, using Control.ResolveClientUrl to get the target script reference and then writing the fixed URL back to the content (indeed, this approach has been taken with a fix adopted by one frustrated Asp.Net user; see later).

However, the other approach is to write a simple server-side control to do it:

/// <summary>
/// Mimics the HtmlLink control - i.e. calling ResolveClientUrl at render time - 
/// except it's done on the 'src' attribute value.  Since HtmlLink writes the resolved
/// path back to it's Href member, this one does the same.
/// </summary>
public class HtmlScript : HtmlGenericControl
{
  public HtmlScript() : base("script")
  {
    
  }
  public HtmlScript(string tag) : base(tag)
  {
  }
  public string Src
  {
    get
    {
      return this.Attributes["src"];
    }
    set
    {
      this.Attributes["src"] = value;
    }
  }
  protected override void RenderAttributes(HtmlTextWriter writer)
  {
    Src = ResolveClientUrl(Src);
    base.RenderAttributes(writer);
  }
}

With this in place, you simply use the <#@ Register directive, or register a site-wide prefix via the web.config, to bring in a tag-prefix (e.g. “asp2”) for the control and then, in your ASPX source you change your <script> to:

<asp2:HtmlScript src="../myscript.js" type="text/javascript" language="javascript">
</asp2:HtmlScript>

Okay, so this isn’t a perfect solution – a site-wide refactoring like this could take some time! However, the best it yet to come…

Content controls in the head – even <link>s don’t get rebased

This is possibly the biggest problem, when you use a content control to inject extra links, references or whatever into a page head section from a content form, the magic CSS <link> rebasing no longer works. Why? The clue is in the above code snippet from HtmlHeadBuilder.GetChildControlType. Consider the following master page, and content page:

Master Page <head> section:

<head runat="server">
  <title>
    <asp:ContentPlaceHolder ID="TitleContent" runat="server" />
  </title>
  <link href="../../Content/Site.css" rel="stylesheet" type="text/css" />
  <asp:ContentPlaceHolder ID="CustomHeadContent" runat="server" />
</head>

Content Page Content Placeholder:

<asp:Content ID="customHead" ContentPlaceHolderID="CustomHeadContent" runat="server">
    <link href="../../Content/Site.css" rel="stylesheet" type="text/css" />
</asp:Content>

Okay, so in this example, both pages are referencing the same stylesheet, so it’s slightly contrived. However, assuming that the css files are in ~/Content/, the master page is in the ~/pages/masters/ folder, and the content page is in the ~/pages/content/ folder, this will work fine when it is run.

However, if we’re using something like MVC (or using the HttpRequest.PathInfo property like we do for our articles, projects etc) in order to beautify your urls, then we will run into problems. For example, if the content page in question is actually displayed in the browser with the url “/home/”, then the CSS link rendered from the head on the master page will correctly come out as “../content/site.css”, however the CSS link from the page’s content control will be left untouched (indeed it will be rendered as a literal), and will generate a 404 when it hits the browser.

This is because in order for the <link> markup in the content control to be turned into an HtmlLink, instead of being left as raw markup, the control builder that parses the <asp:Content> control (ContentBuilderInternal – no MSDN link because it’s understandably undocumented!) would have to have the same logic in it that the head section does – since the ASP.Net build engine sees the <link> tag as a child of the Content Control, not the HtmlHead that the content eventually gets merged into.

This is clearly a bug, or at the very least an oversight, in the Asp.Net framework – and has been reported on MS Connect by Nathanael Jones, who came up with the aforementioned page-parsing solution on his blog. However, this solution does not fix design-time problems (since the VS designer doesn’t run all of the page code when rendering it in the designer), and it is also quite a weighty solution which will inevitably slow down page rendering; our sites get tens of thousands of hits per day, and adding extra load on the servers is just not an option.

Our solution

Our solution fixes rebasing of all css <link> tags in the head of content pages, and it also automatically picks up and rebases all <script> references as well. You’ll need the HtmlScript control, whose source code was presented earlier on; then add this code:

/// <summary>
/// It's a placeholder, really - it simply tells the framework to use the RebasingContainerBuilder to 
/// create the controls that will emit the html.  Then, the designer is set to ControlDesigner - 
/// the same one that the ContentControl uses.
/// That ensures design-time support for the CSS and Javascript links on content pages.
/// </summary>
[ControlBuilder(typeof(RebasingContainerBuilder)), 
  Designer("System.Web.UI.Design.ControlDesigner, System.Design, " + 
  "Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"), 
  ConstructorNeedsTag(false)]
public class RebasingContainer : HtmlGenericControl
{
  public RebasingContainer()
  {
  }
  protected override void RenderBeginTag(System.Web.UI.HtmlTextWriter writer)
  {  /*doesn't render it's own tag*/ }
  protected override void RenderEndTag(System.Web.UI.HtmlTextWriter writer)
  {/*doesn't render it's own tag*/}
}
/// <summary>
/// This class is almost identical to HtmlHeadBuilder.  It's purpose to ensure that embedded
/// links and script tags are always rebased for the client during rendering.
/// The HtmlHead control does this for CSS link tags only, and the functionality is lost 
/// on content controls on content forms.
/// This builder is used by the <see cref="RebasingContainer"/> control
/// </summary>
public class RebasingContainerBuilder : ControlBuilder
{
  public override bool AllowWhitespaceLiterals()
  {
    return false;
  }
  public override Type GetChildControlType(string tagName, System.Collections.IDictionary attribs)
  {
    /* copied this code from System.Web.UI.HtmlControls.HtmlHeadBuilder */
    if (string.Equals(tagName, "link", StringComparison.OrdinalIgnoreCase))
    {
      return typeof(HtmlLink);
    }
    if (string.Equals(tagName, "script", StringComparison.OrdinalIgnoreCase) 
        && attribs.Contains("src"))
    {
      //only rebase script tags that have a src attribute!
      return typeof(HtmlScript);
    }
    return null;
  }
}

If you’ve placed this code inside the same assembly and namespace as the aforementioned HtmlScript class, then register a tag-prefix for that assembly/namespace (as before – the best place is in the web.config so it’s available on all pages) and you can start using it.

In order for content page script and css references to get rebased you simply change this:

<asp:Content ID="indexCustomHead" ContentPlaceHolderID="CustomHeadContent" runat="server">
  <link href="/Content/TestSite.css" rel="stylesheet" type="text/css" />
  <script src="/Content/script.js" type="text/javascript" language="javascript" />
</asp:Content>

To this:

<asp:Content ID="indexCustomHead" ContentPlaceHolderID="CustomHeadContent" runat="server">
  <asp2:RebasingContainer runat="server">
    <link href="/Content/TestSite.css" rel="stylesheet" type="text/css" />
    <script src="/Content/script.js" type="text/javascript" language="javascript" />
  </asp2:RebasingContainer>
</asp:Content>

Note that the asp2: prefix here depends on what prefix you actually register with Asp.Net. At run-time and design time, the RebasingContainer control doesn’t render it’s own tags – it simply acts as a placeholder to instruct Asp.Net to delegate to RebasingContainerBuilder which, in turn, contains the logic to map <link> and <script> tags to the server side controls that will perform the magic rebasing – just like the HtmlHead control.

Similarly, you can use the rebasing container to rebase <script> tags in the master page’s <head> content (or indeed any page’s head content) – simply move all your existing script tags into a new RebasingContainer, and it will work (don’t worry about the <link> tags – since the HtmlHeadBuilder will take care of those for you).

When you first do this, the designer might get upset and tell you that it doesn’t recognise the RebasingContainer – simply do a full build, and it will update. Now flip to design view on one of your content pages that previously didn’t work and smile at the fact that any extra stylesheets are now working in design view! Notice also that inline javascript will pick up intellisense from any script references that have had to be rebased in order to be picked up.

A better, but near-impossible, solution

One solution that has been suggested by the community is to sub-class HtmlHead and it’s HtmlHeadBuilder in order to force <link> tags to be always mapped to HtmlLink controls. There are numerous problems with this, the first and most crucial is that HtmlHead is a sealed class, so that’s impossible. Even it were possible, however, it wouldn’t really help because, as explained earlier, the HtmlHeadBuilder is only used to parse the direct content of a runat=”server” <head> control, and would never actually get a chance to process the ASPX markup for a page’s Content control.

Another solution that we researched was to do a similar thing, but for the Content control; that is, sub-class the Content control and it’s associated control builder class, making it <link> and <script> aware. This would have meant simply changing the control tag used on a content page to inject content in the page’s header. The big problem with this, however, is that the Content control uses an control builder that’s internal to Asp.Net: ContentBuilderInternal, which by definition cannot be subclassed. It’s possible to call into internal classes and methods (using dynamic code generation – e.g. via Linq Expression Trees or DynamicMethod), but we didn’t have the time to build a proxy and all the code generation logic.

In the meantime, we hope that this explains the problem and provides you with an easy-to-use solution!

VS2010 Beta 2 – Fatal Error Occurred installing Help

2009-11-13T17:14:35Z

Update (10th Feb 2010) – a more thorough investigation of this issue has been carried out by us (after being asked to help Microsoft fix the problem) and a new fix is detailed here.

We’ve been playing around with the beta 2 of Visual Studio 2010 and really like the Web Install idea - although the actual VS2010 part of the download appears to be throttled at around 4Kb, which is a little frustrating!

At the very end of the installation you get to install the new MSDN Help 3 Collections, choosing online or local help as your primary source. If you’re running behind a web proxy that requires authentication you will see a “fatal error occurred” message pop up, and that’s it – there’s nothing you can do to get help on to your machine (except to get your SysAdmin to allow you out of the building without going through the proxy).

However, we have found a rather cheeky solution to this problem – the rather marvellous Fiddler Http Web Debugging Proxy.

To get it to work

Scenario 1: All you should have to do is download Fiddler, install it and run it, then hit the ‘Get Content Online’ link again. Eventually you’ll see all the content offerings, select the ones that you want and then hit the Update button.

You should see web requests in Fiddler going through to packages.mtps.microsoft.com once the transfer starts. Don’t worry if they appear to take for ages to complete – you’re watching a multi-part download, and nothing really interesting is going to happen until it’s finished!

Scenario 2: If this doesn’t work – then it’s possible you might have to force Fiddler to remember your credentials for your proxy. The best way to do this is to load a page in IE, and you should see a proxy login box appear. Enter the correct credentials for your proxy and check that the web page then loads correctly.

Now try to get the help content again – it should work.

Why does Fiddler make it work?

Here’s our theory: Obviously the Help installer client is not proxy-aware. So it attempts to make a connection to the package download location, but hits the system proxy. When this happens, the proxy returns a request for credentials, but the code isn’t written to send them over, and so it simply crashes.

The reason why fiddler makes it work is because if it receives a Proxy Authentication request (Http Status Code 407) from the upstream proxy, it responds with the identity of the user running Fiddler (Scenario 1), or a token that has previously been known to work for the current session (Scenario 2).

If that isn’t good enough, it then pushes the 407 back to the client. Now we already know that the Help Installer itself can’t cope with this, so in our Scenario 2 we use Internet Explorer to cheat, since it displays a login box when credentials are required.

IE then sends those credentials to Fiddler, which in turn are then sent to your proxy – if the proxy then lets the request out, Fiddler will cache that authentication token to be used with ALL future requests (not just those coming from IE). So when the Help Installer then makes another request, it signs that request as authenticated even though the Help Installer didn’t know it needed to.

Remote debugging code on Windows 2003 from Vista or Windows 7/2008 R2

2009-11-05T12:30:10Z

We’re in the process of upgrading our development environment from Windows Server 2003 x64 to Windows Server 2008 R2. Apart from the desktop experience being far superior, we’re also moving our web servers up to IIS 7.5, therefore we want all our web developers to be running the same. Windows 7 is of course an option for this (and it works extremely well as a web development environment – unlike Windows XP Pro), however it doesn’t offer Hyper-V, which is a must for many developers who want to run their own database servers and mock web servers.

Inevitably, however, we still have to support our existing codebase - all of which runs on Windows Server 2003. Occasionally, therefore, we need to debug something that’s running live because a problem arises that is not easy to reproduce in a model office environment. This typically means installing the Visual Studio Remote Debugger on the server that’s running the code and then attaching to it from the local Visual Studio. Many people would say that this is bad practise; in our opinion bad practise is to go the whole hog and install Visual Studio on the server machine! Remote debugging, on the other hand, is a reality of working as a developer in a technology-driven business.

The problem

Today we had to do just this, but when we tried to connect to the remote machine we got the error message "a security package specific error occurred". If you do a web search for this error, you’re likely to see this link from MSDN forums, where the user eventually solved the issue by creating a local user on both the target and local machines, and ran both VS2008 and the Remote Debugger as that account.

The solution presented is to:

Create two, administrator-level, identical local users on both the debugger machine and the debuggee machine
Run the msvsmon.exe application (the Remote Debugger binary) as that local user on the debuggee
Startup Visual Studio on the debugger machine as that local user
Attach to the debuggee

We weren’t happy with this resolution, because it points to a more fundamental problem – if Remote debugging works between two Windows 2008 R2 or Windows 7 machines, using the same VS binaries, why wouldn’t it work when talking to a Windows Server 2003 machine? Equally, we don’t want to have to go around creating local users on developer machines and servers just to get remote debugging to work. When you’ve reached this stage, it typically means that you’ve got an error you can’t reproduce any other way and you just need to ‘get it fixed quick’. Also, in a TFS environment, the Visual Studio instance in which you’re doing the debugging will not be able to connect to TFS, because the local user that you’re using is unlikely to have access to source control etc.

The root cause of this problem turns out to be slightly more subtle than the user on the aforementioned MSDN forum post assumed. It is indeed authentication that is failing, but not the authentication of the user doing the debugging against the server they wish to debug. In fact, after the remote debugging service receives a request to attach, it then authenticates back to the machine that the request originates from, and it is at this point that authentication is going wrong (we were able to generate the error that is described in that previous link by using a local account for the debugger service on the target machine but without creating it on the development machine).

Even though our service was running as a domain account which has logon rights to the development machine, for some reason the NTLM network logon token for that account that is sent back by the debugging service would appear not to work properly.

By enabling network level NTLM auditing (in Local Security Policy, enable the two settings “Network security: Restrict NTLM: Audit Incoming NTLM traffic” and “Network security: Restrict NTLM: Audit NTLM Authentication in this domain”) we were able to see the logon occur, but then the logon session is destroyed (presumably as a result of the “Security package specific error” that occurs).

A (slightly) easier solution

The good news from this is that we can offer a streamlined solution to the one provided in the above MSDN link, the bad news is that you’ll still need to create the local users:

Create the two users as before on the two machines – however, it only needs to be an administrator on the target machine.
Modify the remote debugger service on the target machine to use the new local user, start the service.
Run Visual Studio 2008 as you would normally (no need to run as the local user), and attach the debugger as normal.

Tell Microsoft!

We have logged a bug on Microsoft Connect so that the VS team can investigate the issue – if you are affected by this issue then please vote for it! At present we do not know if this also affects Visual Studio 2010 (we have a development lab set up and will investigate), if you can verify that it does, then add your experience to that bug so that we can ensure that at least the next version of VS isn’t plagued by this issue, even if it doesn’t get fixed in the current version.

Website update 16th October 2009

2009-10-16T13:57:00+01:00

Firstly, our apologies to some of you over the past few days who’ve had problems accessing our site feed from your browsers’ RSS auto-discovery icon. We’ve been catching some errors at this end where some of you have apparently been trying to access our feed url (http://www.labs.jobserve.com/jslabs.svc/feed) from one of our virtual sub-folders (such as the Articles.aspx sub folder that you’ll see in your browsers address bar on this page).

We’ve modified our master page now so that the feed url specified in the RSS <link /> tag will always be correct – we hope this didn’t stop you from being able to access our site feed!

Why did it happen?

A lot of work has gone into this website at our end to make it easy for us to develop, test and deploy. Part of this is ensuring that the site works as well under an IIS virtual directory (which is what developers usually do) as it does as its own IIS root website (when it’s deployed to our live servers).

Usually you can use the ‘~/’ notation in Asp.Net to work around this problem; however a lot of our pages, although they are ASPX pages, do not use server-side controls and tags. This is to keep the load on the webservers to a minimum, but it also means that we can’t always use virtual paths on our links, images and all the other stuff that makes up a web page.

Throw into the mix that we’re using the HttpRequest.PathInfo property on many of our pages (an idea garnered from Scott Guthrie’s article on URL rewriting for SEO purposes) – which means that a lot of relative paths can no longer be taken for granted.

For example – a <head> link to styles/labs_base.css works fine if the page in the browser is simply Articles.aspx (since that’s in the root of the website), but since all articles are actually served from Articles.aspx/[Article-Title], then your browser will incorrectly make a css request for Articles.aspx/styles/labs_base.css – which, of course, doesn’t exist.

A typical solution this problem would be to change the path in the master page to /styles/labs_base.css – but this would break the site in development, unless we always enforce our developers to run their development site in a website in IIS (using the Cassini isn’t an option because there are a few other tricks being employed by this site which don’t work when running in that). Equally we could make all our <link/>s, <a hrefs et al to be runat=”server” – but that’s making the web server do more work than it really needs to.

How do we fix the problem, then? We use a <base /> tag, which is placed within the <head /> section of every page – this tells your browser that all relative Urls start from a given location – in our case www.labs.jobserve.com – which means that we can then use relative links everywhere for everything. Our code for calculating the base Url is partly taken from the request’s Url, as well as the application path:

public string BaseAddress
{
  get
  {
    //stick "http://" or "https://" in front of this as required
    return string.Format("{0}{1}",HttpContext.Current.Request.Url.Authority, 
      HttpContext.Current.Request.ApplicationPath);
  }
}

This code works reliably regardless of whether or not the site is running in a virtual directory or under its own website; meaning that our development team can develop the site however they want and we can deploy the site however we want in the future.

The original problem with our site feed link from the master page was that it was relative, i.e. jslabs.svc/feed, but unfortunately it appears that some, if not all, browsers, were ignoring the <base /> tag. Now, our feed link is rendered absolute – including the http:// bit, and it should work on every page.

So there you go – if you’re developing an Asp.Net website, then coding around base url of your site at deployment time and during development will be an inevitable problem. You can take a brute force approach, force Website/Virtual Directory configuration to be the same from Development to Live – or you can find a more elegant approach that ultimately makes your code much more portable. Ours is just one solution; there are many more floating around out there.

If you were affected by this – we hope that you now enjoy unfaltering access to our site feed!

Building Labs – Writing an OpenSearch Suggestions provider in C# with WCF

2009-08-17T16:50:53+01:00

This is the second in a series of articles on how we’ve built JobServe Labs and some of the projects that are available. In the first article we described how we’re using Windows Live Writer to manage our content, and how we’ve written an XML RPC backend for it to talk to, using WCF in C#.

This time, we’re going to look at the suggestions service that we did for our Web Browser Search Provider, and how you can set up your own.

Before you start reading, please feel free to download the source code that is built by following this walkthrough – it also shows how to output the images in the suggestions that are supported by IE8.

What is OpenSearch?

Originally, OpenSearch enabled a website to expose to interested consumers (be they web browsers, search engines or anything else) any of the search capabilities that they might have. The notion of extracting this out to a standard meant that a consumer could be automatically compatible with lots of search engines once the code had been written to interpret one.

The search facility of a website typically takes the form of a URL that can accept a URL-encoded search phrase in the query string – for example, many people will be familiar with Google’s query string format, since each search results page is represented by a different URL:

http://www.google.co.uk/search?hl=en&q=jobserve – first page of a Google Search for ‘jobserve’

http://www.google.co.uk/search?q=jobserve&hl=en&start=10&sa=N – second page – note the additional ‘start’ parameter, which tells the search engine which result should come first.

If you’re looking to implement OpenSearch for your website, then you should, at a minimum, have a URL that can also ‘seed’ a search in this way.

The interactive suggestions-part of OpenSearch is an extension that was suggested after the original spec was drawn up, and is, primarily, a convenient way that a search service can help users narrow down their search without having to first run multiple searches. For example, a lookup into a known keywords list, a list of popular searches, both of these, or more besides.

Our search suggestions service, for example, responds with actual job results which (in IE8 only, unfortunately) take you directly to the job page.

Before you start

Assuming you do have a URL on your site that you can use to launch your search – you should then take a look at the OpenSearch spec, which is the standard around which your suggestions service is going to be built, and which is one of the easier to understand standards. In particular, you should review the OpenSearch Suggestions Extension specification, which details the URL format that your service should ideally support and format of the data that you should be returning. One thing to note here is that the specification describes the response data you should return in terms of JSON-formatted objects; however this is not your only option. Microsoft have adapted and extended the OpenSearch Suggestions Extension into XML format for use with Internet Explorer 8, and with this you have additional capabilities in IE8, such as separators and images in your suggestions list:

In the above screenshot, you can see the top result has an associated image, and the search service is also using a separator to embed a link to an information page about Microsoft’s newly branded Bing search engine.

When it comes to the choice between the JSON or XML suggestions format, you don’t necessarily have to chose either/or – IE8 supports both, whereas Firefox, for example, only supports the JSON format. However you can expose two services in one OpenSearch Description Document, and both browsers can hang off the one document quite happily (IE8 will favour the XML service when faced with both).

Project Setup

We use Visual Studio 2008 here, so these steps are based on you using that environment.

Create a new C# Asp.Net Web Application
Leave the project’s web properties so that it runs from the Visual Studio Development Web Server – it’s considerably easier to do testing with this - however you will want to manually specify the port so that later on your code knows the base location of the URLs to be generated.
Now add a new WCF service – call it ‘OpenSearch.svc’ (Hint: the ‘WCF Service’ item template is found in the ‘C#’ root folder on the add dialog).
When you create WCF services, the wizard automatically modifies the web.config to add WS-HTTP and Metadata Exchange (MEX) endpoints – you can remove these from the configuration as we’re going to be working with the zero-configuration WebServiceHostFactory which is great for creating RESTful services for XML and JSON.
Also, the wizard abstracts the service implementation from it’s operation contracts by creating an interface, IOpenSearch in this case – which you can delete.
Open the code-behind for the service (OpenSearch.svc.cs) and remove the IOpenSearch interface from the class, and get rid of the ‘DoWork’ method declaration from the class body.
Finally, you need to add a reference to the System.ServiceModel.Web assembly – this brings in the aforementioned WebServiceHostFactory class, as well as the WebGetAttribute and UriTemplate classes, which we need in order to expose our service on the URLs of our choice.

Results Classes

In order to support both the JSON OpenSearch suggestions format (supported by Firefox) and XML OpenSearch suggestions format (used by Internet Explorer 8), we unfortunately have to split our class definitions between the two uses. This is because the JSON format is based on a parallel array approach, with the Completions, Descriptions and QueryURLs being specified in three separate arrays, whereas the XML format is object-oriented, with all three pieces of data being specified as child values of a common suggestion element.

These two approaches can’t easily be represented in one set of classes for WCF serialization purposes, so to solve this problem in the way most convenient for us, our primary result classes on the server are going to be the ones that’ll produce the XML results for IE8 (to provide class->entity mapping for the XML), and we’ll create another standalone class for our JSON result format that can be constructed from those.

The first type you need is for each individual suggestion itself:

[DataContract(Namespace = "http://schemas.microsoft.com/Search/2008/suggestions")]
public class Suggestion
{
  [DataMember(EmitDefaultValue = false)]
  public string Url { get; set; }
  [DataMember]
  public string Text { get; set; }
  [DataMember(EmitDefaultValue = false)]
  public string Description { get; set; }
  [DataMember(Name = "Image", EmitDefaultValue = false)]
  public SuggestionImage Image { get; set; }
}

The keen eye will notice that we’ve also got a dependency on the ‘SuggestionImage’ class, which should be specified as follows:

public class SuggestionImage : IXmlSerializable
{
  public string Source { get; set; }
  public string Alt { get; set; }
  public int Height { get; set; }
  public int Width { get; set; }
  public XmlSchema GetSchema()
  {
    return null;
  }
  public void ReadXml(XmlReader reader)
  {
    //we never actually load our results from XML
  }
  public void WriteXml(XmlWriter writer)
  {
    writer.WriteAttributeString("source", Source);
    writer.WriteAttributeString("alt", Alt);
    if (string.IsNullOrEmpty(Height.ToString()) == false)
      writer.WriteAttributeString("height", Height.ToString());
    if (string.IsNullOrEmpty(Width.ToString()) == false)
      writer.WriteAttributeString("width", Width.ToString());
  }
}

This class might seem a little odd – since it doesn’t use any DataContract or DataMember attribute declarations. The reason for this is because the XML OpenSearch format requires that the image data is specified in XML attributes; this requirement is initially a total show-stopper, because the WCF DataContractSerializer doesn’t natively support XML attribute serialization via its attributes. However, you can take over the XML serialization yourself and write attribute strings for a given type if you simply implement the IXmlSerializable interface.

Now, we need to define the outer types in which we’re going to place our suggestions. If you take a look at the aforementioned XML suggestions format, you’ll see that a root-node is expected, called ‘SearchSuggestion’, inside which we have an element called ‘Query’, and then one called ‘Section’, inside which each of our suggestion items should be written. In order to achieve the latter correctly, we have to use the CollectionDataContract attribute, which allows us to define the inner element that’ll go within the <Section/> element, and give our list the name ‘Section’ at the same time:

[CollectionDataContract(Name = "Section", 
   Namespace = "http://schemas.microsoft.com/Search/2008/suggestions", 
   ItemName = "Item")]
public class SuggestionsSection : List<Suggestion>
{
}

Note the use of the List<> generic here.

And then, finally, we define our outermost result, that our search suggestions service is actually going to return to our callers:

[DataContract(Name = "SearchSuggestion",
  Namespace = "http://schemas.microsoft.com/Search/2008/suggestions")]
public class Suggestions
{
  [DataMember]
  public string Query { get; set; }
  [DataMember]
  public SuggestionsSection Section { get; set; }
}

One of the things that might be confusing about all our [DataContract] classes here is that we repeatedly specify the namespace. We have to do this, otherwise the DataContractSerializer will, (un)helpfully take our class’ namespace and use that. Being XML, the namespace is very important, and IE8 will refuse to parse the XML results you throw back if there are errant namespaces in your returned data. By specifying the same namespace on every class, the DataContractSerializer will only output the namespace on the root node, as it will be satisfied that no namespace changes occur as it drills down through the data that it serializes.

We don’t have to worry about this on our SuggestionImage class, because it is our responsibility to write the namespace attribute out - because we’ve implemented IXmlSerializable – as a result, we simply don’t bother.

Adding support for the Separator element

You will notice that we’re not supporting the <Separator /> element of the XML format - this oversight is for brevity in this article. If you wanted to support this you would need to introduce a common interface, ‘ISuggestionItem’, which the Suggestion object could implement. Then you’d introduce another class, called ‘SuggestionSeparator’, which also implements the same interface. Note here that to support titles on your Separators you’d also need to implement IXmlSerializable again, because the title is expected to be an attribute and, as we saw with the SuggestionImage class, we have to take over serialization when we want to do this.

Finally, you’d change your declaration of the SuggestionsSection class so that it inherits from List<ISuggestionItem>, and add two [KnownType] attribute declarations to that class so as to support Suggestion and SuggestionSeparator.

Moving on to the JSON Result object

As previously mentioned – and as is outlined in the aforementioned specification – the JSON results format is, curiously, not a structure, but an array of arrays. This poses an interesting problem for the DataContractSerializer, because it’s JSON capabilities, provided by the DataContractJsonSerializer, are geared to produce JSON structures with fields mapped to the properties of the .Net type being serialized. The answer? To have your JSON object masquerade as a JSON array, and to add what you would normally think of as fields to that array on construction:

[KnownType(typeof(string))]
[KnownType(typeof(string[]))]
public class JSONSuggestions : List<object>
{
  private string Query;
  private string[] Completions;
  private string[] Descriptions;
  private string[] QueryURLs;
  public JSONSuggestions() { /*default Ctor required by WCF*/ }
  public JSONSuggestions(Suggestions source)
  {
    //build the individual members
    Query = source.Query;
    Completions = new string[source.Section.Count];
    Descriptions = new string[source.Section.Count];
    QueryURLs = new string[source.Section.Count];
    int current = 0;
    foreach (var result in source.Section)
    {
      Completions[current] = result.Text;
      QueryURLs[current] = result.Url;
      Descriptions[current] = result.Description;
      ++current;
    }
    //then add them to the inner list of objects in the 
    //order that the JSON suggestions format decrees.
    this.Add(Query);
    this.Add(Completions);
    this.Add(Descriptions);
    this.Add(QueryURLs);
  }
}

Note that we’re not using the DataContract/DataMember attributes on this class – instead we’re simply leveraging WCF’s POCO support introduced in .Net 3.5 SP1. The solution presented by this class is quite subtle, and worthy of study. First, the class inherits from List<object> – this is because we need to be able to serialize both a string (for the ‘query’ data) and arrays of strings (for the completions, descriptions and URLs data). As a result, we also have to lend the DataContractSerializer a helping hand by using the KnownType attribute, so that it knows in advance that the list could contain either of these two types.

Our only constructor takes in our Suggestions object that we’ve prototyped in the previous section, and simply leeches all the data out of it into temporary holding variables. Finally, these holding variables are added to the list in the order that the OpenSearch suggestions format specifies. When this object is serialized by the JSON serializer, it will be written out as an array, because the outermost type is a collection type.

The Query Page and the Search Service

Now, we’re not going to write an actual search service here – what you’ll be left with is the skeleton code in which you can easily plug your search engine code. We’re going to focus instead on the sprint finish to getting something returned to your browser. We also need a few pages in our website project:

A search page that accepts a query string search parameter, as explained in the opening section
A few additional pages that we can present as actual search results

Our demo service is going to be query-string agnostic; so, although it will accept a query string, the results it returns are not going to be influenced by that input string – that’s for you to do. The final step after this is to craft our OpenSearch description document so that web browsers can pick up our search provider.

So, quickly:

Add three content pages to your web project – we’re going to call ours ‘Result1.aspx’, ‘Result2.aspx’ and ’Result3.aspx’. Put some basic content on there that you can use to discriminate the three from each other.
Add another page, and call it ‘Search.aspx’. Modify the code-behind as follows:

public partial class Search : System.Web.UI.Page
{
  private string _searchPhrase;
  public string SearchPhrase
  {
    get
    {
      if(_searchPhrase == null)
      {
        if (Request.QueryString.Count == 0 ||
          (_searchPhrase = Request.QueryString["q"]) == null)
          _searchPhrase = "";
      }
      return _searchPhrase;
    }
  }
}

Now, open the mark-up for the page, and replace the body with:

<body runat="server">
  <div>
    <h1>
      You searched for '<%= SearchPhrase %>'</h1>
  </div>
  <div>
    Results:</div>
  <div>
    <ul>
      <li>
        <h2>
          <a href="Result1.aspx">Result 1</a>
        </h2>
        The first result page
      </li>
      <li>
        <h2>
          <a href="Result2.aspx">Result 2</a>
        </h2>
        The second result page
      </li>
      <li>
        <h2>
          <a href="Result3.aspx">Result 3</a>
        </h2>
        The third result page
      </li>
    </ul>
  </div>
</body>

With all that in place, it’s now time to go back to our OpenSearch.svc and expose our XML and JSON endpoints.

The first step is to open the .svc mark-up itself and add the highlighted code to it, to enable the WebServiceHostFactory:

<%@ ServiceHost Language="C#" Debug="true" Service="OpenSearchDemo.OpenSearch" CodeBehind="OpenSearch.svc.cs"
Factory="System.ServiceModel.Activation.WebServiceHostFactory" 
%>

Now we open the code-behind file and write our two service methods – one for XML and one for JSON:

[ServiceContract()]
public class OpenSearch
{
  [OperationContract]
  [WebGet(UriTemplate = "suggestions?q={query}",
    RequestFormat = WebMessageFormat.Xml,
    ResponseFormat = WebMessageFormat.Xml)]
  public Suggestions GetSuggestions(string query)
  {
    //TODO: replace with your real search engine code.
    Suggestions toReturn = new Suggestions() 
      { Query = query, Section = new SuggestionsSection() };
    //for brevity, we've fixed our dev web server's port so that we know exactly where
    //these URLs need to point.  In reality you'll either use auto-discovery by taking
    //a look at the WCF WebOperationContext, or via a configuration file.
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 1 Description",
      Text = "Result 1",
      Url = "http://localhost:37082/Result1.aspx"
    });
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 2 Description",
      Text = "Result 2",
      Url = "http://localhost:37082/Result2.aspx"
    });
    toReturn.Section.Add(new Suggestion()
    {
      Description = "Result 3 Description",
      Text = "Result 3",
      Url = "http://localhost:37082/Result3.aspx"
    });
    return toReturn;
  }
  [OperationContract]
  [WebGet(UriTemplate = "json/suggestions?q={query}",
    RequestFormat = WebMessageFormat.Json,
    ResponseFormat = WebMessageFormat.Json)]
  public JSONSuggestions GetJSONSuggestions(string query)
  {
    return new JSONSuggestions(GetSuggestions(query));
  }
}

Note how the implementation of the JSON service is simply a piggy-back off the XML one. When you’re writing XML and JSON services in this way, typically all the code will either go in one half of the implementation, or in another common block of code that both can call. The contents of the WebGet attribute are very important at this point – and you should familiarise yourself with RESTful programming in WCF if you’re going to be writing more of these kinds of services.

NOTE: In the downloadable source code, we’ve included the ‘Image’ property for each of these results, so you can see the attribute-based image data in action as well.

Testing the services

At this point you should be able to build and run without any errors. Once done, you should be able to open the url http://localhost[:devwebport]/suggestions.svc?q=Hello+World and, although you might not actually see any output in the browser, if you open the source for the current page you’ll see the XML response in the format the IE8 will be expecting.

Also, if you instead open http://localhost[:devwebport]/json/suggestions.svc?q=Hello+World, you should be prompted to initiate a file download which, when you open it in notepad, will be a JSON array of arrays as per the actual OpenSearch suggestions spec.

If in doubt, use Fiddler2 to debug the web traffic. Note that you cannot get other people to call this yet, if you’re using the development web server that is, because it only responds to requests from the local machine.

Writing the OpenSearch Description document

Again, we’re not giving a full description of the OpenSearch spec here – everything you need to do is available from the OpenSearch site, we’re just giving you the XML you need to get up and running. Add a new XML file to your web project, call it OpenSearch.xml, paste this in after the <?xml> directive:

<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" 
                       xmlns:moz="http://www.mozilla.org/2006/browser/search">
  <ShortName>JSLabs OpenSearchDemo</ShortName>
  <Description>Expose OpenSearch on your Website, with suggestions!</Description>
  <InputEncoding>UTF-8</InputEncoding>
  <!-- used by all opensearch compatible browsers -->
  <Url type="text/html"
       template="http://localhost:37082/search.aspx?q={searchTerms}"
       method="Get"/>
  <!-- suggestions - used by IE8 only -->
  <Url type="application/x-suggestions+xml"
       template="http://localhost:37082/opensearch.svc/suggestions?q={searchTerms}"
       method="Get" />
  <!-- suggestions - used by Firefox -->
  <Url type="application/x-suggestions+json"
       template="http://localhost:37082/opensearch.svc/json/suggestions?q={searchTerms}"
       method="Get" />
  <!-- best practise to include a link to self -->
  <Url type="application/opensearchdescription+xml" 
       rel="self" 
       template="http://lostlhost:37082/opensearch.xml" />
  <!-- not required, but another extension supported by Firefox -->
  <moz:SearchForm>http://localhost:37082</moz:SearchForm>
</OpenSearchDescription>

Final Step – Wiring the OpenSearch document up for the browser

As this excellent article on MSDN describes – there are a few ways to promote your OpenSearch provider. One is auto-discovery, which is the recommended standards-based way to do it, and is supported by both IE7/8 and Mozilla Firefox 2/3. To enable auto-discovery for our demo site, open up your Search.aspx page, and add this to the <head> region:

<link title="JSLabs OpenSearch Demo Provider" rel="search"
   type="application/opensearchdescription+xml"
   href="http://localhost:37082/opensearch.xml" />

Save the mark-up – and now, open IE 7/8. Browse to /Search.aspx (no query string required) – and you’ll see the quick search box change colour:

Click on the down-arrow and expand the ‘Add Search Providers’ pop-out menu.

Confirm the dialog box that comes up (in IE8 make sure that the ‘Use suggestions from this provider’ box is ticked). Once done, expand the list again, and select the newly added ‘JSLabs OpenSearch Demo’ provider (there’ll be two very similar, it’s the one with the magnifying glass) and start typing, you should see your three suggestions appear:

If you select either of the three results, notice that they take you straight through to the Result[1-3].aspx pages, rather than simply launch the search page again with that suggestion (this is how our Job Search suggestions end up taking you straight through to the job page).

For Firefox, the process is nigh-on identical, browse to the page, the search box will highlight and you can select the provide from a dropdown list:

Again, you’ll be asked to confirm whether you want to use suggestions from this provider, which you do.

At this point you can immediately start typing in the quick-search box, and our suggestions will appear:

This is where Firefox is a bit of a let-down, however (and why our job search provider does not provide suggestions in Firefox), if you select any of these results, you’ll notice that instead of browsing directly to the URL we’re passed back in the results (like IE8), Firefox simply launches the search page with the string ‘Result 1’, ‘Result 2’ or ‘Result 3’ (it’s whichever Completion string is selected). This is a bug in the Mozilla browser, and was raised quite a while ago – we await to see if it’ll be fixed.

The other way to get your provider added to a browser is to use the non-standard AddSearchProvider method of the window.external property – and have a hyperlink or button that users can click to add the provider manually to the browser:

<a href="#"
   onclick="window.external.AddSearchProvider('http://localhost:37082/opensearch.xml')"
   >Add JSLabs OpenSearch Demo provider</a>

This is the technique that we use for our Web Browser OpenSearch Provider, primarily because we are hosting our provider on a different sub-domain to the main site.

As is mentioned in the launch article for the Web Browser OpenSearch Provider, Chrome and Safari do not support OpenSearch. Opera 9 should, according to the documentation, support the auto-discovery method of supplying OpenSearch to the web browser. Support doesn’t just have to stop at the browser, though - remember that there are websites out there which could leverage your OpenSearch capability (not the suggestions, just the basic search url) in order to better your content to a wider audience. All in all, installing OpenSearch into your website is a worthwhile effort – and with WCF, supporting the suggestions extension is surprisingly simple.

Download the source code for this article now!

Building Labs – XML RPC Backend for Windows Live Writer in C# with WCF

2009-07-10T17:48:31+01:00

In the first of our series of articles about how JobServe Labs is built, we’re going to look at how we’re using Windows Live Writer (WLW) to publish our articles, and how we’ve setup a blog service internally with WCF for WLW to talk to. If you’re planning a similar deployment, or if you’re looking for a way to build page-based content management for a website, then you might want to read through – as we point you in the direction of how to build it yourself.

Why use Windows Live Writer?

We needed to be able to create rich content with the minimum of fuss, with the ability to embed content such as video, pictures etc only a few clicks away. While there are ready-made javascript-based HTML editors out there (TinyMCE and FCKEditor are two excellent and very popular examples) we tend to find that non-technical users don’t get on with them too well – many people will copy and paste from word processing applications, but inevitably the resulting content is messy and doesn’t translate to the web page very well.
Writing our own WYSIWYG X-Html editor, whilst not impossible, would have taken far too long.
With WLW, users get a recognisable interface on which they will require minimal training (a real consideration in an enterprise environment).
We can extend WLW through it’s very slim and easy to use plugin APIs, all you need as a developer is to install it, and you have the SDK.
We’re already using WLW to post to our own internal blogs, which are hosted on Microsoft Office Sharepoint, and many of our colleagues are already using this, so all we’re really doing for these guys is adding an additional information outlet for them to post to in an environment they already understand.
WLW already knows how to talk to many popular blogging engines such as Wordpress, Blogger and MetaWebLog-based engines such as BlogEngine.Net. One of our developers is already using it blog to his own personal blog, and we therefore we knew that some of our expectations could be achieved so long as we stuck to a recognised standard.

The architecture

Unlike other out-of-the-box blogging sites/engines, you will not find our blog endpoint on our website. The primary reason for doing this was to minimise attack vectors to the site – if you present a callable endpoint to the outside world then somebody will try to hack it. Anybody that could ever want to interact with the articles on our site will either be directly on our enterprise network, or connected to it remotely via a VPN.

The basic architecture, then, is:

Blog capabilities

The first stage was for us to decide which common blog functionality we wanted to support on our blog, and from there we could then determine which Blog spec we would code against. Using the Windows Live Writer default capabilities listings, we decided that we would use the MetaWebLog specification as our baseline, as it’s the closest to what we wanted. There were a couple of additional features we wanted to add, but these can be done using the <options> element of the wlwmanifest.xml file that our website presents to Live Writer – for those who are interested, here’s the complete listing from ours:

<options>
	<clientType>Metaweblog</clientType>
	<supportsPostAsDraft>Yes</supportsPostAsDraft>
	<supportsFileUpload>Yes</supportsFileUpload>
	<supportsCategories>Yes</supportsCategories>
	<supportsCustomDate>Yes</supportsCustomDate>
	<supportsCommentPolicy>Yes</supportsCommentPolicy>
	<supportsExcerpt>Yes</supportsExcerpt>
	<supportsNewCategories>Yes</supportsNewCategories>
	<supportsNewCategoriesInline>Yes</supportsNewCategoriesInline>
	<supportsMultipleCategories>No</supportsMultipleCategories>
	<supportsHierarchicalCategories>No</supportsHierarchicalCategories>
	<supportsKeywords>No</supportsKeywords>
	<supportsPingPolicy>No</supportsPingPolicy>
	<supportsAuthor>No</supportsAuthor>
	<supportsSlug>No</supportsSlug>
	<supportsPassword>No</supportsPassword>
	<supportsTrackbacks>No</supportsTrackbacks>
	<supportsPages>No</supportsPages>
	<supportsPageParent>No</supportsPageParent>
	<supportsPageOrder>No</supportsPageOrder>
	<!--additional live writer options-->
	<supportsEmptyTitles>No</supportsEmptyTitles>
	<requiresHtmlTitles>No</requiresHtmlTitles>
	<requiresXHTML>Yes</requiresXHTML>
	<supportsScripts>Yes</supportsScripts>
	<supportsEmbeds>No</supportsEmbeds>
	<characterSet>UTF-8</characterSet>
	<maxCategoryNameLength>40</maxCategoryNameLength>
	<supportsAutoUpdate>Yes</supportsAutoUpdate>
</options>

If you’ve had any experience customising Live Writer from the manifest file, you might be wondering why we are specifying values that are the same as the defaults for the MetaWebLog spec. Part of it is to ensure that if Live Writer’s understanding of this spec changes in a future version, it’s understanding of our blog will not. It’s also to make it easier in the future should we wish to add capabilities – changing an existing value from No to Yes is simply faster than having to write the element in first.

An important point to note, if you’re going to develop your own blog engine like this, is that the MetaWebLog specification implies the Blogger specification as well, and WLW mixes it’s XML RPC calls between the two for the different features where the MetaWebLog RPC methods supersede the Blogger methods.

As an example, it will use the metaWeblog.editPost method instead of the blogger.editPost method when you re-publish an existing post; but it will use the blogger.deletePost method to delete a post because MetaWebLog doesn’t define an alternative method for doing so.

XML RPC

The next step was to decide how we were going to expose the blog to Live Writer – which is expecting an XML RPC endpoint to talk to. There is a very popular .Net Framework implementation of XML RPC, called XML-RPC.net, which we did look at, however we are moving all of our network coding over to WCF because your code is then not tied to a particular transport or protocol.

The problem with that is that WCF does not come with built-in message formatters and operation selectors compatible with XML RPC; so we were looking at writing our own. As we all know, however, as developers, we must always exhaust the community first to see if somebody else has done it, and to our joy we stumbled across a project by Clemens Vasters – a senior developer at Microsoft – where he’d been developing just such a thing for DasBlog. The zipped solution, which he talks about and links to on his blog, includes WCF operation selectors and formatters for XML RPC support in the same way that you write any other WCF hosted service (i.e using the ServiceContract attribute and OperationContract attribute). All we did was to take his ServiceHost setup code and move it to a core component that we could then host either in a console application (as in his example), which was great for development, or in a windows service for when we actually released it to the rest of the business.

Helpfully, he also includes WCF-contracted interface definitions (and the corresponding classes for their related structs) for Blogger, MetaWebLog and MovableType, as well as an in-memory implementation of his Blog Engine that runs from the command-line (which Live Writer will happily connect to). We didn’t use these directly, but if you’re following the same path as us, then you can go straight ahead and do so. Clemens stacks his interfaces up like this:

//define the interface for the actual blog api that will be called
[ServiceContract]
public interface IBlogAPI : IMovableType
{
}
//define the interface for the IMovableType interface
[ServiceContract(Namespace = "http://www.sixapart.com/developers/xmlrpc/")]
public interface IMovableType : IMetaWeblog
{
	//example
	[OperationContract(Action = "mt.setPostCategories")]
	bool mt_setPostCategories(
		string postid,
		string username,
		string password,
		Category[] categories /* MovableType category */ );
	// rest of the method definitions ommitted.
}
//define the interface for the IMetaWebLog interface
[ServiceContract(Namespace = "http://www.xmlrpc.com/metaWeblogApi")]
public interface IMetaWeblog : IBlogger
{
	//example
	[OperationContract(Action="metaWeblog.editPost")]
	bool metaweblog_editPost(
		string postid,
		string username,
		string password,
		Post post, /* MetaWebLog post struct */
		bool publish);
	// rest of the method definitions ommitted.
}
//finally, define the base interface for IBlogger - the base API used by all
[ServiceContract(Namespace = "http://www.blogger.com/developers/api/1_docs/")]
public interface IBlogger
{
	//example
	[OperationContract(Action="blogger.deletePost")]
	bool blogger_deletePost(
		string appKey,
		string postid,
		string username,
		string password,
		bool publish);
	// rest of the method definitions ommitted.
}

One of the challenges is that many of the specifications share ‘struct’ definitions that are called the same thing, but which are subtlely different. The MetaWebLog Post struct, for example, is not the same as the Blogger Post struct. Clemens’ code namespaces each of the above interfaces and the classes that they require according to the blog API they implement. As a result, you will find yourself implementing some methods from different interfaces which use slightly different structs even if they have the same name. Thankfully, Windows Live Writer will always send the right struct version for the particular method it’s calling!

Now, to setup your real blog API, you simply declare a class that implements the root IBlogAPI (it’s actually IBloggerAPI in Clemens’ code – but that’s easy to confuse with the Blogger interface!), get Visual Studio to stub out all the interface methods you need, and then start coding against your database. We used Linq to Sql for our database interfacing layer because it’s really quick to get up and running, and is resilient to schema changes; it’s also hack-proof. Read through Scott Gu’s excellent series on using Linq to Sql.

We’re not going to go through hosting the service and the C# code required to do that – it’s all in Clemens’ project.

Bugfixes required to Microsoft.Samples.XmlRpc

During development, we discovered a couple of minor bugs in the XmlRpcDataContractSerializer class – which we had to patch for our use. Unfortunately, we can’t make the amended version of the file available here as it’s got Microsoft’s Copyright on it, so we have sent our amendments to Clemens and hopefully he’ll patch the release that we’ve linked to above. To patch it yourself, open up the file XmlRpcDataContractSerializer.cs and make the following changes:

/* -----------------------------------------------------------------
 * Change 1: Line 65, adding else statement to if block that starts
 * if (dataMembers.ContainsKey(memberName))" on line 44:
 * Fixes: A crash that occurs if a struct field is sent that is
 * not part of the class being used to represent the struct in .Net
 * -----------------------------------------------------------------*/
else
{
	reader.Skip();
}
reader.ReadEndElement(); // value
reader.MoveToContent();
reader.ReadEndElement(); // member
reader.MoveToContent();
/*-------------------------------------------------------------------*/
/* ------------------------------------------------------------------
 * Change 2: Line 128, Replace case statement which deserialises
 * an XmlRpc DateTime value
 * Fixes: Crash when parsing DateTime values passed by Windows Live
 * Writer when setting an explicit publish date on an article.  
 * XML RPC date format is not a .Net date time format.
 * ------------------------------------------------------------------*/
case XmlRpcProtocol.DateTime:
try
{
  returnValue = Convert.ChangeType(reader.ReadElementContentAsDateTime(), targetType);
}
catch
{
  string dateTime = reader.Value;
  //have to fix the current reader position (dirty fix).
  reader.Read();
  reader.Read();
  reader.Read();
  returnValue = XmlConvert.ToDateTime(dateTime, "yyyyMMddTHH:mm:ss");
}
break;
/*-------------------------------------------------------------------*/
/* ------------------------------------------------------------------
 * Change 3: Line 249, replace the else-if
 * Fixes: This is the mirror fix to change 2 - ensures that outgoing
 * DateTimes are formatted correctly.
 * ------------------------------------------------------------------*/
else if (valueType == typeof(DateTime))
{
  writer.WriteStartElement(XmlRpcProtocol.DateTime);
  //modification here to write a 'proper' Xml RPC DateTime.
  if (value is DateTime)
    writer.WriteValue(((DateTime)value).ToString("yyyyMMddTHH:mm:ss"));
  else
    writer.WriteValue(value);
  
  writer.WriteEndElement();
}
/*-------------------------------------------------------------------*/

Layout and Preview modes for WLW

So now we have our blog service ready to go (assuming you’ve done your database and your XML RPC Endpoints are setup) the last thing we have to do is to tell Live Writer where our blog is, and what our pages look like.

Before you begin this bit, you should have a website that can display your blog posts – at a minimum this must have a root page which shows either all posts, or the most recent n posts – the top post should be the one that was posted most recently. Use this as your blog’s homepage from WLW’s point of view. Ours is our All Articles page.

For the best editing experience in WLW you should also have two additional Html Files, one for WebLayout view (editing view in WLW) and WebPreview view (the preview shown in WLW). The Live Writer Team have written guidance on MSDN as to how to make these two files – they use placeholders for the title and the article body – so the most basic version of these would simply look like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<body>
	<h1>{post-title}</h1>
	{post-body}
</body>
</html>

Live Writer simply substitutes the {post-title} and {post-body} macros for the content that it’s got from the current post – if the page requests CSS or Image files, then WLW will go and download them.

From our experience with this process, we also have a few suggestions and guidelines for you to follow regarding these files:

WebLayout View

If your site has a graphical theme and funky layout (ours, for example uses a master page that puts the header and footer on, surrounds most content with a rounded border, and sticks a load of other links on), you’ll typically leave it off this page so as to maximise the default editing experience.

For example, this article looks like this in my editing view:

Note how the title is much larger than it appears on this page, and how there are no borders anywhere. The font and link styles, however, are maintained.

WebPreview View

This one should be as close to the design used for the page that displays a single article (in this case the page that you’re looking at right now) – ours, for example is an aspx page that includes the master page and a lot of the extra HTML layout content.

Because you’re likely to be throwing all manner of CSS tricks at this page, it’s possible that it’ll not be identical to the actual web page; my preview, for example, takes a few liberties with the placement of our Feeds and Categories panels on the right hand side (WLW does a good job of it’s HTML compliancy, but it’s not perfect). However, the important thing is that it accurately displays the post inside the rounded panel surrounding the content:

The manifest file

Now that you have your preview and layout htmls, you can write your manifest file. From the information that we’ve mentioned so far, you’ve almost got what you need to do it. First, I’ll link again over to the Windows Live Writer Customisation API topic on MSDN, so you’ve got all the material to back you up.

Let’s assume that you are going to implement MetaWebLog exactly as is, with no customisations. Let’s also assume that your homepage is default.aspx, and that your WebLayout and WebPreview htmls are called ‘WebLayout.html’ and ‘WebPreview.html’. Your manifest file is going to look something like this:

<?xml version="1.0" encoding="utf-8" ?>
<manifest xmlns="http://schemas.microsoft.com/wlw/manifest/weblog">
	<!-- leave the options bit out, because we're sticking with what
		live writer uses as the defaults for MetaWebLog -->
	<weblog>
		<serviceName>My Blog</serviceName>
		<homepageLinkText>http://my.domain.com/default.aspx</homepageLinkText>
	</weblog>
	<views>
		<default>WebLayout</default>
		<view type="WebPreview" src="/WebPreview.html"/>
		<view type="WebLayout" src="/WebLayout.html"/>
	</views>
</manifest>

As the MSDN documentation states, if you place this file in the root of your website and call it ‘wlwmanifest.xml’, then Windows Live Writer will be able to auto-detect it. If, however, you need to point Live Writer specifically at a particular file (we actually have two Blog endpoints for our site, one for our Articles and one for our Projects, so we need two manifests), then you use a <link> tag in the <head> section of the blog’s homepage. Here, it’s the ‘rel’ and ‘type’ attributes that do the magic:

<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="MyWlwManifest.xml" />

With this in place, Windows Live Writer can auto-discover the manifest that you want it to use.

Final Step – Adding your Blog to Windows Live Writer

If you’re at this stage:

1) You’ve got a website that has a manifest file for Live Writer to download as per the last section – let’s say that the address is http://localhost/default.aspx for homepage.
2) You’ve got your blog API listening on an address on a local machine as per the section about Xml Rpc. Let’s say that’s on http://localhost:9999/blog/api.

Then you’re ready to go!

Go to the ‘Blogs’ menu of Live Writer and select the bottom option “Add Blog Account”
Select ‘Other Blog Service’
Set the web address of the blog to http://localhost/default.aspx
If you’ve added permissions to your blog, then set the username and password. Click next.
Now WLW wants to know what type of provider it’s going to be talking to – select “Metaweblog API”
In the address box type http://localhost:9999/blog/api, click next again.

That should be it (other than confirming your blog’s name). If Windows Live Writer shows a message about making a temporary post in order to detect the theme of your site, then either it hasn’t picked up your manifest file, or it’s not been able to grab the WebPreview and WebLayout files referenced in the manifest. Review those URLs (ours use absolute paths to ensure that there can be no confusion) and try again (best to remove the account and add it again).

Debugging the XML RPC Calls

As with all situations where you have an application calling another by a remote mechanism, it’s nice to be able to see what is actually being sent to and fro. What we’ve done during development is to use Message Logging and Tracing in WCF (however, it’s not exactly real-time, because until your app shuts down it doesn’t actually sign-off the trace files, causing the Service Trace Viewer to throw errors) and to use Fiddler to debug the Http traffic travelling between Live Writer and your service. Unlike Internet Explorer, which automatically uses Fiddler when it starts up, you have to tell Live Writer to use your local proxy. There’s a setting in Tools –> Options –> Web Proxy which controls this; as soon as you change that you’ll start seeing your Xml RPC Payloads going to and fro.

Conclusion

There are a lot of content management systems out there, and they all offer a huge amount of features. One of the primary areas of distress for anybody wanting to roll out such a site, however, is the ability to get content, update it and post it again in a way that offers the most flexibility. By using a rich WYSIWYG offline editor like WLW you get a whole host of features immediately after installation, plus you also get a platform upon which you can build very quickly, thanks to its plug-in interface. We have a plug-in purely for sending up additional files (like the Vista Sidebar Gadget .gadget file) to the blog, because WLW only automatically uploads media content. Developing this took one day to achieve.

After hooking WLW into your blog, you’re then free to code the workflows behind the service that it uses. This post, for example, will be posted to our blog as a draft for another member of the team to open and review. They can use WLW to download the draft from it’s ‘Recently Posted’ feature. If that person is happy with that content, they can send it back to the blog as a published article simply by hitting the ‘Publish’ button.

We also have some role-based security behind the scenes, which hangs off of Active Directory, which means that only some people are allowed to actually publish the articles. The rest are restricted to posting drafts, which must then be ‘properly’ published by someone with permissions.