We’re in the process of upgrading our development environment from Windows Server 2003 x64 to Windows Server 2008 R2.  Apart from the desktop experience being far superior, we’re also moving our web servers up to IIS 7.5, therefore we want all our web developers to be running the same.  Windows 7 is of course an option for this (and it works extremely well as a web development environment – unlike Windows XP Pro), however it doesn’t offer Hyper-V, which is a must for many developers who want to run their own database servers and mock web servers.

Inevitably, however, we still have to support our existing codebase - all of which runs on Windows Server 2003.  Occasionally, therefore, we need to debug something that’s running live because a problem arises that is not easy to reproduce in a model office environment.  This typically means installing the Visual Studio Remote Debugger on the server that’s running the code and then attaching to it from the local Visual Studio.  Many people would say that this is bad practise; in our opinion bad practise is to go the whole hog and install Visual Studio on the server machine!  Remote debugging, on the other hand, is a reality of working as a developer in a technology-driven business.

The problem

Today we had to do just this, but when we tried to connect to the remote machine we got the error message "a security package specific error occurred".  If you do a web search for this error, you’re likely to see this link from MSDN forums, where the user eventually solved the issue by creating a local user on both the target and local machines, and ran both VS2008 and the Remote Debugger as that account.

The solution presented is to:

• Create two, administrator-level, identical local users on both the debugger machine and the debuggee machine
• Run the msvsmon.exe application (the Remote Debugger binary) as that local user on the debuggee
• Startup Visual Studio on the debugger machine as that local user
• Attach to the debuggee

We weren’t happy with this resolution, because it points to a more fundamental problem – if Remote debugging works between two Windows 2008 R2 or Windows 7 machines, using the same VS binaries, why wouldn’t it work when talking to a Windows Server 2003 machine?  Equally, we don’t want to have to go around creating local users on developer machines and servers just to get remote debugging to work.  When you’ve reached this stage, it typically means that you’ve got an error you can’t reproduce any other way and you just need to ‘get it fixed quick’.  Also, in a TFS environment, the Visual Studio instance in which you’re doing the debugging will not be able to connect to TFS, because the local user that you’re using is unlikely to have access to source control etc.

The root cause of this problem turns out to be slightly more subtle than the user on the aforementioned MSDN forum post assumed.  It is indeed authentication that is failing, but not the authentication of the user doing the debugging against the server they wish to debug.  In fact, after the remote debugging service receives a request to attach, it then authenticates back to the machine that the request originates from, and it is at this point that authentication is going wrong (we were able to generate the error that is described in that previous link by using a local account for the debugger service on the target machine but without creating it on the development machine).

Even though our service was running as a domain account which has logon rights to the development machine, for some reason the NTLM network logon token for that account that is sent back by the debugging service would appear not to work properly.

By enabling network level NTLM auditing (in Local Security Policy, enable the two settings “Network security: Restrict NTLM: Audit Incoming NTLM traffic” and “Network security: Restrict NTLM: Audit NTLM Authentication in this domain”) we were able to see the logon occur, but then the logon session is destroyed (presumably as a result of the “Security package specific error” that occurs).

A (slightly) easier solution

The good news from this is that we can offer a streamlined solution to the one provided in the above MSDN link, the bad news is that you’ll still need to create the local users:

• Create the two users as before on the two machines – however, it only needs to be an administrator on the target machine.
• Modify the remote debugger service on the target machine to use the new local user, start the service.
• Run Visual Studio 2008 as you would normally (no need to run as the local user), and attach the debugger as normal.

Tell Microsoft!

We have logged a bug on Microsoft Connect so that the VS team can investigate the issue – if you are affected by this issue then please vote for it!  At present we do not know if this also affects Visual Studio 2010 (we have a development lab set up and will investigate), if you can verify that it does, then add your experience to that bug so that we can ensure that at least the next version of VS isn’t plagued by this issue, even if it doesn’t get fixed in the current version.