As you may remember, a while back I decided to repave my Notion laptop with Windows Server 2008 Standard x64 so that I could make full use of the 4gb of ram and start playing with Hyper-V.
Since Win2k8 is a server OS, it doesn't come through with a bunch of niceties like Vista x64 does. In fact, the base installation has most services turned off to reduce the attack surface of the OS (very nice). That is another reason that I installed it over Vista x64. By using an opt-in model, my performance was better than that of Vista on the same hardware. At least it was until I added the Hyper-V role...
As you may have read in my earlier post on Hyper-V, once you install this role your OS is fundamentally reconfigured. Your OS originally sat directly on top of the hardware but after installation, your OS becomes the "Parent" partition (VM) and the Hyper-V layer sits between the hardware and all of the VMs.
So what's the problem?
After configuring the Hyper-V role I began creating VMs to play with. I first ported the TFS "Rosario" CTP VM from Microsoft from Virtual Server 2007 to Hyper-V and then ported the VSTS/TFS 2008 All-in-one VPC image. Once these two were installed I connected my copy of Visual Studio 2008 Team Suite to the TFS 2008 instance on the VM to see how it all worked. I noticed there was a bit of a lag when working with the VM but it wasn't anything that I couldn't live with and I really didn't have the time to investigate further.
How was it investigated?
Then came Mike Azocar's post about setting up your Win2k8 server as a workstation. I had initially configured my laptop using most of the same posts that Mike referenced. The one I didn't use was the Win2k8 Workstation Converter which basically automates all of the tweaks and hacks you would 
do by hand (and possibly screw up). I ran most of the utilities and was able to install the Vista x64 Sidebar and then added my favorite gadget, the CPU usage monitor. The first thing I noticed on the monitor was that my overall CPU usage was always up around 50%, even just after booting up.
This lead me to investigate further. The first thing that I did was grab the Process Explorer tool from SysInternals Live. After firing it up I noticed that one of the SvcHost processes was consistently running at 45% - 50%. To determine if this was the culprit I did what anyone would do...I killed the process. After blowing it away I saw the overall CPU usage drop to the single digits. A-ha...I was right, that's the problem. 
Now to figure out what app is running under the SvcHost process.
I didn't have to wait long. While checking email I noticed that the CPU usage had jumped back up again. I reopened Process Explorer and did a quick scan of the CPU usage column until I found my SvcHost process gone wild. By selecting this process I can hover the mouse over this entry to see what's running within it. As you can see, the first thing there is the DNS Client service (DNSCache). This is the service that keeps track of the IP addresses of sites you've visited so that you don't have to keep making round-trips to your Nameserver for resolution.
When I killed the process and checked the DNS Client service in the Services applet I noticed that it was not running. When I restarted it from the Services applet my CPU pegged out again. I think I'm getting closer to the root problem.
The next thing to do is to look at the lower pane of Process Explorer to see what files, registry keys, directories, ports, etc. that are being used. It also color-codes the entries to show opening (green) and closing (red). As you can see from the screenshot below, the DNS Client service keeps opening and closing the same registry key:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet003\Services\Tcpip\Parameters\ DNSRegisteredAdapters\{5F0F822F-7D54-4716-8809-65A83C851F8F}
What does {5F0F822F-7D54-4716-8809-65A83C851F8F} refer to?
I grabbed the GUID from the registry key, opened RegEdit.exe and did a search to see if I could figure out what the problem was. The first hit (shown below) showed that the GUID belongs to the Microsoft Virtual Network Switch Adapter. This device was created to allow the VMs to share the single network connection on my machine under Hyper-V. So that's why I didn't see this behavior prior to installing the Hyper-V role.
So why is it having problems accessing registry entries?
To find this out I had to resort to another SysInternals tool; Process Monitor. The first thing to do is to press CTRL + E to stop it from collecting events. This improves performance greatly. Next, clear out the event window (CTRL+X) so that when you apply the filter it doesn't have any work to do with the current buffer.
Since I can grab the PID of the SvcHost process I can use it to filter down all the information that Process Monitor collects to only those that belong to this process.
The core issue
After the filter was applied I started it collecting events (CTRL + E) again. I stopped it after a couple of seconds since I could clearly see the problem. The DNS Client service was trying to read a specific registry key entry that could not be found. The problem is that it the DNS Client service's way of handling the NAME NOT FOUND result is to retry the read. Unfortunately it doesn't ever stop trying, hence the infinite loop.
Let's go registry spelunking!
I opened up RegEdit again and navigated to the key in question. Lo and behold, there is no Flags entry, just like Process Monitor said.
To find out what else was missing I opened the key just above it. There are a lot more keys here.
So how do I fix this??
The easiest way is to add all of the missing values to the broken key. The problem is that I don't know what the correct values should be. So when all else fails, cheat! I opened up 3 other keys above and below the broken one and compared values. They all had the same entries. That made me a bit more confident. I exported one of the good keys to a text file (keys.reg) and then modified the path to point to the broken GUID. Once I saved it back and ran it the broken one looked just like the rest.
The real test is to turn the DNS Client service back on and check its behavior. After turning it on I found its entry in Process Explorer and checked the lower pane. There were no more tell-tale key open/close cycles and no excessive CPU usage.
The Root Cause
I have no idea why the DNSRegisteredAdapters entry for the Microsoft Virtual Network Switch Adapter were incomplete. I had been having issues getting my networking up and running since I stupidly tried applying what I know about Virtual Server 2007 networking to Hyper-V. They are as similar as VSS is to TFS Version Control so it's no wonder I screwed it up.
My pain, your gain!
And that's how Steve got burned today!
RSS 3.0
