How to Set Up Kernel Debugging For Windows in VMWare ESX

Written by: Electric Bee

You’ll know from my previous blog posts that we have an extensive virtual machine infrastructure here at CloudBees (the Continuous Delivery company) to support our build and test process (as many of our customers do). Thought I’d share a neat trick I picked up while rolling out our system.
Most of us won't need to debug at the kernel level with any real frequency. However, when you do need to debug at the kernel level in Windows virtual machines, this may come in handy. The instructions given below will allow you to set up a Windows kernel debug session in a virtual environment.

The Virtual Machines

Getting started: You will need two virtual machines, both deployed on the same physical ESX server. That may mean that you manually deploy them to a specific ESX server VMware Infrastructure Client , or use a LabManager Configuration that contains both machines, or specifically target an ESX server in LabManager as you deploy the configuration.

The Debugee

This is the machine that we will connect to and that is being debugged. The general setup is fairly straight forward:
Log into VMware Infrastructure Client:

  1. Deploy/locate the machine you want to debug in the tree view at the left.

  2. Click on that name in the tree view.

  3. Make sure the virtual machine is powered off. If there is a link in the Getting Started panel for this virtual machine, you need to either connect to the console of this machine and power it off, or press the Power off virtual machine link in the Getting Started panel.

  4. Click on Edit virtual machine settings in the Getting Started panel for this virtual machine.

  5. On the Hardware tab, click the Add... button to add a new device.

  6. Choose Serial Port and push Next .

  7. Choose Connect to named pipe and push Next

  8. Enter a name for the pipe (for example: mypipe).

  9. Pick Server for the near end.

  10. Pick A virtual machine for the far end.

  11. Make sure you choose Connect at power on .

  12. Deselect the Yield on CPU Poll checkbox.

  13. Push the Next button.

  14. Check the summary and push the Finish button.

  15. Push the Ok button in the main Settings dialog (for me the serial port label remains in "...adding" forever, I assume that is just a refresh problem).

  16. Deploy the virtual machine (same ESX server as the other machine).

Log into the machine as Administrator:

  1. Right click on My Computer .

  2. Choose the Properties menu item.

  3. Click on the Hardware tab.

  4. Click on the Device Manager button.

  5. Expand the Ports tree item.

  6. Right click on the newly created COM port and choose Properties . Sometimes these are numbered out of order, depending on what is already there, but usually this will be COM1. If there are multiple ports, and you are not sure which one, I recommend just changing all of them.

  7. Click on the Port Settings tab.

  8. Choose Baud Rate: 115200 .

  9. Choose 8, None, 1, None for the other parameters (defaults).

  10. Push Ok and close all the dialogs.

  11. Edit c:boot.ini (and add the /debugport and /baudrate options):

    • multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /debugport=com1 /baudrate=115200

  12. Reboot the machine

The Debugger

The machine that runs the kernel level debugger controls the machine being debugged. Again... once you know what you need, the setup is fairly straight forward:
Log into VMware Infrastructure Client

  1. Deploy/locate the machine you want to debug in the tree view at the left.

  2. Click on that name in the tree view.

  3. Make sure the virtual machine is powered off. If there is a link in the Getting Started panel for this virtual machine, you need to either connect to the console of this machine and power it off, or press the Power off virtual machine link in the Getting Started panel.

  4. Click on Edit virtual machine settings in the Getting Started panel for this virtual machine.

  5. On the Hardware tab, click the Add... button to add a new device.

  6. Choose Serial Port and push Next .

  7. Choose Connect to named pipe and push Next

  8. Enter a name for the pipe (for example: mypipe. IMPORTANT! This is the same as the other machine).

  9. Pick Client for the near end (IMPORTANT! This is different than the other machine).

  10. Pick A virtual machine for the far end.

  11. Make sure you choose Connect at power on .

  12. Deselect the Yield on CPU Poll checkbox.

  13. Push the Next button.

  14. Check the summary and push the Finish button.

  15. Push the Ok button in the main Settings dialog (for me the serial port label remains in "...adding" forever, I assume that is just a refresh problem).

  16. Deploy the virtual machine (same ESX server as the other machine).

Log into the machine as Administrator:

  1. Right click on My Computer .

  2. Choose the Properties menu item.

  3. Click on the Hardware tab.

  4. Click on the Device Manager button.

  5. Expand the Ports tree item.

  6. Right click on the newly created COM port (sometimes these are numbered out of order, depending on what is already there, but usually this will be COM1), and choose Properties .

  7. Click on the Port Settings tab.

  8. Choose Baud Rate: 115200 .

  9. Choose 8, None, 1, None for the other parameters (defaults).

  10. Push Ok and close all the dialogs.

  11. Download and install windbg .

Debugging

So now we have two machines, eagerly awaiting debugging.

  1. Go to the debugger machine and start windbg .

  2. Click on the File menu button.

  3. Choose the menu item Kernel Debug .

  4. Enter the proper baud rate (from before), and COM port.

  5. You can now press Control-Alt-K to initiate a connection

  6. Have fun looking at assembler code, threads etc.

Interesting Stuff

One interesting thing to note is that on the ESX server (which is Linux based), named pipes are currently always created under /var/log/vmware , and should be just regular file names. During my research I created a named pipe there directly from the OS, which resulted in VMWare throwing up a dialog box stating that the named pipe was already there. It then offered to overwrite the named pipe, which I did.
Based upon my research I expect that you could do the whole setup falling back to the physical serial port on different ESX servers and then use a serial cable, but I haven't tried that. Happy debugging!

Stay up to date

We'll never share your email address and you can opt out at any time, we promise.