Tutorial for Debugging ReactOS with VirtualBox, com0com, and HyperTerminal in Windows
To the ReactOS rookie, the thought of debugging ReactOS can be intimidating. Almost as intimidating is figuring out how to get it done. I finally worked up the nerve to try debugging when encountering a particularly nasty bug, so I'll share my newfound knowledge in a simple 20-step process. To my knowledge, there aren't any step-by-step tutorials on your first debugging experience, so I'll write it for you.
For this tutorial, we'll be working with Windows XP, and the included HyperTerminal application. We'll also be using the open-source VirtualBox and com0com applications. Once you grow comfortable with debugging, you can use different applications of your choosing, but, to get you started, we'll use these programs. I'll be assuming you already have some familiarity with VirtualBox and ReactOS. If not, feel free to test the waters and get a feel for the software before debugging.
1. Download VirtualBox, if you haven't already. http://www.virtualbox.org/
2. Download a ReactOS ISO. Trunk builds are preferred for debugging purposes. http://www.reactos.org/getbuilds/
3. Create a ReactOS virtual machine on VirtualBox.
4. Download com0com. http://sourceforge.net/projects/com0com/files/
5. Install com0com. For the sake of simplicity, have it on your start menu.
6. Change the default names of your virtual serial ports. They'll be something like "\\.\CNCA0" or whatever. You can do this by going to the "Setup" program. If you've installed com0com to the Start menu, the file path is Start>All Programs>com0com>Setup. If you see a small window with a bunch of checkboxes and red, black, and green dots connected by lines, you're there. For this tutorial, name your first serial port COM4 and your second port COM5.
7. Go into the settings for your ReactOS virtual machine in VirtualBox.
8. Right-click your ReactOS virtual machine and select the "Settings" option. (Alternately, highlight your ReactOS virtual machine and click the "Settings" icon at the top of the screen.)
9. Under Settings, you should see a serial ports icon. Select it and you'll be presented with a menu. Under the "Port 1" tab, check the "Enable Serial Port" option. This will allow you to click the options underneath.
10. Set the "Port Number" drop box to "COM1". "Port Mode" should be "Host Device". For "Port/File Path", type in "COM4" (without the parentheses, of course). This is the first virtual serial port we made earlier.
11. Click "OK".
12. Open HyperTerminal. From the Start menu, the filepath is All Programs>Accessories>Communications>HyperTerminal.
13. You will be presented with a message box entitled "Connection Description". The name and symbol aren't really that important. I called my first connection "ReactOS Debug" and chose the atom with the thingys flying around it (far, far right) as my Icon, since it seemed appropriate for ReactOS, but it's your call.
14. The next message box will be entitled "Connect To". Under the "Connect using" dropbox, select "COM5". This is the second virtual serial port you created with com0com. Then click "OK".
15. You will now see a menu that says "COM5 Properties". Set the "Bits per second" drop box to "115200" and click "OK". Everything is fine as-is.
16. HyperTerminal is now running. You should have a blank screen in HyperTerminal.
17. Back in VirtualBox, right-click your ReactOS virtual machine and select "Start". (This can also be achieved with the "Start" icon at the top of the VirtualBox main menu.)
18. The virtual machine should start. When FreeLoader boots up, select the ReactOS (Debug) option.
19. ReactOS will start booting up, and you should start recieving output in HyperTerminal. Take a minute and pat yourself on the back for a job well done, and maybe post on your Facebook about how l33t you are, like I did.
20. When you're done debugging, you can shut down HyperTerminal or ReactOS. You won't break anything by shutting one or the other down first. Welcome to the world of ReactOS debugging!
Once you've gotten the hang of getting output from your serial port, now the next step is learning how to file bug reports! (To be continued...)
I used the following articles on the ReactOS wiki for information.http://www.reactos.org/wiki/Debugginghttp://www.reactos.org/wiki/Com0comhttp://www.reactos.org/wiki/VirtualBox#Getting_debug_output
All credit goes to the writers of those articles. I'm just compressing them in an easy-to-read format.Also, kudos to hto, Haos, and vicmarcal for guiding me in the right direction on debugging.
I don't currently run Vista or Windows 7, so if someone wants to correct me or add more information, let me know what to add. If you're trying to debug on Linux, and you already know how to use Linux, you're already more l33t than me, so I can't help you.