One of the most important tasks in software development is testing. Without testing developers would not know about certain bugs that they themselves might not be encountering. It is also very important in large projects such as Operating Systems where regressions can often crop up unnoticed. It is important to find and quickly deal with regressions as these are easier to fix when their cause is known (ie. a recent commit). Up ahead we'll start of with the basics.
Now I know what most of you are thinking, hardware. Now we don't have anything against hardware, ReactOS should and in most cases (while you cross your fingers) does, but we can't be sure about YOUR hardware. We can't know if you misconfigured your BIOS, or $insert_hardware_manufacturer "tweaked" the hardware somehow. Right now ReactOS is in too early a stage to worry about such things. So we use Virtual Machines to do most of the testing, not only because of the above reason, but it's also more convenient for most people.
Other Virtual Machines are acceptable as long as a developer has access to it.
Testing in hardware is trickier than in a virtual machine. Most people drag out some ancient machine that hasn't seen the light of day in years and just pop in a ReactOS cd, thinking they are helping when they do this. I can tell you from first hand experience this is far from the truth. In order to do any good the machine should be able to accept another, well tested, operating system such as Windows 2000 or a popular linux distribution. This machine should of course meet the minimum system requirements for ReactOS. Take this moment to use the OS to write down as much hardware information as possible, even what IRQ and DMA channels it is using. You never know when this information will come in handy.
It is important to note that some BIOS have "special" settings for different operating systems. Please disable memory gaps, reserved memory areas and reserved memory caching, it goes without saying that this is not an OS/2 clone. If testing a USB mouse or keyboard you must enable USB compatibility mode, USB 2.0 is not supported, if possible disable it. Disconnect all non essential hardware, from experience i can say that ReactOS hanged on bootup because I had a USB-Camera attached.
ReactOS currently doesn't come with many drivers, just a few basic ones that (hopefully) take advantage of standard hardware. Most important is a VBE 2.0 compatible video card (VESA 1994 standard). For network connectivity you need a RTL 8139 or NE 2000 compatible card. If you really want to make other hardware work, you will need to get a windows 2000 driver for it.
Before you start any test you need to know what is going on, why and how the test is failing. The most important way to get information about an issue or bug is the debug output. When compiled with DBG = 1, and booted in debug mode ReactOS can send text to a file, the screen, or the serial port. For more information on configuring this please refer to Debugging
Getting more information
Before you can do anything you should be able to compile ReactOS. Meaning that you got a Git client to download the source and the Build Environment to compile it, and furthermore that you know how to use these. Please refer to these items wiki articles if you need help.
How to Test
When you start testing you should have a fresh ReactOS install, from a clean compile or rosbuild output cd. The usual way to proceed is just to boot into ReactOS and attempt by natural means to produce unwanted behavior, such as crashes, error messages graphical glitches or anything else that might NOT be considered a feature.
The following things can help to get useful testing results:
- Winetests. See also: Build winetests
- Write your own tests. See also: writing tests
- 3rd party apps: For functionality and regression testing of the most important 3rd party apps see Testing Central Testing Apps. These apps are significant in what they use or require of an operating system. If you know of an app that you feel should be in the list you should read this.
How to report a Bug
Main article: File Bugs
Bugs are filed in our Jira Bug-Tracking Software. Jira is used in this project to gather information about failing or misbehaving components and allows us to coordinate bug fixing efforts.
Regression testing should be always done in master branch
Let's explain how this work:
- Revision: Each time a change (fix/patch/small piece of code) is commited our buildbot creates a new Revision (ie: r55567).
- This way we can easily know which piece of code introduced a specific wrong behavior.
So testers doesn't use Releases as a reference (0.3.13 to 0.3.14 had more than 4000 different small-to-big changes) but daily revisions.
How to regress test?
- Since the migration to Git regression testing became even easier with git bisect feature
If you don't want to clone entire repository, you may follow these steps:
- Revisions can be downloaded from GetBuilds page, generally use bootcdXXXXX-dbg.isos. Dbg means DEBUG, useful to obtain DebugLogs.
- Find a revision that doesn't have the bug. (If you can't find it maybe it's not a regression but an unimplemented functionality or a known bug)
- Find a revision that has the bug.
- Then use Binary Search (basically pick a revision within the Working-NonWorking Gap)
- This way you will be able to find the exactly revision that introduced the bug. This specific revision is called Guilty Commit (as being the Guilty of this particular regression)
Knowing which revision is the Guilty one helps us to know which modified lines have awaken or produced a bug.
- New bug: Bug introduced by a specific commit
- Awaken bug: Bug that was in the OS and after a change reveals it. This is more natural than you ever expect