[ros-kernel] CVS regressions and checkin system idea - DistributedRegression Testing

[Casper Hornstrup]

> -----Original Message-----
> From: ros-kernel-bounces at reactos.com 
> [mailto:ros-kernel-bounces at reactos.com] On Behalf Of Steven Edwards
> Sent: 15. april 2004 00:59
> To: ros-kernel at reactos.com
> Subject: [ros-kernel] CVS regressions and checkin system idea 
> - DistributedRegression Testing
> 
> Hello All,
> I have been thinking a little about the problems with CVS or 
> any other version control system if your patch compiles clean 
> then it must also not make a regression test fail. This is a 
> great idea except for the performance and time hit its going 
> to take. If we implement a strict testing system on the 
> server side then we are going to need a freaking large 
> cluster to commit->build->test->merge and there is going to 
> be a real lag in development.
> 
> /me is thinking outside of the box here again but stay with 
> me for a sec.....
> 
> We really a system where the developer is forced to check the 
> patch and run the regression tests before merging changes. 
> Does anyone think it would be possible to do something over 
> SSH+CVS or SSH+SVN were we have a build daemon running on the 
> client that forces you to rebuild/reg-test before it will 
> pass you a token allowing you to merge?
> 
> This sort of system would keep the load off the 
> server/cluster and force the developer to better test the 
> code and they could find the bugs and fix them faster. The 
> load of the build/test/commit cycle needs to be distributed 
> and if we dont try and distribute that load to the developers 
> then we are going to need a freaking huge cluster to handle 
> commits as this project grows.
> 
> Thanks
> Steven

I'm not sure about the "freaking large cluster". At least not at
this time. My estimates are that it will take less than 30 minutes
with a single 2.4GHz machine to complete the cycle. This means that
the system would be able to handle 48 patches a day. Statistics
from the last 7 days are:

Day	      No. Patches
Friday      15
Thursday    9
Wednesday   13
Tuesday     20
Monday      31
Sunday      14
Saturday    14

Peak        31
Average     17

So even a 2.4GHz machine would be enough to handle our current load.
Of course, as we add more functionality and tests to ReactOS, it
will take longer to complete the cycle so eventually we will need
more build machines. Later we will also need to support more target
architectures.

Because it is difficult for all developers to use exactly the same
build environment, their development machine should not decide what
goes into the stable branches. For instance when a new MinGW comes
out, some might want to upgrade right away and others might wait and
some might not upgrade at all. It requires too much dicipline to do
this. There need to be setup a pool of build machines using strict
guidelines.

I hope the system can eventually handle a scenario where the developer
only need to run the tests on his development machine if he is actually
developing a test. If you've just fixed a bug, then just make sure ReactOS
builds for you after applying your bugfix and have the integration system
notify you if your change broke some other test or caused ReactOS to not
build in a particular configuration. 

My box to be used for testing is expected to be online on Monday. I'll
need a few days to set it up, then we can begin testing and figure out
what works, what doesn't work, and what would be nice to have. It is a
Windows box, so it can only test the Windows/MinGW configuration. It
would be nice if eventually someone had a Linux machine that could be
used to build using Linux/MinGW configuration, but for the testing
period this is not required (actually I haven't implemented support for
multiple build platforms yet).

It has two CPUs, so if anyone knows how to fix our build system so that we
can use make -j2, then it will be able to use them both, nearly doubling the
speed of the integration process.

Casper