Can sb tell something about testman ?

[wildschwein]

http://www.reactos.org/testman/

I heard about it, but can please sb explain for community in a few words what it is, and how can one use it, and what is the benefit of it in testing.

is testman new or old ?

[EmuandCo]

Its a automatic test suite comparing winetests of every build to let us scan for regressions. I see no point for a normal user to work with it in any way.

[wildschwein]

It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )

But why is the "red total fail counter" in later revisions higher ?

in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !

Should't this amount got to zero, there higher the revision is ?

[mattsharp]

I guess this is because some tests are dependent on other tests. now imagine 100 tests are dependent on one test that failed all the time and with the revision 53xxx you can make the one test work but 60 of the 100 dependent tests fail. the result is, that you are one big step further, but the log shows you 59 more errors.

[milon]

So, any tests that can't be performed get marked as a pass? Or they're just excluded? It would make more sense to me to mark any that can't be tested as an automatic fail.

[Z98]

A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.

[Smiley]

It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )

I think that it would be much more correct to say that greem fields mean that no bugs have been found so far in this area... That means that if new tests are added they may reveal new bugs and more importantly, these tests do not test even 50% of the features of windows so it is very likely that there are sooooo many things that are not covered by these tests

What these tests are really useful is that they let us know if a commit has caused a very bad result. For example is someone breaks a very important part of the os, the problem will be reflected in the results in testbot.

[dark]

It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )

But why is the "red total fail counter" in later revisions higher ?

in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !

Should't this amount got to zero, there higher the revision is ?

Not really, and I don't think anyone has quite answered this question yet. The reason it never seems to get closer to zero and there are more total failed tests is that new tests are added to testman quite often and this makes such one to one comparisons of revisions so far apart difficult. If you look at r53100 and r55842 you see that r55842 has 278,131 more tests run on it than r53100 did, new test categories, and some categories with less tests that have probably been moved to a different category.

[vicmarcal]

As dark says, from time to time we update the winetest suites. So each time new tests are being added and reveals new bugs

[milon]

A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.

Oh, oops. I missed the part about it being regression testing. Yeah, that makes sense.