Can sb tell something about testman ?
Moderator: Moderator Team
-
- Posts: 420
- Joined: Tue Sep 16, 2008 1:13 pm
Can sb tell something about testman ?
http://www.reactos.org/testman/
I heard about it, but can please sb explain for community in a few words what it is, and how can one use it, and what is the benefit of it in testing.
is testman new or old ?
I heard about it, but can please sb explain for community in a few words what it is, and how can one use it, and what is the benefit of it in testing.
is testman new or old ?
- EmuandCo
- Developer
- Posts: 4731
- Joined: Sun Nov 28, 2004 7:52 pm
- Location: Germany, Bavaria, Steinfeld
- Contact:
Re: Can sb tell something about testman ?
Its a automatic test suite comparing winetests of every build to let us scan for regressions. I see no point for a normal user to work with it in any way.
ReactOS is still in alpha stage, meaning it is not feature-complete and is recommended only for evaluation and testing purposes.
If my post/reply offends or insults you, be sure that you know what sarcasm is...
If my post/reply offends or insults you, be sure that you know what sarcasm is...
-
- Posts: 420
- Joined: Tue Sep 16, 2008 1:13 pm
Re: Can sb tell something about testman ?
It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )
But why is the "red total fail counter" in later revisions higher ?
in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !
Should't this amount got to zero, there higher the revision is ?
But why is the "red total fail counter" in later revisions higher ?
in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !
Should't this amount got to zero, there higher the revision is ?
Re: Can sb tell something about testman ?
I guess this is because some tests are dependent on other tests. now imagine 100 tests are dependent on one test that failed all the time and with the revision 53xxx you can make the one test work but 60 of the 100 dependent tests fail. the result is, that you are one big step further, but the log shows you 59 more errors.
Re: Can sb tell something about testman ?
So, any tests that can't be performed get marked as a pass? Or they're just excluded? It would make more sense to me to mark any that can't be tested as an automatic fail.
Re: Can sb tell something about testman ?
A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.
Re: Can sb tell something about testman ?
I think that it would be much more correct to say that greem fields mean that no bugs have been found so far in this area... That means that if new tests are added they may reveal new bugs and more importantly, these tests do not test even 50% of the features of windows so it is very likely that there are sooooo many things that are not covered by these testswildschwein wrote:It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )
What these tests are really useful is that they let us know if a commit has caused a very bad result. For example is someone breaks a very important part of the os, the problem will be reflected in the results in testbot.
Re: Can sb tell something about testman ?
Not really, and I don't think anyone has quite answered this question yet. The reason it never seems to get closer to zero and there are more total failed tests is that new tests are added to testman quite often and this makes such one to one comparisons of revisions so far apart difficult. If you look at r53100 and r55842 you see that r55842 has 278,131 more tests run on it than r53100 did, new test categories, and some categories with less tests that have probably been moved to a different category.wildschwein wrote:It is right, that when all fields are green, that means, ROS 99% kompatibel to Windows (1% undocumented funkctions )
But why is the "red total fail counter" in later revisions higher ?
in about 53xxx in was about 5000, and in the 55xxx from now it is about 10 000 !
Should't this amount got to zero, there higher the revision is ?
Re: Can sb tell something about testman ?
As dark says, from time to time we update the winetest suites. So each time new tests are being added and reveals new bugs
Re: Can sb tell something about testman ?
Oh, oops. I missed the part about it being regression testing. Yeah, that makes sense.Z98 wrote:A test that is unrunnable and gets marked as an automatic fail is a false negative. If an API is outright missing or unimplemented, it is not a regression and is just noise when developers are trying to determine if their changes broke previously passed tests.
Who is online
Users browsing this forum: No registered users and 32 guests