Testman Blacklist
Moderator: Moderator Team
Testman Blacklist
Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?
-
- Posts: 420
- Joined: Tue Sep 16, 2008 1:13 pm
Re: Testman Blacklist
please explain, and provide some links, that testers can understand the problem. thanks !
Re: Testman Blacklist
Erratic behaviors are related to potential race conditions,memory bugs,etc...dark wrote:Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?
They are difficult to catch, as they are inherent to the operating system so there isnt a clear way about how testers can help in this case.
You can help with CRASHED tests, reporting why they are crashing(run that particular winetest and obtaining a debuglog).Please check they haven't been reported yet
Re: Testman Blacklist
There's a handful of tests on reactos.org/testman that give random results and crashes so much that they are no longer included in the totals. They are marked by a black outline now.wildschwein wrote:please explain, and provide some links, that testers can understand the problem. thanks !
If we can only help on crashes, then it appears we would have to monitor revisions in testman for crashes, download dbgwin build from /getbuilds, run it to try to cause crash, then get debug log and create or add to bug report.
edit: and here's a wiki page with some relevant information: http://www.reactos.org/wiki/Winetest_crashes
Tried this: can get random results, no crashes. thinking crash is merely a function of memory leaking and would need to know how testman runs scripts (as batch file?) and then do the same when testing.
Re: Testman Blacklist
We will eventually make bug reports for all flaky tests and testers can then check if the erratic behavior is caused by a particular revision and report back. But I'm afraid some of those tests are random "by design". Wine has a similar blacklist of misbehaving tests too by the way.dark wrote:Is there anything testers can do to help resolve the erratic behavior of the blacklisted tests in testman?
Random crashes are usually caused by underlaying problems which may be very hard to find. "Random" is the keyword here. When assembling the initial black list I tried to replicate the issues myself without any success (using the same test cd iso as the test bot).
The tests are ran by an app called rosautotest, you can find it in rostests folder in the main repo.dark wrote:Tried this: can get random results, no crashes. thinking crash is merely a function of memory leaking and would need to know how testman runs scripts (as batch file?) and then do the same when testing.
Who is online
Users browsing this forum: Ahrefs [Bot], Google [Bot] and 33 guests