Difference between revisions of "AHK Test Suite"

From ReactOS Wiki
Jump to: navigation, search
(Compiling and running test manually)
m
 
(49 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{stub}}
 
 
== What is AHK Test Suite ==
 
== What is AHK Test Suite ==
...
+
AHK Test Suite is a testing framework written in the [https://www.autohotkey.com/ AutoHotKey] scripting language.<br />
 +
Its purpose is to automatically test applications running under the ReactOS operating system.<br />
 +
<br />
 +
AHK Test Suite runs in the VBox virtual machine (VM).<br />
 +
Configure VBox as follows:
 +
* 512MB RAM assigned to VM;
 +
* VM has 800x600 screen resolution
 +
* Hardware:
 +
: OCZ Sata3 SSD
 +
* Virtual HDD: SATA AHCI 8GB
 +
* 10-20MB/s Internet connection
 +
* 60sec from last debug output until "[SYSREG] timeout" and OS is reset.
 +
* VM OS username: Administrator
 +
 
 
== How to download AHK Test Suite ==
 
== How to download AHK Test Suite ==
...
+
{| class="wikitable"
 +
|-
 +
| Note: for details on git see [[ReactOS_Git_For_Dummies|Git for Dummies]].
 +
|}
 +
 
 +
If you have [https://tortoisegit.org/ TortoiseGit],<br />
 +
1) Create a folder you want AHK Test Suite to be downloaded to (e.g. R:\src\ahk_tests).<br />
 +
2) Navigate your file manager to the created folder.<br />
 +
3) Right click and choose 'Git Clone'. 'Git clone' window will appear. Type '[https://github.com/<your%20username>/ahk_tests https://github.com/<your username>/ahk_tests]' in 'URL of repository' field and hit OK button and wait until things are downloaded.<br>
 +
[[File:Ahk clone.png|TGit clone]]
 +
<br>
 +
Then, download Applications and Media from [https://drive.google.com/folderview?id=0B2d0UoGDxVqTQjRJX3Z1ZnFhUUE&usp=sharing here].<br />
 +
{| class="wikitable"
 +
|-
 +
| Warning: these files are unmaintained. See [https://jira.reactos.org/browse/ONLINE-716 ONLINE-716].
 +
|}
 +
'''If something is missing you can ask Edijus or Caemyr in Mattermost channel [https://chat.reactos.org/reactos/channels/town-square Town Square].'''
 +
 
 
== How to see test results ==
 
== How to see test results ==
 
=== Online tests ===
 
=== Online tests ===
Line 13: Line 42:
 
5) Hit 'Compare selected Results' button. <br>
 
5) Hit 'Compare selected Results' button. <br>
 
Picture below demonstrates all the steps. <br>
 
Picture below demonstrates all the steps. <br>
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/testman.jpg
+
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/testman.jpg <br>
 +
Scroll down the newly opened page until you see tests that start with 'zz_' prefix. <br>
 +
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Test_results.jpg <br>
  
 
=== Compiling and running test manually ===
 
=== Compiling and running test manually ===
In order to run test you need to compile it. First, download the test (how to do so check titles above). When download is done, you need to compile using [http://www.autohotkey.com/ Ahk2Exe.exe] (AutoHotKey v1.0.48.05 is used when compiling ReactOS tests). If you are planing to compile using GUI tool, there is the steps: <br>
+
In order to run a test, you need to compile it.<br />
1) Run Ahk2Exe.exe; <br>
+
1) First, download the test (how to do so check titles above).<br />
2) When Ahk2Exe window appears, hit "Browse" button in in "Source (script file)" and open file that matches 'zz_*_test.ahk' file pattern (e.g. 'zz_firefox_2.0.0.20_test.ahk'); <br>
+
2) When download is done, you need to compile using Ahk2Exe.exe ([https://www.autohotkey.com/download/1.0/ AutoHotKey v1.0.48.05] is used when compiling ReactOS tests).<br />
3) Hit "> Convert <" button and dialog box saying 'Conversion complete' will appear. <br>
+
<br />
Picture below demonstrates all the steps. <br>
+
If you are planning to compile using GUI tool, there is the steps:<br>
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Ahk2Exe.jpg <br>
+
1) Run Ahk2Exe.exe.<br>
If compilation went successfully a 'zz_*_test.exe' (e.g. 'zz_firefox_2.0.0.20_test.exe') file will be created. There are dependencies. You gonna need exact the same test application version as described in test (e.g. 'Firefox 2.0.0.20'). Sadly, we do not provide any host links to test media we have for AHK tests. In case you managed to get everything you need (asked Caemyr, Edijus?) you need to create two folders in target machine 'Apps' and 'Media'. You put test program *.exe (e.g. 'Firefox 2.0.0.20 Setup.exe') in 'Apps' and rest of stuff (e.g. 'FF_2_Search_Arrow.jpg') in 'Media' folder. The compiled script executable goes into root folder. <br>
+
2) When Ahk2Exe window appears, hit "Browse" button in in "Source (script file)" and open file that matches 'zz_*_test.ahk' file pattern (e.g. 'zz_firefox_2.0.0.20_test.ahk').<br>
 +
3) Hit "> Convert <" button and dialog box saying 'Conversion complete' will appear.<br>
 +
Picture below demonstrates all the steps.<br>
 +
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Ahk2Exe.jpg<br>
 +
If compilation went successfully, a 'zz_*_test.exe' (e.g. 'zz_firefox_2.0.0.20_test.exe') file will be created. All .ahk files that are #included will be bundled into the output 'zz_*_test.exe' as well.<br />
 +
There are dependencies. You are going to need the exact same test application version as described in test (e.g. 'Firefox 2.0.0.20'). Download Applications and Media from [https://drive.google.com/folderview?id=0B2d0UoGDxVqTQjRJX3Z1ZnFhUUE&usp=sharing here].<br />
 +
You need to create two folders in target machine 'Apps' and 'Media'. You put test program *.exe (e.g. 'Firefox 2.0.0.20 Setup.exe') in 'Apps' and rest of stuff (e.g. 'FF_2_Search_Arrow.jpg') in 'Media' folder. The compiled script executable goes into root folder.<br />
 +
Note: All compiled scripts go into the same folder and all use the same 'Apps' and the same 'Media' folder.<br>
 
Picture below demonstrates folder hierarchy. <br>
 
Picture below demonstrates folder hierarchy. <br>
 
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Folders.jpg <br>
 
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Folders.jpg <br>
Line 32: Line 70:
  
 
== How to write new tests ==
 
== How to write new tests ==
..
+
If you want to write a test for a completely new application, use these as templates: <br>
 +
* [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/IrfanView/4.23/ IrfanView 4.23] as normal template; <br>
 +
* [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/SciTE/1.77/ SciTE 1.77] for 7-Zip self-extracting archives; <br>
 +
* [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/Audio%20Grabber/1.83%20SE/ Audio Grabber 1.83 SE] for installers that haves dialog where you have to choose setup language; <br>
 +
* [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/Java/6.25/ Java 6.25] for installers that use MsiExec; <br>
 +
* [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/Firefox/12.0/ Firefox 12.0] for applications that haves no nice control names. <br>
 +
Create two folders: 1 - Application name, 2 - Application version. E.g.: Firefox\2.0.0.20<br>
 +
This is so we can have the same application tested, but different version e.g. Firefox\2.0.0.20, Firefox\3.0.11, Firefox\12.0, etc. <br>
 +
There are different ways of making an application do things. You can use mouse to click controls and you can use keyboard. Sending keystrokes is not the most reliable thing to do, but sometimes applications do not have control names. In such case (e.g. [http://svn.reactos.org/svn/project-tools/trunk/ahk_tests/Firefox/12.0/ Firefox 12.0] case) send keystrokes (e.g. Alt+N to click 'Next' button, etc.).<br>
 +
Tests for installers: first of all you have to remove all previous versions of application. Use uninstaller with silent switches. If not possible, just delete hard-coded default path and settings if any. The path and registry modifications can be detected on installation using tools such as [http://technet.microsoft.com/cs-cz/sysinternals/bb896645.aspx ProcMon]<br>
 +
Let's say some window takes long to close (some installers takes some time to extract files). Do not use something like ''WinWaitClose, Window Caption, Window Text, 60'', because window may never close because of bugs in ReactOS code and some windows just pop-out. Instead, write a loop. Something like this: <br>
 +
<syntaxhighlight lang="autohotkey">
 +
TestsInfo("'Window Caption (Window Tex)' window appeared, waiting for it to close.")
 +
iTimeOut := 60
 +
while iTimeOut > 0
 +
{
 +
    IfWinActive, Window Caption, Window Text    {
 +
        WinWaitClose, Window Caption, Window Tex, 1
 +
        iTimeOut--
 +
    }
 +
    else
 +
    {
 +
        WinGetActiveTitle, ActiveWndTitle
 +
        TestsInfo("'" ActiveWndTitle "' window poped-up.")
 +
        break ; exit the loop if something poped-up
 +
    }
 +
}
 +
 
 +
WinWaitClose, Window Caption, Window Tex, 1
 +
if ErrorLevel
 +
    TestsFailed("'Window Caption (Window Tex)' window failed to close (iTimeOut=" iTimeOut ").")
 +
else
 +
    TestsOK("'Window Caption (Window Tex)' window closed (iTimeOut=" iTimeOut ").")
 +
</syntaxhighlight>
 +
We want tests to consume as little time as possible. Sysreg will time out (''[SYSREG] timeout'') if there is no debug output in 60 seconds.<br>
 +
Do not create too many indentations - 5-8 is the number we would like to see.<br>
 +
If you do 'SendInput, Some text', do 'ControlGetText' or <br>
 +
<syntaxhighlight lang="autohotkey">
 +
clipboard = ; clean the clipboard
 +
SendInput, ^a ; Ctrl+A
 +
SendInput, ^c ; Ctrl+C
 +
ClipWait, 2
 +
if ErrorLevel
 +
    TestsFailed("Unable to copy text onto clipboard.")
 +
else
 +
{
 +
    szExpectedString = Some text
 +
    IfNotInString, clipboard, %szExpectedString% ; You want clipboard to go first
 +
        TestsFailed("Copied text onto clipboard and got unexpected results. Is '" clipboard "', should be '" szExpectedstring "'.")
 +
    else
 +
        TestsOK("Filled 'Some field name' with '" szExpectedString "' in 'Some window (window text)'.")
 +
}
 +
</syntaxhighlight>
 +
to make sure you sent text to correct window and not some error box.
 +
File name and specific test name must match. For example test name '2.address_bar' and file name 'address_bar.ahk'. Test and file names are case sensitive. If you are trying to access some file, make sure it exists first. If you close an application, make sure that not only window has closed, but application process too.
 +
 
 +
== How to get control names ==
 +
You can use Spy++ application or this AHK script: <br>
 +
<syntaxhighlight lang="autohotkey">
 +
DetectHiddenText, Off ; Hidden text is not detected
 +
Sleep, 3500
 +
 
 +
 
 +
WinGetTitle, ActiveWndTitle, A
 +
WinGetText, ActiveWndText, %ActiveWndTitle%
 +
ControlGetFocus, FocusedName, %ActiveWndTitle%
 +
ControlGetText, FocusedText, %FocusedName%, %ActiveWndTitle%
 +
MsgBox, ActiveWnd: %ActiveWndTitle% `nFocused: %FocusedText% `nText: %ActiveWndText%
 +
 
 +
 
 +
; This example allows you to move the mouse around to see
 +
; the title of the window currently under the cursor:
 +
#Persistent
 +
SetTimer, WatchCursor, 100
 +
return
 +
 
 +
WatchCursor:
 +
MouseGetPos, OutputVarX, OutputVarY, id, control
 +
WinGetTitle, title, ahk_id %id%
 +
WinGetClass, class, ahk_id %id%
 +
ControlGetText, FocusedText, %control%, A
 +
ControlGet, ControlHwnd, Hwnd,, %control%, A
 +
PixelGetColor, color, %OutputVarX%, %OutputVarY%
 +
SetTitleMatchMode, 2
 +
IfWinNotActive, Notepad
 +
    ToolTip, ahk_id %id%`nahk_class %class%`n%title%`nControl: %control%`nControlText: %FocusedText%`nMousePos: %OutputVarX%x%OutputVarY%`nColor: %color%
 +
return
 +
</syntaxhighlight>
 +
 
 +
There is how all stuff works:
 +
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/seku_diagrama.jpg<br>
 +
 
 +
[[Category:Tests]]

Latest revision as of 23:13, 17 June 2021

What is AHK Test Suite

AHK Test Suite is a testing framework written in the AutoHotKey scripting language.
Its purpose is to automatically test applications running under the ReactOS operating system.

AHK Test Suite runs in the VBox virtual machine (VM).
Configure VBox as follows:

  • 512MB RAM assigned to VM;
  • VM has 800x600 screen resolution
  • Hardware:
OCZ Sata3 SSD
  • Virtual HDD: SATA AHCI 8GB
  • 10-20MB/s Internet connection
  • 60sec from last debug output until "[SYSREG] timeout" and OS is reset.
  • VM OS username: Administrator

How to download AHK Test Suite

Note: for details on git see Git for Dummies.

If you have TortoiseGit,
1) Create a folder you want AHK Test Suite to be downloaded to (e.g. R:\src\ahk_tests).
2) Navigate your file manager to the created folder.
3) Right click and choose 'Git Clone'. 'Git clone' window will appear. Type '<your%20username>/ahk_tests https://github.com/<your username>/ahk_tests' in 'URL of repository' field and hit OK button and wait until things are downloaded.
TGit clone
Then, download Applications and Media from here.

Warning: these files are unmaintained. See ONLINE-716.

If something is missing you can ask Edijus or Caemyr in Mattermost channel Town Square.

How to see test results

Online tests

If you want to see results of online tests you need:
1) Enter http://reactos.org/testman/ in your browser address;
2) Specify ReactOS revision number you want to check results of;
3) Enter 'CMake_x86_GCCWin' in 'Source' field;
4) Select up to five revisions using checkboxes provided;
5) Hit 'Compare selected Results' button.
Picture below demonstrates all the steps.
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/testman.jpg
Scroll down the newly opened page until you see tests that start with 'zz_' prefix.
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Test_results.jpg

Compiling and running test manually

In order to run a test, you need to compile it.
1) First, download the test (how to do so check titles above).
2) When download is done, you need to compile using Ahk2Exe.exe (AutoHotKey v1.0.48.05 is used when compiling ReactOS tests).

If you are planning to compile using GUI tool, there is the steps:
1) Run Ahk2Exe.exe.
2) When Ahk2Exe window appears, hit "Browse" button in in "Source (script file)" and open file that matches 'zz_*_test.ahk' file pattern (e.g. 'zz_firefox_2.0.0.20_test.ahk').
3) Hit "> Convert <" button and dialog box saying 'Conversion complete' will appear.
Picture below demonstrates all the steps.
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Ahk2Exe.jpg
If compilation went successfully, a 'zz_*_test.exe' (e.g. 'zz_firefox_2.0.0.20_test.exe') file will be created. All .ahk files that are #included will be bundled into the output 'zz_*_test.exe' as well.
There are dependencies. You are going to need the exact same test application version as described in test (e.g. 'Firefox 2.0.0.20'). Download Applications and Media from here.
You need to create two folders in target machine 'Apps' and 'Media'. You put test program *.exe (e.g. 'Firefox 2.0.0.20 Setup.exe') in 'Apps' and rest of stuff (e.g. 'FF_2_Search_Arrow.jpg') in 'Media' folder. The compiled script executable goes into root folder.
Note: All compiled scripts go into the same folder and all use the same 'Apps' and the same 'Media' folder.
Picture below demonstrates folder hierarchy.
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Folders.jpg
Now, all you need is to start test executable by passing test name via command line (e.g. 'D:\AHK_Tests\zz_firefox_2.0.0.20_test.exe 1.install').
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/Start_Test_CMD.jpg
Results will be displayed in debug output. Debug output can be monitored with DebugView application.
Picture below demonstrates how results are displayed in DebugView application.
https://sites.google.com/site/thricesoftas/Home/stuff/reactos/DebugView.jpg

How to write new tests

If you want to write a test for a completely new application, use these as templates:

Create two folders: 1 - Application name, 2 - Application version. E.g.: Firefox\2.0.0.20
This is so we can have the same application tested, but different version e.g. Firefox\2.0.0.20, Firefox\3.0.11, Firefox\12.0, etc.
There are different ways of making an application do things. You can use mouse to click controls and you can use keyboard. Sending keystrokes is not the most reliable thing to do, but sometimes applications do not have control names. In such case (e.g. Firefox 12.0 case) send keystrokes (e.g. Alt+N to click 'Next' button, etc.).
Tests for installers: first of all you have to remove all previous versions of application. Use uninstaller with silent switches. If not possible, just delete hard-coded default path and settings if any. The path and registry modifications can be detected on installation using tools such as ProcMon
Let's say some window takes long to close (some installers takes some time to extract files). Do not use something like WinWaitClose, Window Caption, Window Text, 60, because window may never close because of bugs in ReactOS code and some windows just pop-out. Instead, write a loop. Something like this:

TestsInfo("'Window Caption (Window Tex)' window appeared, waiting for it to close.")
iTimeOut := 60
while iTimeOut > 0
{
    IfWinActive, Window Caption, Window Text    {
        WinWaitClose, Window Caption, Window Tex, 1
        iTimeOut--
    }
    else
    {
        WinGetActiveTitle, ActiveWndTitle
        TestsInfo("'" ActiveWndTitle "' window poped-up.")
        break ; exit the loop if something poped-up
    }
}

WinWaitClose, Window Caption, Window Tex, 1
if ErrorLevel
    TestsFailed("'Window Caption (Window Tex)' window failed to close (iTimeOut=" iTimeOut ").")
else
    TestsOK("'Window Caption (Window Tex)' window closed (iTimeOut=" iTimeOut ").")

We want tests to consume as little time as possible. Sysreg will time out ([SYSREG] timeout) if there is no debug output in 60 seconds.
Do not create too many indentations - 5-8 is the number we would like to see.
If you do 'SendInput, Some text', do 'ControlGetText' or

clipboard = ; clean the clipboard
SendInput, ^a ; Ctrl+A
SendInput, ^c ; Ctrl+C
ClipWait, 2
if ErrorLevel
    TestsFailed("Unable to copy text onto clipboard.")
else
{
    szExpectedString = Some text
    IfNotInString, clipboard, %szExpectedString% ; You want clipboard to go first
        TestsFailed("Copied text onto clipboard and got unexpected results. Is '" clipboard "', should be '" szExpectedstring "'.")
    else
        TestsOK("Filled 'Some field name' with '" szExpectedString "' in 'Some window (window text)'.")
}

to make sure you sent text to correct window and not some error box. File name and specific test name must match. For example test name '2.address_bar' and file name 'address_bar.ahk'. Test and file names are case sensitive. If you are trying to access some file, make sure it exists first. If you close an application, make sure that not only window has closed, but application process too.

How to get control names

You can use Spy++ application or this AHK script:

DetectHiddenText, Off ; Hidden text is not detected
Sleep, 3500


WinGetTitle, ActiveWndTitle, A
WinGetText, ActiveWndText, %ActiveWndTitle%
ControlGetFocus, FocusedName, %ActiveWndTitle%
ControlGetText, FocusedText, %FocusedName%, %ActiveWndTitle%
MsgBox, ActiveWnd: %ActiveWndTitle% `nFocused: %FocusedText% `nText: %ActiveWndText%


; This example allows you to move the mouse around to see
; the title of the window currently under the cursor:
#Persistent
SetTimer, WatchCursor, 100
return

WatchCursor:
MouseGetPos, OutputVarX, OutputVarY, id, control
WinGetTitle, title, ahk_id %id%
WinGetClass, class, ahk_id %id%
ControlGetText, FocusedText, %control%, A
ControlGet, ControlHwnd, Hwnd,, %control%, A
PixelGetColor, color, %OutputVarX%, %OutputVarY%
SetTitleMatchMode, 2
IfWinNotActive, Notepad
    ToolTip, ahk_id %id%`nahk_class %class%`n%title%`nControl: %control%`nControlText: %FocusedText%`nMousePos: %OutputVarX%x%OutputVarY%`nColor: %color%
return

There is how all stuff works: https://sites.google.com/site/thricesoftas/Home/stuff/reactos/seku_diagrama.jpg