Difference between revisions of "Wpkg tests"

Revision as of 11:40, 17 August 2009

WPKG Test - Regression Test Suite for WPKG

Intention

WPKG test is meant to make the life of the developers easier: after making changes to wpkg.js, running WPKG test makes sure that wpkg.js is running properly as before.

Theoretically, when doing test driven development, introducing new features into wpkg.js should be done by first developing new tests.

But for now, we did the following: if we have a bug, we try to make tests which reproduce the bugs, and after correction the WPKG test runs without failure. So, next time, this bug cannot be reintroduced, or at least, we see that something is going wrong.

Download

As WPKG test is only for developers/testers of Wpkg, we assume, that it is no problem to get this from svn only:

SVN: svn co https://wpkg.svn.sourceforge.net/svnroot/wpkg wpkg/wpkg-test

Usage

Cd to wpkg/wpkg-test-Folder and run one of the following commands on your Windows test station (make sure you have administrator rights to install/remove the test applications):

testall <suffix>

testset testN_name <suffix>

testone testN_name testNumber <suffix>

<suffix> is for having different versions of wpkg.js in the wpkg-test folder. Example:

After saving latest wpkg.js to wpkg-test\wpkg-latest.js, you run the full test suite using testall latest on command line.

So I can have wpkg-111.js, wpkg-112RC9.js etc. and test them using testall 111 or testall 112RC9.

Directory structure

The test suite uses the following directory structure. After running one of the cmd files, you find the full debug logs in logs folder and the results (installed packages, xmls from error runs) in results folder.

|-- compare
|   |-- test1_dependencies
|   |   |-- 1
|   |   |   `-- wpkg.xml
|   |   |-- ...
|   |-- test2_check_or
|   |   |-- ...
|   |-- test3_cmdsetup
|   |   |-- ...
|-- logs
|-- packages
|   |-- testsetup.exe
|   |-- testsetup2.exe
|   |-- wpkgtest
|   |   |-- wpkgtest-1.0.0.msi
|   |   `-- wpkgtest-1.1.0.msi
|-- results
|-- test1_dependencies
|   |-- 1
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   |-- 2
|   |   |-- ...
|   |-- ...
|   `-- Checks.txt
|-- test2_check_or
|   |-- 10
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   `-- Checks.txt
|-- test3_cmdsetup
|   |-- 1
|   |-- 2
|   |-- ...
|   `-- Check.txt

Within packages folder there are binaries for the installation tests, but most checks can be done using creation/deletion of simple text files.

All test folder names start with "test", followed by a number, an underscore and the short description of the tests.

Within a test folder, the tests are numbered from 1 ... 99, where at least test number 99 must make sure, that all used test files / test packages are removed, so that the next test has a clean test machine.

In the "checks.txt" files should be a short description of each test:

Test1: dependencies

Test2: check_or

Test3: cmdsetup