Wpkg tests

WPKG Test - Regression Test Suite for WPKG

Intention

WPKG test is meant to make the life of the developers easier: after making changes to wpkg.js, running WPKG test makes sure that wpkg.js is running properly as before.

Theoretically, when doing test driven development, introducing new features into wpkg.js should be done by first developing new tests.

But for now, we did the following: if we have a bug, we try to make tests which reproduce the bug, and after correction the WPKG test runs without failure. So, next time, this bug cannot be reintroduced, or at least, we see that something is going wrong.

Download

As WPKG test is only for developers/testers of Wpkg, we assume, that it is no problem to get this from svn only:

SVN: svn co https://wpkg.svn.sourceforge.net/svnroot/wpkg wpkg/wpkg-test

Usage

Cd to wpkg/wpkg-test-Folder and run one of the following commands on your Windows test station (make sure you have administrator rights to install/remove the test applications):

> testall <suffix>        
  ... all tests are run from test1 ... testn containing tests 1 ... 99

> testset testN_name <suffix>
  ... a test set is run, going through test 1 ... test 99

> testone testN_name testNumber <suffix>
  ... one single test is run

<suffix> is for having different versions of wpkg.js in the wpkg-test folder. Example:

After saving latest wpkg.js to wpkg-test\wpkg-latest.js, you run the full test suite using testall latest on command line.

So I can have wpkg-111.js, wpkg-112RC9.js etc. and test them using testall 111 or testall 112RC9.

> testinit
  ... deletes all log files and resets the wpkg.xml in wpkg-test folder

Remark: This does not clean installed packages as "test 99"s should do. So when you are implementing/changing tests, you have to make sure yourself that there are not leftovers from the test packages.

Directory structure

The test suite uses the following directory structure. After running one of the cmd files, you find the full debug logs in logs folder and the results (installed packages, xmls from error runs) in results folder.

|-- compare
|   |-- test1_dependencies
|   |   |-- 1
|   |   |   `-- wpkg.xml
|   |   |-- ...
|   |-- test2_check_or
|   |   |-- ...
|   |-- test3_cmdsetup
|   |   |-- ...
|-- logs
|-- packages
|   |-- testsetup.exe
|   |-- testsetup2.exe
|   |-- wpkgtest
|   |   |-- wpkgtest-1.0.0.msi
|   |   `-- wpkgtest-1.1.0.msi
|-- results
|-- test1_dependencies
|   |-- 1
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   |-- 2
|   |   |-- ...
|   |-- ...
|   `-- Checks.txt
|-- test2_check_or
|   |-- 10
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   `-- Checks.txt
|-- test3_cmdsetup
|   |-- 1
|   |-- 2
|   |-- ...
|   `-- Check.txt

Within packages folder there are binaries for the installation tests, but most checks can be done using creation/deletion of simple text files.

All test folder names start with "test", followed by a number, an underscore and the short description of the tests.

Within a test folder, the tests are numbered from 1 ... 99, where at least test number 99 must make sure, that all used test files / test packages are removed, so that the next test has a clean test machine.

One test consists of a config.xml, hosts.xml, profiles.xml and packages.xml as you can see in above directory structure. The tests must not influence the local wpkg.xml of the system (if such exists) so the test suite can be run at a system, which itself is part of another WPKG installation. By setting <param name='settings_file_path' value='.' /> in each config.xml, we make sure that the wpkg.xml in the wpkg-test/ folder is used. Please, use the config.xml of an existing test, if you create new tests, to ensure this.

In the "checks.txt" files there should be a short description of each test:

Test1: dependencies

Test2: check_or

Test3: cmdsetup

The compare folder is essential for getting information about success or failure of the different tests. It contains the correct "wpkg.xml" which is automagically compared with the local wpkg.xml of the test run.

When we design a new test, e.g. test0815_mytest\38, there is no compare\test0815_mytest\38\wpkg.xml file, so at first test run it is assumed that the created wpkg.xml is correct - it will be copied automatically to the compare folder.

Test run

Running the tests shows only one line for each test, e.g. Doing test1_dependencies 10....

If a test goes wrong (which means wpkg.xml and compare\testN_name\NN\wpkg.xml are not identical) a line stating the error is shown: Test test1_dependencies 10... failed.

If there are no errors - fine!

When you see error lines, please have a look at the corresponding logs/results files in the logs/results folders to get an idea what's wrong.

Invitation

Since Wpkg is getting more and more complex, tests for all possible situations/configurations/package checks etc. are essential.

So, I invite you to write tests for the test suite for all kinds of checks, so we can assure a good quality and backward compatibility for WPKG.

Have a look at the existing tests and make sure, that tests 10/20/30 ... 99 cleanup the test system. Thanx in advance!

Please, send new tests and suggestions to: nospam_ft at smi-softmark dot de

--Ftrojahn 14:14, 17 August 2009 (CEST)