Difference between revisions of "Wpkg tests"

Revision as of 12:04, 17 August 2009

WPKG Test - Regression Test Suite for WPKG

Intention

WPKG test is meant to make the life of the developers easier: after making changes to wpkg.js, running WPKG test makes sure that wpkg.js is running properly as before.

Theoretically, when doing test driven development, introducing new features into wpkg.js should be done by first developing new tests.

But for now, we did the following: if we have a bug, we try to make tests which reproduce the bugs, and after correction the WPKG test runs without failure. So, next time, this bug cannot be reintroduced, or at least, we see that something is going wrong.

Download

As WPKG test is only for developers/testers of Wpkg, we assume, that it is no problem to get this from svn only:

SVN: svn co https://wpkg.svn.sourceforge.net/svnroot/wpkg wpkg/wpkg-test

Usage

Cd to wpkg/wpkg-test-Folder and run one of the following commands on your Windows test station (make sure you have administrator rights to install/remove the test applications):

> testall <suffix>        
  ... all tests are run from test1 ... testn containing tests 1 ... 99

> testset testN_name <suffix>
  ... a test set is run, going through test 1 ... test 99

> testone testN_name testNumber <suffix>
  ... one single test is run

<suffix> is for having different versions of wpkg.js in the wpkg-test folder. Example:

After saving latest wpkg.js to wpkg-test\wpkg-latest.js, you run the full test suite using testall latest on command line.

So I can have wpkg-111.js, wpkg-112RC9.js etc. and test them using testall 111 or testall 112RC9.

Directory structure

The test suite uses the following directory structure. After running one of the cmd files, you find the full debug logs in logs folder and the results (installed packages, xmls from error runs) in results folder.

|-- compare
|   |-- test1_dependencies
|   |   |-- 1
|   |   |   `-- wpkg.xml
|   |   |-- ...
|   |-- test2_check_or
|   |   |-- ...
|   |-- test3_cmdsetup
|   |   |-- ...
|-- logs
|-- packages
|   |-- testsetup.exe
|   |-- testsetup2.exe
|   |-- wpkgtest
|   |   |-- wpkgtest-1.0.0.msi
|   |   `-- wpkgtest-1.1.0.msi
|-- results
|-- test1_dependencies
|   |-- 1
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   |-- 2
|   |   |-- ...
|   |-- ...
|   `-- Checks.txt
|-- test2_check_or
|   |-- 10
|   |   |-- config.xml
|   |   |-- hosts.xml
|   |   |-- packages.xml
|   |   `-- profiles.xml
|   `-- Checks.txt
|-- test3_cmdsetup
|   |-- 1
|   |-- 2
|   |-- ...
|   `-- Check.txt

Within packages folder there are binaries for the installation tests, but most checks can be done using creation/deletion of simple text files.

All test folder names start with "test", followed by a number, an underscore and the short description of the tests.

Within a test folder, the tests are numbered from 1 ... 99, where at least test number 99 must make sure, that all used test files / test packages are removed, so that the next test has a clean test machine.

One test consists of a config.xml, hosts.xml, profiles.xml and packages.xml as you can see in above directory structure. The tests must not influence the local wpkg.xml of the system (if such exists) so the test suite can be run at a system, which itself is part of another WPKG installation. By setting <param name='settings_file_path' value='.' /> in each config.xml, we make sure that the wpkg.xml in the wpkg-test/ folder is used. Please, use the config.xml of an existing test, if you create new tests, to ensure this.

In the "checks.txt" files there should be a short description of each test:

Test1: dependencies

Test2: check_or

Test3: cmdsetup

The compare folder is essential for getting information if the different tests succeed or failed. It contains the correct "wpkg.xml" which can be compared with the local wpkg.xml of the test run.

When we design a new test, e.g. test0815_mytest\38, there is no compare\test0815_mytest\38\wpkg.xml file, so at first test run it is assumed that the created wpkg.xml is correct - it will be copied automatically to the compare folder.

Test run

testone.cmd

Invitation

Since Wpkg is getting more and more complex, tests for all possible situations/configurations/package checks etc. are essential.

So, I invite you to write tests for the test suite for all kinds of checks, so we can assure a good quality and backward compatibility for WPKG. Have a look at the existing tests