XTest Configuration

Version: 1.5
Last update: January 31, 2006
Author: Milan Kubec , Libor Martinek, Martin Brehovsky

1. Test Attributes
1. Logical Operators
2. Executor
3. Compiler
4. Default executor, compiler and result processor
5. Testbag priority and reusing IDE userdir
6. Testbag timeout
7. How to execute tests
8. Test set, patternset, patternset attributes
9. Expected fail
   
10. Order of testsuites
11. Results Processor
12. Generate config for failed or excluded tests
    

Abstract: This document describes how to organize and configure execution of tests under xtest framework.

XTest framework didn't have the possibility to define set of tests, which will be executed in some flexible way. We've found that convenient way would be to us a xml configuration file, which would allow to define number of test configurations along with test attributes, which will have to be satisfied. Each group of tests defined by such config will have own executor and own results processor. This layout allows very flexible way of test execution.

There is some new terminology needed to understand how new configuration framework works. The set of tests defined by one xml file will be called test group. Test group is then divided into test bags and each test bag can contain one or more test sets . Name of config file will be based on major test type defined by the config file. Enough theory, here is an example:

Let's say we have tests for openidex module. Tests should be stored in  followingstructure:

openidex/ |- test/ |- unit/ |- src/org/... |- perf/ |- src/org/... |- regr/ |- src/org/... |- mytests |- src/org/... |- build.xml |- cfg-unit.xml |- cfg-perf.xml |- cfg-regr.xml |- cfg-mytests.xml

   Folders such as unit/, perf/, regr/, serve just as containers for similar or related tests. Not as folders saying what type of tests is below. The actual test configuration is created by cfg-[name_of_config].xml files. You can define test groups across all directories (couple of tests form unit/ some from mytests/ and so on), you can assign test attributes to given testbags and so on. Here is example of such config file for unit tests - cfg-unit.xml:

<mconfig name="openidex unit test config"> <testbag testattribs="stable" executor="code" name="testbag_1" resultsprocessor="unit"> <testset dir="unit/src"> <patternset> <include name="org/openidex/nodes/looks/*.class"/> <exclude name="org/openidex/nodes/looks/LookNodeTest.class/testGetActions"/> <exclude name="org/openidex/nodes/looks/LookNodeTest.class/testGetPropertySets"/> </patternset> </testset> </testbag> <testbag testattribs="failing" executor="code" compiler="special-compiler" name="testbag_2" resultsprocessor="unit"> <testset dir="unit/src"> <patternset> <include name="org/openidex/nodes/looks/LookNodeTest.class/testGetActions"/> <include name="org/openidex/nodes/looks/LookNodeTest.class/testGetPropertySets"/> </patternset> </testset> </testbag> <executor name="code" antfile="build-unit.xml" target="run_tests_in_jvm"/> <compiler name="default-compiler" default="true" antfile="build-unit.xml" target="default-compiler"/> <compiler name="special-compiler" antfile="build-unit.xml" target="special-compiler"/>   </mconfig>

Note: Name of the file cfg-unit.xml is mandatory for test group which will be referenced as unit test type or configuration.

Test attributes

   In previous piece of config file you can see that test group consists of two test bags, each test bag groups tests with the same executor (= code) but different attributes. One has attribute "stable" and second has attribute "failing". These attributes are intended to distinguish between tests, which although are of the same sort, can satisfy different attributes. In this case one test bag contains stable tests which run in code execution and the second contains failing tests for code execution. Using attributes has no restrictions, you can assign any attribute to some test bag. Then if you want to run tests with some particular attributes you just specify them and those tests, which satisfy given attributes will be executed.

Logical operators

   During design work we've found out that only specifing name of attribute is not sufficient to flexibly define test attributes. Imagine situation when you want some attribute to be specified and some not, e.g. tests can run against one database, but not against other one. From that reason simple logical operators were added to test attributes definition. Operators are: AND, OR, NOT, !, (), comma (case insensitive). Comma is equivalent to OR. This allows you to define e.g.:

testattribs="(stable OR experimental) AND oracle AND NOT pointbase"

Test Executor

   Test attribute "code" is just a mark, the actual executor is defined by attribute "executor". Value of this attribute is name of executor, which is then defined as separate element in the same file (see cfg-unit.xml example above).  Executor in this terminology is just name of ant script and target in that script. This script will be executed and set of selected tests (from all testbags, which satisfy given attributes) will be passed to it as ant property, along with proper classpath to execute tests. Also some other properties are set, such as name of test bag. E.g. for running tests inside IDE, you create new executor and provide its name and element in config file. That executor (ant script) will be responsible for setting up IDE and execution selected tests inside the IDE. The

   This way of executors definition allows to define executor with special functionality (e.g. startup IDE multiple times) by having empty <testbag> element with the special executor (which is just ant script doing special things). Such special tests can of course define own attributes and own results processors.

The simplest executor definition  is shown in the following example:

<target name="run_tests_in_jvm">
     <executeTests  pluginName="jvm" classpath="${netbeans.home}/lib/openide.jar, ${netbeans.home}/lib/core.jar"/>
</target>   

As you can see in the example, running the tests is a matter of one ant command, called 'executeTests'.

Note: code and ide executors will be provided by default as well as default results processor for unit tests.

Test Compiler

   As well each testbag has executor it can also have compiler. Value of attribute compiler is name of compiler which is defined as separate element. Compiler is just target from ant script which is used for compiling . Having more compilers can be useful for cases when each testtype or testbag has different requirements for compilation.

The simplest compiler definition  is shown in the following example:

<target name="default-compiler">
     <buildTests srcDir="unit/src" classpath="${netbeans.home}/lib/openide.jar"/>
</target>   

As you can see in the example, the simplest compiler takes only srcDir argument and classpath. Please note, JUnit framework and NbJUnit extensions are added automatically to the classpath.

Default executor, compiler and result processor

   One executor, compiler or result processor can have attribute default with value true. Such executor (compiler, result processor) will be used for testbags without defined executor (compiler, result processor).

Testbag priority and reusing IDE userdir

   If you want to guarantee order of testbag execution you can use testbag priority. Each testbag can have optional attribute "prio" with integer value. Testbag with less number is executed before testbag with greater value. Testbags without prio attribute is executed as last. Testbag priority is useful together with reusing IDE userdir. To reuse IDE userdir you have to write own second executor in which you have to set property xtest.ide.reuse.userdir to "true".

<testbag testattribs="stable" executor="ide" prio="1" name="testbag_1">
   ...
</testbag>      

<testbag testattribs="stable" executor="ide" prio="2" name="testbag_2">
   ...
</testbag>      

<executor name="ide" antfile="build.xml" target="runidetest"/>
<executor name="ide-2ndstart" antfile="build.xml" target="runidetest-2ndstart"/>    

Testbag timeout

   You can assign to each testbag amount of time (timeout) which should not be exceeded. If testbag runs over this time it will be terminated. To set timeout , add attribute 'timeout' with value in milliseconds to element testbag.

<testbag testattribs="stable" executor="ide" timeout="600000" name="testbag_1">
   ...
</testbag>      

How to execute tests

   If you want to execute tests defined in some cfg file as e.g. validation test group (or configuration or testtype) you need to specify property:

-Dxtest.testtype=validation

and then you should also specify attributes which you want to be satisfied by executed tests, so e.g.:

-Dxtest.attribs=stable.

Finally you might want to execute tests just for selected executor(s). This can be specified by property:

-Dxtest.executors=code.

Then cfg-validation.xml will be loaded, test bags will be compared with specified attributes and selected tests will be performed by correct executor. These properties can be defined either in build.xml file or as command line arguments. Default settings for these properties are xtest.testtype=unit, xtest.testattribs=stable and executors are defined all available for given testtype.

    To execute just one test, use property xtest.run.singletest.

         ant -Dxtest.run.singletest=org/openide/options/*Test.class

It will use default executor, compiler and resultsprocessor so you have defined them in cfg file. Tests is taken from directory ${xtest.testtype}/src.

Note: Every test author will be required to add entry to proper cfg file in order to execute their tests.

   If you want to use your own config file (e.g. one generated from failed tests) set property xtest.module.configfile with path to config file

        ant -Dxtest.module.configfile=/path/to/my-cfg.xml

Test set, patternset, patternset attributes

   Test bag can contain one or more test sets, where each of them can select tests from different folder (in given example there is only unit/src). It allows great flexibility in definition of the set of tests you want to execute. Element <testset> has subelement <patternset> (with subelements <include> and <exclude>), which define simple regexp like pattern for including or excluding tests in/from given test bag.

   In some cases it may be useful to filter patternset by some additional attribute. You may want to execute only patternsets which contains some property or patternset which doesn't contain some attribute.

      <testset dir="unit/src">
          <patternset>
              <include name="test1.class"/>
          </patternset>
          <patternset attribs="i386">
              <include name="test2.class"/>
          </patternset>
          <patternset attribs="sparc">
              <include name="test3.class"/>
          </patternset>
          <patternset attribs="i386,localized">
              <include name="test4.class"/>
          </patternset>
          <patternset attribs="sparc,localized">
              <include name="test5.class"/>
          </patternset>
          <patternset attribs="localized">
              <include name="test6.class"/>
          </patternset>
      </testset>

The pattern attribs have to be selected via additional property xtest.patternset.attribs when tests are run: ant -Dxtest.attribs=stable -Dxtest.patternset.attribs=i386,!localized. Exclamation mark means negation. When no pattern attribute is specified, all patternsets are used.

patternset attributes	executed tests
i386	test2, test4
i386, sparc	test2, test3, test4, test5
!localized	test1, test2, test3
i386, !localized	test2
!sparc, !localized	test1, test2

Note: don't mix up testbag attributes and patternset attributes.

Expected fails

If some test permanently fails because of a bug in tested product, you can mark this test as it's expected to fail. All such tests are counted separately and each test has assigned text why it's expected to fail (e.g. bug number). If such test passes it's result is 'unexpected pass'. To mark test as 'expected fail' add attribute 'expectedFail' to element include. If you use suite() method you have to use NbTestSuite. Otherwise expected fail is not taken into account.

  <patternset>
      <include name="org/openidex/nodes/looks/*.class"/>
      <include name="org/openidex/nodes/looks/LookNodeTest.class/testGetActions" expectedFail="bug #12345"/>
      <include name="org/openidex/nodes/looks/LookNodeTest.class/testGetPropertySets" expectedFail="bug #6789"/>
  </patternset>

Order of testsuites

When executing, XTest tries to preserve order of testsuites defined in config. When wildcard in include is used or testsuite is included more than once, order is not clear and in this case order will be defined by first include witch match the criteria.

Example 1:

<include name="org/openide/test/*.class"/>
<include name="org/openide/test/Test1.class"/>
<include name="org/openide/test/Test2.class"/>
<include name="org/openide/test/Test3.class"/>

In this case order will not be defined, because all suites are already
included by first include.

Example 2:

<include name="org/openide/test/Test1.class/testA"/>
<include name="org/openide/test/Test2.class"/>
<include name="org/openide/test/Test1.class/testB"/>

Order will be: Test1.class, Test2.class

Results Processor

   Once we allowed that flexibility that one test bag can contain tests from different folders we have to specify what kind of results will be generated by these tests. In previous version of Xtest framework it was given by name of folder (unit meant just unit tests, perf just performance tests and so on), but name of folder is not important now to distinguish type of results. Therefore there is new attribute called "resultprocessor" . So far the resultprocessor is only used as name of folder where results of given formatter will be stored (we have only one type of test results so far - unit). But some next version of xtest framework could define something similar to executors for formatting results.

Generate config for failed or excluded tests

XTest can generate configs for failed and excluded tests, so user is able to run just failed or excluded tests after e.g. fix and not to bother with number of successfully passed tests. These configs are created from testreport-failures.xml (for failed tests) file or from given cfg-{testtype}.xml (for excluded tests). As follows from already said it is applicable only for a particular modules.
To be able to use this feature you need to add the following to <module>/test/build.xml:

   <target name="generateFailedConfig">
	<xtest executeAction="generateFailedConfig"/>
   </target>

   <target name="generateExcludedConfig">
       <xtest executeAction="generateExcludedConfig"/>
   </target>

Then call this from <module>/test directory:

   ant generateFailedConfig

It will generate <original-config>-failed.xml file containing failed test cases from the last test run. It means it expects that there exists <module>/test/results/xmlresults/testreport-failures.xml file which contains failed test cases from the latest execution of your tests.

   ant generateExcludedConfig -DconfigFile=cfg-qa-functional.xml

Above command will generate <original-config>-excluded.xml file with test cases which were excluded in original config file. If parameter configFile is omitted it process all available config files.

To use generated config files for test execution, you can use for example the following command:

   ant runtests -Dxtest.module.configfile=cfg-qa-functional-failed.xml