Writing Test Buildscripts

Author: Libor Martinek (libor.martinek@sun.com) , Martin Brehovsky (martin.brehovsky@sun.com)
Last modif.: Feb 18, 2004
Version: 1.5

---

What is this document for?

This document is intended for developer who wants write new tests (or convert existing) for xtest. This document describes scripts which are used for executing tests for one module.

Description of module's test buildscripts

Every module has tests in directory "test". This directory contains file build.xml which is main script for building and running tests for this module. See template of such build.xml. This script should be same for all modules so just copy it and paste it to your module.

Compilers and executors are in separated file, one for each testtype.  The naming convetion for this file is 'build-${xtest.testtype}.xml'. For example tests with test type unit should have all compilers and executors in 'build-unit.xml' file. Example of such a file you can see (link to CVS ????). You can also download a template of this file, which you can further customize.

Look at XTest Properties for a list of properties which you have to/may set or use.

Main buildscript - build.xml

This buildscript serves for building and running tests. This buildscript is rather general and you should not change its targets, because the targets are used by XTest harness to execute tests in a standard way.

• buildtests - target for bulding tests
• cleantests - target for cleaning test sources
• cleanresult - target for cleaning results from previous runs.
• realclean - target for cleaning tests and results
• runtests - target for running tests.

Properties

build.xml contains these properties, which you should change to be correct (you can also specify these properties via command line when running the tests)

• xtest.module - name of tested module
• netbeans.dest.dir - path to tested IDE
• xtest.home - home of xtest
• xtest.testtype - default test type when no one is specified in commandline
• xtest.attribs - default attributes when no one is specified in commandline

These properties are used only when tests are executed for module from directory test. You can override xtest.testtype and/or xtest.attribs property in commandline when you want execute another testtype/testbags then default. When tests are executed from higher level (instance, driver) then these properties are set in that level and override properties set in build.xml.

There are several other available properties, which can be used when running tests from commandline:

• xtest.testbags - If you want to execute only testbags with given names. Set this property with names of testbags separated with comma (","). Test attributes are ignored when xtest.testbags is used.
  
• xtest.userdata.propertyfile - if value of this property points to property file, all properties from this file are loaded. Properties in the property file must have same format as properties for master config: xtest.userdata(<attrib>)|name=value or xtest.userdata|name=value. Only properties which has right <attrib> and properties without <attrib> will be passed to tests. Part before '|' will be removed.
  
  Example:
  xtest.userdata|name1=value1
  xtest.userdata(stable)|name2=value2
  xtest.userdata(unstable)|name3=value3

  If tests with attribute "stable" is executed then property name1 and property name2 will be set. Third property hasn't right attribute.

     If you want to define tests only by following two properties, use empty testbag in cfg file.

• xtest.includes - includes pattern will be merged with pattern set in cfg files
• xtest.excludes - excludes pattern will be merged with pattern set in cfg files

Other properties

• xtest.archive.ide.userdir - If true, then IDE userdir will be archived in results. This property has to be set in main build.xml and thus it's valid for whole module. If you want to set it only for one testtype then move result processor (target test_report) from build.xml to testtype's buildscript , change resultprocessor element in cfg file (change antfile attribute) and add this property to result processor's target. If you want to set it for each testbag, create more result processors.

Testtype's buildscript (build-<testtype>.xml)

It's recommended to have separate buildscript for each testype, recomended name is build-<testype>.xml, where <testype> is name of testtype. These buildscript contains compilers and executors, which are defined in config file (cfg-<testype>.xml).

Executors

Config file has to define executors. Executors are targets within an ant build script, which basically order XTest what to test (they might do a lot of other things, though, it all depends on your needs).  You can read more about executors in this document.

Compilers

Config file has to define also compilers. Similarly as with executors, compilers are within an ant build script, which compiles test source files and packages them to test archive, which is then used when tests are executed. Also similarly as with executors, you can read more about compilers in this document.

Properties to be specified when compiling/executing tests

Each plugin can supply its own properties, which are used to influence its behaviour when compiling/packaging or executing tests. To get more information about the properties you have to look at the documentation provided with the plugin. For plugins supplied by XTest, you can see the documentation in this document.

How to work with template

Suppose you have created tests for new testtype, e.g. qa-functional. Put them to new directory for the testtype <module>/test/qa-functional/src/... Create config file cfg-qa-functional.xml for this new testype and put it to directory <module>/test.

Note: all paths should be relative.

1. Copy template of build.xml to directory <module>/test.
2. Changel property xtest.module to name of tested module.
3. Assure that property xtest.home points to xtest (xtest module or distribution, not instance). In case when you don't know where xtest is (or will be) then you have to set this property from commandline (ant -Dxtest.home=/opt/xtest) when executing tests.
4. Set property netbeans.dest.dir to point to tested IDE. In case when you don't know where it is (or will be) then you have to set this property from commandline (ant -Dnetbeans.dest.dir=/opt/netbeans) when executing tests.
5. Change following properties with default values according to your tests: 
   • xtest.testtype - e.g. qa-functional if you write test of this testtype.
   • xtest.attribs - put here attributes of testbags you want tu execute.
6. Copy template of testtype's buildscript to directory <module>/test. Name it build-qa-functional.xml for tests of qa-functional testtype.
7. Define all executors you need in config file and implement them in testtype's buildscript.
8. Define all compilers you need in config file and implement them in testtype's buildscript.
9. If your test requires to add some other argument into ide commandline, use property xtest.ide.commandline.suffix . You can insert this property into each executor if you need different one for each executor.