XTest Quick Start Guide for Test Developers
Warning: Parts of this document may be out-dated. Checked with NetBeans 3.6.
Version: 1.3
Author: Martin Brehovsky
Abstract: This document is a quick start guide for anybody
who is looking to write tests with the XTest framework.
- Document History:
- 22 Jan 2002 : Martin Brehovsky, created
- 16 Dec 2002: Martin Brehovsky, added information about TestTools module
- 1 July 2003 : Martin Brehovsky, synchronized document with latest development
- Contents:
- 1. Install XTest
- 2. Setup test workspace
- 3. Write tests
- 4. Write scripts and config files
- 5. Build, run and browse results
- 6. Integrate tests to the XTest instance
- 7. Using the TestTools module
- 8. Looking for other information?
Before you can start writting and running tests in the XTest framework you
need to install the framework in a suitable place. You can install XTest
by building your own version from sources or by downloading an already
built binary distribution of XTest. Although building your own version
might look like a good idea because you can be sure you always have
the latest version of it, it is not recommended for users who don't have
any experience with building the NetBeans IDE from sources.
You also need to have installed Ant
tool, at least version 1.5.1 for trunk or version 1.4(.1) for
release35 branch. Please note: you should always use the same version of
Ant as is officially used by buildmasters to build the IDE.
1.1 Installing from sources
Installation of XTest from sources is described in the detailed
XTest installation
document.
1.2 Installing from binaries
An easier way to install XTest is to download already built binaries
from the downloads page. Unpack the downloaded
file to a directory which will become your xtest home. When testing
NetBeans, it is recommended to unpack this file in the nb_all/xtest
directory, which is used as a default xtest home for all NetBeans tests.
Please note: if you would like to use XTest examples, you should checkout
the XTest module from cvs prior to unpacking the binary file. You can also look
at the XTest installation
document to see more details about binary installation.
1.3 Post install tasks
After either installing XTest from binaries or compiling your own
version you should make sure you do some post install tasks:
- As written in the XTest installation
document, when planning to use the NetBeans
extension to JUnit (basically the ability to perform asserts
on files, using test method independent working directories, etc),
you might need to configure the file-diff extension. For details,
please see the mentioned documents.
- Also when planning to run the tests on the Windows
platform, there is a
bug in Ant which sometimes causes a deadlock
when stopping the IDE. When experiencing such a problem, make
sure you have applied the
patch solving this
issue. You can also
vote for this bug to be fixed :-).
Although this section is focused on the how to setup tests for
NetBeans projects, all the recommendations written in this part should
apply for any other tested project:
- All tests should reside in the test
subdirectory of the tested module.
- Under this directory, there has to be subdirectories,
which divide the tests into several types. Their names are the same as
the test type they contain. Developers (who mostly write unit tests)
usually store their tests in the 'unit' subdirectory, QA (who mostly write
functional tests) usually in the 'qa-functional' subdirectory. Other test
types can be performance (can be shortened to perf), signature, etc.
(Please note: the test type is only a name - it has nothing to do with
the real test type stored in the directory. So if desired, there is no
problem creating a directory named compatibility, which will store unit
tests focusing on some compatibility issues).
- Test sources should be stored under the src subdirectory
of the test type directory.
For example: in module openide, unit test org.openide.filesystems.XMLFileSystemTest
is stored in nb_all/openide/test/unit/src/org/openide/filesystems/XMLFileSystemTest.java.
For more examples, please see the test subdirectory in the openide module.
Other examples can be found also in XTest's examples, in directory
nb_all/xtest/examples/MyModule/test, or in other NetBeans modules
like core, java, ....
Now that you have created a test workspace in your module, you are
ready to write tests.
- The current version of XTest only supports
tests based on JUnit
or preferably
NbJUnit framework. You can start with JUnit here and learn more
about using NetBeans extension to JUnit (NbJUnit)
here. JavaDoc
to JUnit is available online
here.
You can also look at the Unit Tests Tutorial
here. Please note: although it might look like only unit
tests are supported by XTest, in reality the JUnit framework is very
flexible so it can support other types of tests, like performance, functional,
etc.
- Some tests might require system specific
data, which is not known until the actual test run on a specific machine
(e.g. database username, password, URL, location of JDBC driver, ...).
XTest provides some useful functionality to help you with these cases.
You can read more about it in the site
specific setup document. You can also look at a short tutorial
concerning this feature.
- NbJunit examples can be found in the
nb_all/xtest/examples/MyModule/test directory or in the test subdirectories in a number
of NetBeans modules (e.g. openide, core, java - these are mostly
unit tests, openide and debuggercore also contain functional tests).
- Test developers may also find handy the
JUnit module
for NetBeans IDE, which is able to generate skeletons of tests
based on JUnit and compile/execute tests within the IDE. You can download
this module via the update center from your IDE.
A very important step in writing XTest based tests is to write scripts
and config files. These files are required because they allow
writing such tasks as clean, build, run and publish results. Also
they handle conveniently listed tasks for the test developer - because
all the files must conform to a given standard, they allow running tests
from several modules (which are independent of each other and can be developed
by different test developers) in a single testing session (for more details
on this, please see the section named 'Running tests daily' on the documentation overview page).
If you look in any test directory (for example: nb_all/xtest/examples/MyModule/test),
you will see a build.xml file and one or more cfg-*.xml files. A build
script is a standard Ant build file and this script is called by the test
developer when cleaning, building, and running tests from XTest. These
are config files which contain definitions for the set of tests grouped
together, called 'Testbags'.
- Each testbag can be compiled and executed
via a user defined ant task, so in the case of executors, one set of
tests can be executed in a standard JVM and the other set in the NetBeans IDE.
- Config files have to be named as cfg-${xtest.testtype}.xml,
where ${xtest.testtype} is the test type of the configured tests. Please
note: this name has to be in sync with the name of the directory,
which contains the test sources. For example, unit tests stored in the unit/src
subdirectory will have a corresponding cfg-unit.xml config file, functional
tests usually use cfg-qa-functional.xml as the config and are stored
in the qa-functional/src subdirectory of the test directory.
For examples of both build files and config files, please see the nb_all/xtest/examples/MyModule/test
directory. A detailed description of the Ant's build script can be found
here
and a description of config files here.
You can also use a build file template, available herefor download.
Please note: use only relative paths when writing build scripts,
because this assures that tests can be run on different computers
and different platforms without any change in sources/configurations.
Now that you have written tests and created all required scripts
and config files, you are ready to compile and run tests. This can
be done very easily by calling standard targets in the test's build.xml
script. If you have not written any tests, you can try to run example
tests for the module MyModule (available in nb_all/xtest/examples/MyModule).
Here are the steps required to run tests for MyModule from the examples
directory, but they should apply to any other tested code as well.
- Before tests are run, make sure you have
built the tested code. Switch to MyModule directory (nb_all/xtest/examples/MyModule)
and run ant (no target specified, by default it will compile MyModule
and create mymodule.jar in the root directory of the module).
- For the next step, switch to the test subdirectory
(nb_all/xtest/examples/MyModule/test) and run ant (again, no target
specified - by default it will build tests and run them). You should
see ant writing out a number of lines (as targets are executed by the harness)
and one start of NetBeans IDE and at the end ant should report success
by writing out 'BUILD SUCCESSFUL'. We're done with the test run. You
can also experiment with selecting testbags by xtest attributes and running
a different test type:
- To run just tests in your one specific
class, run ant with following attributes set:
ant -Dxtest.attribs=empty -Dxtest.includes=mypackage/MyTest.class
It's even possible to run exactly one test, for example inside the IDE:
ant -Dxtest.attribs=emptyide -Dxtest.includes=mypackage/MyTest.class/myTestMethod
Running one specific test is useful especially when you're developing the
test and you need to run it frequently to ensure its correct functionality.
- To run stable tests in code mode, run ant
with the property xtest.attribs set to stable,code (ant -Dxtest.attribs=stable,code).
XTest will select only those testbags which can only run when attributes
'stable' and 'code' are set. For more information about attributes,
please see the XTest configuration
document and config files in the nb_all/xtest/examples/MyModule/test
directory.
- To run test type 'logging', just run ant
with the property xtest.testtype set to logging (ant -Dxtest.testtype=logging).
- Now you are ready to browse the results. If you
look a couple of lines above the 'BUILD SUCCESSFUL' message, you will see
a URL to check for the results report (by default it should be
nb_all/xtest/examples/MyModule/test/results/index.html).
- Open this report in your favorite
browser (please note: reports are organized in frames, so your
browser has to support them) and you can see the results of your
first test run.
- You should see a report for the module MyModule
and two testbags (one running in plain JVM (code mode), the other in the IDE
(ide mode), each with one failing test (for demonstration purposes).
Please note: if you are running tests for one module several times, all
reports are combined together in one big report. You can easily differentiate
between each test run (i.e. each call of the runtests target) in the report, because
each result from each testrun are grouped together under the TestRun item
in the tree structure shown in the left frame.
- If you would like to start with clean results
(i.e. results from only the last testrun),
run the target cleanresults before running the tests (ant cleanresults runtests).
- You can further experiment with cleaning,
building, and running tests. Here's the list of available targets with
an explanation of what they do:
- ant cleantests - cleans all built tests
- ant cleanresult - cleans results
- ant realclean - clean both built tests
and results
- ant buildtests - build tests (do not run
them; only create jar)
- ant runtests (default target) - run tests
Please note: if you don't have xtest installed in the standard location
(nb_alll/xtest) you have to run ant with the xtest.home property set to
your xtest home (-Dxtest.home=your-xtest-home). The same thing applies
to the NetBeans IDE in the case you are testing the IDE (using ide executors)
and it is not installed in the standard location (this location is taken
from the developers perspective - so it is in nb_all/nbbuild/netbeans
- the place where NetBeans is stored when built from sources), so run ant
with the netbeans.home property set to your netbeans home (-Dnetbeans.home=your_netbeans_home).
A final and optional step for test developer is to integrate created tests
to the XTest instance. The XTest instance is basically a configuration for XTest
harness which is able to run tests from several modules in one testing
session. You can find more documentation about the instance here and about its configuration
here. Please note, that
this integration should be done together with the XTest instance administrator,
so please don't integrate anything on your own.
If you are tired of all the manual settings of XTest and even test development,
you can use TestTools module, which installs directly to your NetBeans IDE
(via Tools/Update Center). This module lets you setup the XTest workspace,
including configuration files and neccessary build scripts. It can also help
you with the development of unit tests (it contains the JUnit module) as well as GUI
tests (it contains Jemmy, JemmySupport and JellyTools modules) and other stuff.
You can read more information about this module at its
homepage
.
If you look for any information not explained in this document or documents
linked from this place, you can look at XTest's
frequently aksed questions. If you still cannot find the relevant information,
feel free to ask either on nbdev@netbeans.org or nbusers@netbeans.org.
