XTest Quick Start Guide for Test Administrators
Version: 1.4
Author: Martin
Brehovsky
Abstract: This document describes a quick start guide for persons
running tests regularly on several machines (test administrators) and
presenting results from those tests.
- Document History:
- 26th Feb. 2002 : Martin Brehovsky, created
- 6th June 2002 : Libor Martinek, added XTest Server info
- 1st July 2003: Martin Brehovsky, updated to reflect the latest
development
- Contents:
- 1. Before you start using XTest
- 2. Setup test machines
- 3. Setup publishing server
- 4. Run tests automatically
- 5. Watch results
Before you start using XTest, make sure you understand the following
terms.
Basic terms used in this document:
- Driver - a part of XTest which is able to
install the NetBeans IDE, checkout/update tests from CVS based on information
obtained from specified XTest instance, run the tests specified by instance
config, and ship results to publishing server. You can learn more about
the driver here.
- XTest instance - one or
more configurations of tests from one or more modules in one CVS repository.
These tests can be run via the instance in one testing session. XTest instance
also allows creating a configuration which can use system specific the setup/teardown
feature of XTest. More information about XTest instance is available on
this page.
- Publishing server - a part of XTest which collects
results created by the XTest driver and integrates them into the overall results,
thus users can browse them seamlessly. Currently XTest contains only a simple
publishing server, which produces only static html pages based on the received
results. More information about the publishing server is available here.
- XTest Server - automatically runs tests when a new build is available.
Can be configured to test more projects. More details here.
Each testing machine will need to have the following software installed:
- JDK - since you will be testing a Java based
product, you will need one or more JDKs (1.4 or higher for trunk, 1.3 or
higher for release35 branch) installed on the testing machines :-).
- Ant - XTest requires running Ant at least
version 1.5.1 for trunk and version 1.4.1 for release35. Please note: when
running tests on the Windows platform, you might need to apply a patch because
in Ant there is a bug which sometimes causes a deadlock when stopping
external processes (like NetBeans IDE, CVS, ...). You can download the patch
here. For more information see the details of the issue
on Ant's bugzilla.
- CVS - command line CVS client. It is required
by the XTest driver when checking out/updating tests from CVS.
- XTest - install XTest binary distribution (you
can make your own from XTest sources by typing 'ant makedist') to a suitable
directory.
- Diff - when running tests performing assertions
on files (feature available in NetBeans extension to JUnit, on which
XTest's tests are based), you will likely need a command line diff utility.
- scripting environment - for setting environment
variables and running a command line application like Ant you might find
handy advanced, unix like, scripting environment like bash, ksh (korn shell),
or even the plan sh (bourne shell). Unix based systems have at least one
of the listed scripts installed by default. On Windows you can use a product
called CygWin, which
is basically an implementation of GNU open source libraries, compilers
and utilities known from Linux and other Unixes for Windows based machines.
This product contains bash (open source scripting environment) as well as
many other command line utilities, including diff and cvs.
Before starting periodical testing, you need to spend some time on XTest
configuration. You will need to configure the following:
- If you plan to use a native diff utility, you need
to configure the junit.properties file in your user.home directory. Details of
this configuration are available on this
page.
- A very important step is the configuration of
the driver. There you describe which IDE (or any other tested product)
will the driver install for testing, which tests from which CVS will be checked
out and run against the installed IDE.This document contains detailed information
about the driver and its configuration files.
- Depending on tests you will be running, you might
need to setup site specific scripts/properties, like scripts for starting
databases, web servers, application servers, or any other application/script
which is system specific, or property files containing system corresponding
values. Test developers should provide sample scripts/properties to their
testcases and give you details about these files. You can read more about
site specific setup in this
document.
- If you want to use XTest Server you will need to setup its config files.
See XTest Server documentation.
Publishing server collects test results produced by testing machines,
organizes the results in a structure suitable for web browsing and performs
some basic statistics over the collected results. In the current version
of XTest, this server does not provide dynamic pages (this feature is planned
soon), thus for publishing results you need just a basic HTTP server (e.g.
Apache) without any dynamic capabilities.
You will need to setup a machine which will provide you with all results
gathered from the testing machines. This machine does not have to be a dedicated
server, you can easily use one of your testing machines (as long as you
are not running performance tests on it :-)) as a publishing server. You can
read more about the publishing server and its setup and usage in this document.
When you have installed XTest and provided all required setup to your
testing machine(s), the publishing server is running and is ready for the test
results, you can start automatic testing. This part deals with testing a
product, when a new build is released.
For checking availability of the new build you can useXTest Server, which periodically looks at the
last build available and when a new one is found, it starts the XTest's driver
with the appropriate script.
Please note: if you plan to test the NetBeans IDE or any Java product which
starts another Java process on the Windows platform, you should make sure all
Java processes are finished when XTest is finished. There is a problem
with killing child Java processes on Windows, because although XTest is able
to kill the parent process, it is not able to kill the children (because
of the Windows architecture). Thus it is highly recommended to kill the remaining
Java processes when the XTest driver is finished. For the killing itself, you
can use numerous command line utilities available for free (try to look on
any major site providing Windows free software, like www.tucows.com).
Now you are ready to wait for the first results from automatic testing.
Point your favorite browser (Mozilla recommended) at the homepage of your
published test and watch the changes :-).
