XTest Architecture Overview
Version:0.2
Author: Martin
Brehovsky; Martin Balin
Abstract: This document is an overview of the XTest framework and also
presents several important future plans.
- Document History:
- 11 Oct 2001 : version 0.1, created
- 16 Jan 2002 : version 0.2, update to reflect latest development
- Contents:
- What is XTest
- Basic Structure of XTest
- Harness
- Publishing Engine
- Driver
XTest is a new testing framework which is being primarily
developed for testing the NetBeans IDE, but is independent enough from it to
be used for testing any other Java based products. XTest is based
on Ant, which means it allows users to use all ant tasks including custom
extensions written for Ant. This harness was originally developed by developers
for developers, but because of its light weight and open design, it is
suitable for project-wide usage, i.e. it can be used even by Quality Assurance
and Release Engineering groups. The main features of the XTest framework are:
- Platform independence and extensibility in an easy
way.
- Allows tests to compile before run.
- Creates test distribution, which can be further tested
on other machines.
- Select tests to be run on attributes which they
must satisfy.
- Run tests either in NetBeans IDE or without IDE
at all.
- Publish test results in a form of XML documents,
which are further transformed via XSLT into other types of documents (HTML
pages, plain text).
- The format and style of the transformed results
can be changed as different users can have their own requirements and priorities.
- Seamlessly combine several test types (e.g. unit
tests, signature tests, performance tests, ...) at once.
- XTest is already being used by several NetBeans
developers to write their unit tests (for details see test directory under
core, openide, openidex, or java module).
- XTest utilizes only open source technologies.
XTest can be divided into three functional parts (see
figure 1), which are mutually independent, hence possible modifications in
one part does not significantly affect other parts.
Figure 1 - XTest structure
The first functional part is the harness, which is responsible for
running individual tests. The second part is the publishing engine, which transforms
test results produced by the harness into human readable form. The publishing
engine can also generate some statistics from the tests run. The last part
of XTest is the driver, an infrastructure which performs site specific
setup steps, defines a set of tests to be run from the harness, and calls the publishing
engine to transform results into the desired form.
In the following section each part will be described
in more detail. It will summarize the current status of the tool, describe
goals which we would like to implement in the near future and also present
several long term suggestions/plans.
The harness is the core of the XTest testing framework.
Its main task is to select a set of tests from defined configurations of
tests, run the selected tests and deliver results in the form of XML files.
The XTest harness is designed in such a way that it allows tests to run
based on several test types in one test run. For example the test writers
can create unit tests and they can be run together with performance tests,
signature tests and other types of tests.
Because XTest was intended from the beginning to be a really open tool,
there is no support for any test types implemented directly, but rather
in a form of independent modules. This means that new test types can be
added without any major changes in the harness. The only things to be done
when adding new test type support is to write a custom executor, which
would run the tests, and a reporter, which would return results as XML files.
Current Status
- A new configuration method for defining groups of tests
is already implemented. This new configuration is very flexible, so it
can satisfy requirements from various views (e.g. developers, quality assurance,
etc).
- Tests can be run either in NetBeans IDE or outside
of the IDE (harness is able to handle starting and stopping IDE).
- Currently XTest has implemented support for JUnit
based tests (Unit test type), which can be used in 80-90% of all tests
written for the NetBeans/Forte IDE. There are several other test types in
the process of being implemented (performance tests, signature tests,
etc).
- Allows tests to compile before run.
- Currently XTest supports creation of a new
working directory for each test.
- Simplified build.xml file (this file resides under the
${module}/test directory). Not all configurations reside at this level -
some common ones are shared from module_harness.xml.
Future Plans
- Support for other test types
will be implemented. The implementation would ideally include corresponding
functionality in theNetBeans IDE, which would help the test developer to generate
test skeletons and implement tests in a more convenient way.
Since results from run tests are produced in the
form of XML files (each test type may produce different information), they
need to be transformed into a human readable form (usually HTML pages or
just plain text documents). This functionality is achieved by the XTest's
publishing engine.
The publishing engine uses technologies like XSLT and XPath to provide
a tool which aggregates output from tests and transforms them into other
formats. The underlying technology (XPath) also has the capability to perform
some basic statistics over the transformed files, hence simple summary pages
can be generated automatically without such support from the harness itself.
The separation of the publishing engine from the harness offers several
advantages:
- The transformed output can be in several forms,
because each department has its own requirements and priorities for generated
information.
- Changes in the harness output does not have to mean
changes in the transformed output and vice versa.
- Results from different test computers can be integrated
together, so all results can be browsed seamlessly (the reader of the results
will not have to know on which test computer to look for particular results,
the integration can also be used to automatically provide basic overall
statistics).
Current Status
- Results from the harness can be transformed into
a form of HTML pages as well as into plain text.
- Test results from each test are aggregated and a basic
summary is provided.
- The publishing engine was separated from the harness.
It was also split into two parts. One part is run on the machine where the tests are
executed and creates the basic structure of the results. The second, called Server side
PE, works on the server and creates a complex structure of test results from
different builds, machines, ...
Future Plans
- Implement the ability to support archives of test results
stored in a database (for more details see the next part)
The final part of XTest is the driver, which can be
viewed as the glue between the harness and the publishing engine. The driver's
responsibilities are:
- Perform site specific setup (usually required by
client/server and platform dependent tests).
- Run the harness over the specified set of tests.
- Run publishing engine, so the end user can browse the
results in a convenient form.
- Optionally archive results, so they can be browsed
later.
Current Status
- The driver allows starting the IDE, checking out necessary
sources from CVS and supports XTest instances in CVS (an instance is just a
configuration to run XTest binaries for a particular product).
- Provides the possibility to carry out site-specific
setup and teardown for tests, including starting and stopping of supporting
applications (e.g. database servers, J2EE RI server) with correct site-specific
values.
- Supports storing of results in archives. Now it
archives results in the form of structured directories.
Future Plans
- Later on, support for storing results in a database
will be provided. This kind of archives would also enable performing complex
queries over stored data, which can help to analyze the data in many ways.
