XTest design specification

Version: 0.2
Author: Daniel Pfeffer

Abstract: This document describes the design and functionality of Xtest.

Document History:

12 Feb 2001 : version 0.1, created

23 Feb 2001 : version 0.2, updated

Related documents

XTest Signature tests support design specification

 Contents:

Introduction

Description Of The Design

Unknowns

Constraints

Introduction

XTest is designed to support NetBeans based products tests.
The framework should cover support for the following types of tests:

• Unit tests
• Signature Tests
• Performance Tests
• Integration Tests
• Stability Tests

General Test Schema

Each test can be generally described by the following schema:

• Test launcher is an application which starts the test harness (e.g.: command line interface or NetBeans IDE)
• Test harness is a program or tool that controls test execution by setting up a test environment, running a selected set of test suites, determining whether the test suite passed or failed, sending notifications, and printing reports (e.g.: ant or perl ).
• Test board is a process in which the test and tested code runs (e.g.: tested instance of NetBeans)
• Test framework is supporting code (library), which is used in test code and which allows collecting of test rsults (e.g. JUnit or Tonga )
• Test code is code which is testing the tested code (e.g.: JUnit module or Open API)
• Tested code is the code which is the subject of the test

General test pattern

• The tests are hierarchically organized within Test projects .
• Each test consists of:
  • Test specification document describing what and how to test.
  • Test script
  • Test data ("golden files" if applicable)
  • Test application(s)
  • Test results

General test structure

A set of test suites can be called a Test bag .
Tests can be divided into test Suites covering some functional unit tests (e.g. class).
Each test suite consists of Test cases - test elements which can be checked to pass (e.g. method functionality test)

XTest platform

XTest is designed as a lightweight test support environment designed to be used for NetBeans tests.

The concept is based on the usage of existing open source products:

• Ant is used as a test harness
• JUnit is used as a test framework collecting test results
• Xalan is used for test scripts and test results formatting in XML and HTML

Description Of The Design

General Concept

XTest usage scenarios

There are several scenarios for how the test framework will be used:

1. Tests in development process
   During the coding and bug fixing phase, these are test related tasks:
   • Unit tests
   • Methods Signature compatibility tests
   • Performance tests

   The tests run mainly on developer's workstation.

2. Automated tests
   These tests run automatically and are often included in an automated build process.
   The tests run mainly on preinstalled build and test frameworks on dedicated test bed machines.
   The tests are started from the comand line prompt and the test results should be "ready to publish" (in HTML).
   Short test results summary should be possible to send as an e-mail or recorded in a test database.

Xtest is designed to support both test environments in a consistent way (i.e.: Tests being developed during development process should be possible to include in automated tests without any change needed).

Proposed implementationis based on the folowing concept.

• The XTest framework is a set of tools and utilities using ANT, Junit, Xalan and NetBeans as a platform.
• The test applications development is supported by modules integrated in NetBeans (JUnitand Ant Modules).
• The test scripts are written in Ant.
• The test results are generated in XML file format (Junit test runner XML test results formatter, ...), so they can be linked to be "parsable" on all test levels.
• The test results are transformed in HTML using XSLT templates so they can be linked together to be "browsable" on all levels.
• The test results can be hierarchically linked together, so it will be possible to easily generate test results summary on all levels.
• It is possible to write test results in a "tests database" using a "Test record" utility.
• The tests can be performed in a cascading manner using the Ant feature which allows starting another Ant task.

The hierarchy design allows building tests independently and then integrating them in bigger tests suites (see picture below).

The same Ant test script should be able to be invoked from the command line or the NetBeans IDE.

The general concept of Ant usage as a test and build harness is illustrated in the following picture:

Note: JUnit can be used for unit and integration tests. For other types of tests (signature, performance, ...) another test framework needs to be used.

Test files

The test files can be classified in the following categories:

• XTest generic files (utilities, templates, ...)
• Project files

This is also reflected in the directory structure.

XTest generic files

XTest is packaged in a simple directory structure.

XTest directory structure

/www

/src

/lib

/examples

Project files

The hierarchical structure of tests is also reflected in the test file structure.
The test projects should be under version control in the "test" subdirectory of the project.

Project directory should then contain the following subdirectories:

/src

/bin

/www

/test

The test directory can consist of "sub-test" test directories (with the same directory structure).
The project test directory will usually contain several test subdirectories (unit, signature, performance, integration, ... ).

Each Test directory should contain the following structure:

build.xml

manifest.mf

/src

/lib

/results

This structure enables running the tests on different levels using simple Ant command line calls. Using just the "ant" command will run all tests in the current directory subtree.

Unknowns

None

Constraints

None