HOME > Community > Projects > xtest

Continuous Testing of NetBeans Platform Based Applications

Author: Jiri Skrivanek
Last update: February 12, 2007

This document shows how to create and run tests for modules of applications based on NetBeans platform. You can learn how to run tests inside NetBeans IDE but also how to deploy tests to be run after every successful build of your application. We will use XTest harness for test execution, Hudson for build management and several supporting modules for test development.

Content

Install supporting modules
Create tested application
Create test infrastructure
Write simple unit test - build, run, debug, measure coverage inside IDE
Write simple functional test - build, run, debug, measure coverage inside IDE
Install Hudson
Build application using Hudson
Test application using Hudson
Process results using XTest PES
Summary

Install supporting modules

To easily create and run tests inside IDE you should install supporting modules. Go to NetBeans Update center and install the following modules:

NB JUnit - extension to JUnit
NB JUnit IDE - integration into IDE
Jellytools - library for developing NetBeans UI functional tests
Jemmy Module - library for developing Java Swing UI functional tests
XTest Module - test harness; XTest templates and actions

Create tested application

We expect you want to develop an application based on NetBeans Platform. As an example of such application can serve sample Paint Application. Open New Project wizard, select 'Samples|NetBeans Plug-in Modules' and choose 'Paint Application'. Select project location and confirm the wizard. The Paint Application is created and opened in NetBeans IDE. It consist of two modules ColorChooser and Paint. Later we will show how to write tests for Paint module but the same way you can write tests for other modules. It is also recommended to store your modules into some versioning control system.

Create test infrastructure

Now we can generate files needed for test execution. Open New File wizard, select Paint project, choose 'Testing Tools' category and 'XTest Infrastructure' file type. On the next page you will see files which will be generated. Finish the wizard and files are created and opened in editor. You don't need to change anything and you can close all these files. Also you should see 'Unit Test Packages' and 'Functional Test Packages' nodes under Paint project node.

Write simple unit test - build, run, debug, measure coverage inside IDE

We have created necessary infrastructure and we can write unit and functional tests. To create unit test call 'Tools|Create JUnit Tests' action on 'Paint|Source Packages|org.netbeans.paint|PaintCanvas.java' node. Open PaintCanvasTest.java stored in Unit Test Packages and implement just one test case. Everything else can be deleted. Finally it should look like this:

package org.netbeans.paint;

import org.netbeans.junit.*;
import org.netbeans.paint.PaintCanvas;

public class PaintCanvasTest extends NbTestCase {
    
    public PaintCanvasTest(String testName) {
        super(testName);
    }
    
    public static void main(String[] args) {
        junit.textui.TestRunner.run(suite());
    }
    
    public static NbTestSuite suite() {
        NbTestSuite suite = new NbTestSuite(PaintCanvasTest.class);
        return suite;
    }
    
    public void testSetDiam() {
        PaintCanvas paintCanvas = new PaintCanvas();
        paintCanvas.setDiam(10);
        assertEquals("Diam should be set.", 10, paintCanvas.getDiam());
    }
}

Before you run tests you have to build Paint Application. Then you can build, run or debug unit tests. You can even measure code coverage of tests. Just call an item in XTest sub menu on Paint project node. Test results are then opened in default HTML browser.

Write simple functional test - build, run, debug, measure coverage inside IDE

Now let's create a functional test. Open the New File wizard. Select Paint project and choose 'Testing Tools' category and 'JellyTestCase Test' file type. Place it into some package under 'Functional Test Packages' (e.g. validation.OverallTest.java). Implement several test cases using Jemmy and Jellytools UI testing libraries. It can be the following:

package validation;

import org.netbeans.junit.NbTestSuite;
import org.netbeans.jellytools.*;
import org.netbeans.jellytools.actions.Action;
import org.netbeans.jemmy.operators.*;

public class OverallTest extends JellyTestCase {
    
    public OverallTest(String name) {
        super(name);
    }
    
    public static NbTestSuite suite() {
        NbTestSuite suite = new NbTestSuite();
        suite.addTest(new OverallTest("testBrushSize"));
        suite.addTest(new OverallTest("testPainting"));
        suite.addTest(new OverallTest("testClear"));
        suite.addTest(new OverallTest("testColorChooser"));
        return suite;
    }
    
    public static void main(java.lang.String[] args) {
        junit.textui.TestRunner.run(suite());
    }
    
    public void setUp() {
        System.out.println("########  "+getName()+"  #######");
    }

    public void testBrushSize() {
        new Action("File|New Canvas", null).perform();
        JSliderOperator slider = new JSliderOperator(MainWindowOperator.getDefault());
        slider.scrollToMaximum();
        slider.scrollToMinimum();
        slider.scrollToMaximum();
    }
    
    public void testPainting() {
        TopComponentOperator tcOper = new TopComponentOperator("Image");
        tcOper.clickMouse(tcOper.getCenterX(), tcOper.getCenterY(), 1);
        tcOper.dragNDrop(tcOper.getCenterX(), tcOper.getCenterY(), tcOper.getWidth()-1, tcOper.getHeight()-1);
        tcOper.dragNDrop(tcOper.getWidth()-1, tcOper.getHeight()-1, 0, tcOper.getHeight()-1);
        tcOper.dragNDrop(0, tcOper.getHeight()-1, tcOper.getCenterX(), tcOper.getCenterY());
    }
    
    public void testClear() {
        new JButtonOperator(new TopComponentOperator("Image"), "Clear").push();
    }
    
    public void testColorChooser() {
        fail("Not yet implemented.");
    }
}

Without any additional setup you are able to run functional tests. Just call context menu item 'XTest|Run qa-functional Tests' on Paint project node. XTest starts Paint Application and tests are executed against it. You can call also other actions like for unit tests - build, debug, measure code coverage. If a test case fails, test results contain screen shot at the time of failure and Jemmy logs. It helps to analyze the reason of the failure.

Install Hudson

Now you are able to build and test application based on NetBeans Platform inside IDE. Of course you can do the same from command line. But to be more productive you can think about automation of building and testing processes. Great idea is to setup continuous building environment which will run when source code changes. We will show how to use Hudson system.

Install Hudson web application as described in Hudson documentation. If everything went smoothly, you should be able to reach Hudson application at the address http://host:8080/hudson/. Hudson application uses working directory (HUDSON_HOME property) which we will use for storing our libraries. You can find out location of this directory in 'Manage Hudson|System Information'. If you unzip hudson.zip into hudson home directory and restart the server, you can only modify already created jobs and download libraries.

Build application using Hudson

To build our application we need NetBeans Platform distribution. Install or just unzip platform distribution into Hudson home. Now create a new Hudson job for building our application. Enter 'PaintApp-build' job name and move to configuration page. If you store Paint Application into versioning control system, supply required parameters. Otherwise pick None in Source Code Management section and to make it work, copy everything from PaintApp directory into $HUDSON_HOME/jobs/PaintApp-build/workspace.

You can also use sample application from NetBeans CVS repository. Then parameters are the following:

CVSROOT :pserver:anoncvs@cvs.netbeans.org:/cvs
Module(s) xtest/examples/PaintApp

In section Build check 'Invoke top-level Ant targets'. To clean and build application type in targets with absolute paths to platform:

clean build-zip -Dnetbeans.dest.dir=D:/Hudson/netbeans -Dharness.dir=D:/Hudson/netbeans/harness

To archive built products go to 'Post-build Actions' section, check 'Archive the artifacts' and type dist\*.zip in 'Files to archive' text field. Save the configuration and start building. It should be finished successfully in a minute. Now you can schedule building to run for example every 30 minutes or when something has changed in source code (only if versioning control system is used).

Test application using Hudson

We can also run our tests on Hudson. First we need to have all necessary libraries. You can download Jemmy, Jellytools and XTest or you can re-use distribution from your userdir (to get its location look at 'Help|About|Details' in IDE). Copy ${userdir}/xtest-distribution/** into ${HUDSON_HOME}, ${userdir}/modules/ext/jemmy.jar into ${HUDSON_HOME}/jemmy, ${userdir}/modules/ext/jelly2-nb.jar into ${HUDSON_HOME}/jellytools. Directory structure under ${HUDSON_HOME} should be the following:

${HUDSON_HOME}
    - jellytools
    - jemmy
    - jobs
    - netbeans
    - xtest-distribution

Now we have to create driver.properties file which contains paths to libraries and other properties. We put it to xtest-distribution/lib directory and content is this:

# auxiliary property
hudson.home=D:/Hudson
# which machine does the testing 
xtest.machine=builder
# which project is tested 
xtest.tested.project=Paint Application
xtest.tested.project_id=PaintApp
# which group tests this project
xtest.testing.group=Quality Engineering
# which kind of tests are run
xtest.tested.type=unit & qa-functional
# Path to file/directory used for installing IDE
ide.install.path=${hudson.home}/jobs/PaintApp-build/workspace/dist/paintit.zip
# What type installation will be used. Legal values are 'zip', 'installator' and 'dir'.
ide.install.type=zip
# Actions that driver should do
xtest.execute=install-ide,execute
# List of test repositories with config name from master-config
xtest.instance.cvs.workdir_PaintApp=${hudson.home}/jobs/PaintApp-build/workspace
xtest.instance.location_PaintApp=xtest/instance
xtest.instance.config_PaintApp=all-tests
# Where to store results
xtest.results=${hudson.home}/jobs/PaintApp-test/workspace/results
# Where to ship results
xtest.ship.results.to=${hudson.home}/xtest-pes/ship
# Testing libraries home
jemmy.home=${hudson.home}/jemmy
jellytools.home=${hudson.home}/jellytools
# NetBeans platform home
netbeans.dest.dir=${hudson.home}/netbeans
harness.dir=${hudson.home}/netbeans/harness

To be able to run together unit and functional tests or to run tests from more than one module we need to create so called XTest instance. It is a build script and configuration file. Sample build script can be found here. master-config.xml should contain one row for each test type:

<testconfig>
    <config name="all-tests">
        <module name="Paint" testtypes="unit" attributes="stable"/>
        <module name="Paint" testtypes="qa-functional" attributes="stable"/>
    </config>   
</testconfig>

Location of XTest instance is defined in driver.properties. Both files should be placed into PaintApp repository:

PaintApp
    - xtest
       - instance
           - build.xml
           - master-config.xml

Now we can finally create a new Hudson job for testing of our application. Enter 'PaintApp-test' job name and move to configuration page. In 'Source Code Management' section let None selected because we will reuse workspace of PaintApp-build job. To run test when building is finished check 'Build after other projects are built' in section 'Build Triggers' and type 'PaintApp-build' in the text field. In section Build check 'Invoke top-level Ant targets' and type the following:

-f ../../../xtest-distribution/lib/driver.xml

To archive test results go to 'Post-build Actions' section, check 'Archive the artifacts' and type results/** in 'Files to archive' text field. Save the configuration. Go to project 'PaintApp-build' and run it. When it finishes, 'PaintApp-test' is started. Wait until it is finished and then you can browse results. There are merged unit and qa-functional tests together.

Process results using XTest PES

To get test results of every build to a single place we can use XTest Publishing Engine Server (PES). It also allows to track history of failures which helps to discover regressions. First we have to download XTest PES distribution and unzip it into ${HUDSON_HOME}/xtest-pes. Then we set properties in xtest-pes/bin/xtest-pes.sh:

PES_HOME=D:/Hudson/xtest-pes
PES_CONFIG=$PES_HOME/bin/config.xml
JAVA_HOME=D:/jdk1.5.0_10

We don't need to set JAVA_HOME if we PES using Hudson. It is already defined. If you run it from command line, you have to set it. Now we create config.xml file.

<PESConfig loggingLevel="WARNING" incomingDir="D:/Hudson/xtest-pes/ship" workDir="D:/Hudson/xtest-pes/work">
    <PESWeb description="Paint Application PES Web" webroot="D:/Hudson/xtest-pes/web"/>
        <PESProjectGroup name="Paint Application" main="true" description="Paint Application Tests"
                         currentBuilds="10" detailedData="10" historyMatrices="10" deleteAge="20"/>
    <PESWeb/>
</PESConfig>

Configuration file belongs to xtest-pes/bin directory as defined in xtest-pes.sh:

${HUDSON_HOME}
    - xtest-pes
       - bin
          - config.xml
          - xtest-pes.sh

To see the results properly you should configure your web server. For example for Tomcat server you should add context definition to server.xml:

<Context path="/results" docBase="D:/Hudson/xtest-pes/web" debug="0" reloadable="true"/>

And enable listing of directories in web.xml:

<init-param>
    <param-name>listings</param-name>
    <param-value>true</param-value>
<init-param/>

Now we are prepared to run XTest PES. We can use command line but we can easily utilize Hudson. Create a new Hudson job and name it 'XTest-PES-run'. Go to section Build in configuration page, check 'Execute shell' and type the following command in:

../../../xtest-pes/bin/xtest-pes.sh run

Save the configuration and start it. After while you will find overall results from already executed tests at the address http://host:8080/results/. From now XTest PES will wait for incoming results and merge them into existing results. You can see summary of executed tests and history matrix of failed tests. All log files are available as well.

It is handy to create similar Hudson job to stop XTest PES. Create a new job and name it 'XTest-PES-stop'. Go to section Build in configuration page, check 'Execute shell' and type the following command in:

../../../xtest-pes/bin/xtest-pes.sh stop

If you run it, it will safely stop XTes PES.

Summary

Well done! We have installed powerful continuous building and testing system. You should have four jobs in Hudson dashboard:

Job	Last Success	Last Failure	Last Duration
PaintApp-build	3 minutes (#22)	23 hours (#18)	54 seconds
PaintApp-test	2 minutes (#17)	23 hours (#13)	48 seconds
XTest-PES-run	13 hours (#5)	N/A	13 hours
XTest-PES-stop	1 minute (#4)	N/A	2 seconds

If you use versioning control system and you configured PaintApp-build job to run automatically when something is changed in versioned source repository, you don't need to build and test application yourself. Hudson, XTest and other infrastructure does work for you. You just write a source code and tests.

xtest

Project Home
Memebership
Announcements
Discussion Forums
Mailing Lists
Documentation & Files
Version Control
Issue Tracker

NetBeans.org