Automatic Regression Testing

Version: $Id: XTest4Java.html,v 1.6 2003/10/06 10:15:30 miw Exp $
Authors: Mick Wever
Abstract: This document summarizes how to use XTest for a non NetBeans application.

Document History:
06.10.2003: Mick Wever, first draft.
17.10.2003 Martin Brehovsky, editing.

---

“Incremental testing and continuous builds are two of the cornerstones of the Extreme Programming methodology. Combine the two in a single, automated process -- and throw in automatically generated e-mail reports -- and you'll be well on your way to XP nirvana.”

Ant and JUnit bring you one step closer to XP nirvana

There is a lack of automation tools and statistical publishing of results for JUnit tests, XTest provides this automation and publishing around your collection of JUnit tests.

XTest is a testing framework used for testing Java products through JUnit, HttpUnit (and hopefully soon Cactus) type tests, and publishing the results into html pages. It integrates testing from multiple platforms, products, product versions, and test types, to provide upto date comprehensive statistical information about the status of your java code.

---

How to Configure XTest for Your Java Product.

This document assumes familarity with these XTest documents:

• XTest Home Page
• XTest Architecture Overview

1. XTest Installation Guide:
   Download the NbSources (only openide, core, xml/external , nbbuild, libs and xtest modules from CVS repository are necessary) and build the XTest.

   - See howto build NetBeans sources, an excellent comprehensive document on invovlement with the sources.

   - Run “ant clean makedist-pes” in NbSources/xtest to build your Publishing Engine Server (PES). This will create xtest-pes-distribution.zip, the clean command is necessary, because it removes parts of xtest not required by PES ...
   Note: You will need a graphical environment to do this as the third party binary licenses requiring accepting through a graphical client. There is a enhancement request for a command line interface for license acceptance.

   - The NbSources/xtest/lib can act as the XTest Server location, or you can download binary distributions for XTest version. The latest binary distribution should be always the same version as the one in sources.

2. Unzip the xtest-pes-distribution.zip somewhere, this will be your PES home. (xtest-distribution.zip contains the harness).

3. Copy the NbSources/xtest/examples/MyExample to your java product code base. This is a XTest Instance.
   Example:

   MyProduct/
    build/
    etc/
    javadoc/
    src/
    test/ <-- Copy 'MyExample' here, renaming it 'test'.
       

4. Edit build.xml to suit your product, and path configurations.

   Example:

       <!-- Name of tested module -->
       <property name="xtest.module" value="SamboWeb"/>
       <property name="xtest.results" value="results"/>
   
       <!-- Home of Xtest -->
       <property name="xtest.home" location="../../NbSources/xtest"/> 
      
       <!-- default testtypes, attributes used when no value is supplied from command line -->
       <property name="xtest.testtype" value="unit"/>
       <property name="xtest.attribs" value="all,code"/>
       
       <!-- Points to antfile with module's harness. 
            Targets from that antfile is called from this build.xml -->
       <property name="xtest.driver.antfile" location="${xtest.home}/lib/driver.xml"/>
       <property name="xtest.module_harness.antfile" location="${xtest.home}/lib/module_harness.xml"/>
   
       <!-- Says how to create classpath for compilation and for execution code tests by default.
            Can have values src, jar, ide -->
       <property name="xtest.source.location" value="src"/>
       
       <!-- Publishing Properties -->
       <property name="xtest.tested.project" value="${xtest.module}"/>
       <property name="xtest.testing.group" value="Daily"/>
       <property name="xtest.tested.type" value="${xtest.testtype}"/>  
       <!-- <property name="ide.build" value="0.2"/>   -->
       <property name="xtest.testing.team" value="Sambo Web Utviklere"/>   
       <property name="xtest.driver.backup" value="driver-backup"/>    
       <property name="xtest.ship.results.to" value="/home/cvs/XTestPublishingEngine/incoming"/>
       <property name="xtest.machine" value="sambotestapp.bokklubbene.no"/>

5. Remove IDE specific tests (NbJUnit) out of the example.

   Example from build.xml (same as above):

      <property name="xtest.attribs" value="all,code"/>

   Also comment out all testbag elements with attribute executor="ide" in cfg-unit.xml.

6. Enable only the unit tests to start with. You can leave the other test types in the folder, they can only help as further examples when you go to implement them.

   Example from build.xml (same as above):

       <property name="xtest.testtype" value="unit"/>
       

7. Add to build.xml to allow shipping of the results to the PES.

       <target name="ship-results">
            <echo message="Shipping results for build "/>
            <property name="archive.filename" 

   value="xtr-${xtest.tested.project}-${ide.build}-${xtest.testing.group}-${xtest.tested.type}-${xtest.machine}.zip"/>

            <property name="archive.file" value="${xtest.results}/${archive.filename}"/>
            <zip zipfile="${archive.file}">
               <fileset dir="${xtest.results}" excludes="xtr*.zip"/>                          
            </zip>
            <mkdir dir="${xtest.driver.backup}"/>
            <copy file="${archive.file}" toDir="${xtest.driver.backup}"/>
            <ant dir="." antfile="${xtest.driver.antfile}" target="ship-results"/>
            <delete file="${xtest.driver.backup}/${archive.filename}"/>  
   </target>

   - It is also possible to run the XTest Instance from the XTest Server but more work is required to decouple the IDE from Ant build files.

   - A patch also had to be made to NbSources/xtest/lib/module_harness.xml so that the testreport.xml file would be created every testrun.

8. Run your XTest Instance with Ant. Use «ant runtests»

9. Create the PES configuration file and start PES. (Use $PES_HOME/bin/xtest-pes.sh start).

Trouble Shooting

• Debugging PES can be done by editing $PES_HOME/bin/xtest-pes.sh and changing the comment between the two debug argument lines. (Search for 'debug').

• Make sure to define the project, version, and group attributes so PES knows how to unpack the testrun.

Extensions

• HttpUnit Tests, see already written documentation.
• Cactus Tests, [TODO]...

• Scarab (Bug Tracking System) Integration.

  Scarab provides a xmlrpc interface. Given this and issues that have a 'regressed' state in the 'status' attribute, and a 'test' text attribute. It is quite simple to make XTest change the status of any issue to regressed where its test field contains the name of failed test. Another approach is to simply append a comment to these issues ensuring the owner recieves an email that their test (and issue) has regressed.

  On the client side I inserted:

          <java failonerror="false" fork="true" classname="ScarabIntegration">
              <arg value="${xtest.results}/xmlresults/testreport-failures.xml"/>
              <classpath location="xmlrpc-1.2-b1.jar"/>
              <classpath location="jdom.jar"/>    
          </java>

  into the runtests target (after it had run and before it called the ship-results target). The class 'ScarabIntegration' uses jdom to collect the list of failed test names and their corresponding failure messages. This is sent via xmlrpc to the Scarab server which processes it. Example code of xmlrpc usage in Scarab can be found here. There has also been a few mails on this Scarab mailing list about this.

Just One Usage Example

• One example of Xtest's usage is at De norske Bokklubbene where along with different junit test types (unit, httpUnit, cactus), and platforms, tests are also grouped (by using the department attribute) into 'Weekly Build', 'Nightly Build', and 'CVS-user' builds corresponding to each cvs commit per user (using cvsroot/loginfo). This makes the production environment very resistant to regression because regressions can quickly be traced to the exact cvs commit and user.