HOME > Community > Projects > xtest

XTest Publishing Engine Server

Version: 1.6
Author: Martin Brehovsky

Abstract: This document describes a part of Test framework which handles processing results from tests on the web.

What is this document for?

This document is intended for test administrators who need to gather results from several testing machines and integrate them together to make them available from a single entry point.

Overview

Publishing Engine Server (also known as Publishing Server or just PES) gathers data delivered to the known directory, integrates them to already received results and creates corresponding web pages, including summary pages and history matrix pages. Currently, publishing server works only as a static service independent on HTTP server, all the pages are created only when new results are being processed.

Publishing server does not need any special configuration, nor supporting tools. All you need is JDK at least version 1.4 and very likely some HTTP server (Apache, ...), which will serve the results to the users.

The server itself works in a very simple manner. It watches directory where testing machines stores results from test runs (configurable on testing machines via XTest's driver ), checks if the results are not yet integrated, unpacks the archive with results and updates summary files. When all result files are processed, the server is done. The engine runs in an infinite cycle with some sleep between runs, so the results are kept up to date.

Understanding Structure of Results Directory

Because tests produce a huge amount of information, XTest organizes results in the following structure (please note the whole structure and all files are automatically generated by XTest, so you should not manually edit anything here):

results/
 |- index.html
 |- htmlresults/
     |- main-navigator.html
     |- systeminfo.html
     |- testreport.html
     |- testreport-failures.html
 |- testrun_${YYMMDD_HHMMSS-1}/
     |- htmlresults
     |- testbag_1/
         |- htmlresults
             |- testbag.html
             |- suites/
                 |- TEST-${suite-name-1}.html
                 |- ...
                 |- TEST-${suite-name-N}.html
         |- sys
             |- userdata.properties
             |- ide

                 |- messages.log
                 |- ide.zip
         |- user
             |-{test's user directories}
         |- xmlresults
             |- testbag.xml
             |- testbag-failues.xml
             |- suites/
                 |- TEST-${suite-name-1}.xml
                 |- ...
                 |- TEST-${suite-name-N}.xml
     |- .../
     |- testbag_N/
     |- xmlresults/
         |- testrun.xml
         |- testrun-failures.xml
 |- .../
 |- testrun_${YYMMDD_HHMMSS_N}/
 |- xmlresults/
     |- testreport.xml
     |- testreport-failures.xml

Under the 'results' directory, you can find 'index.html' file, which is a report index file (it is usually automatically used when your browser points to the results directory). Directory 'htmlresults' contains overall results in the html file format. The files stored there are: 'testreport.html' - main summary report from the run tests, 'testreport-failures.html' - report with all test's failures, 'main-navigator.html' - page with direct links to the particular testreports, modules, testbags, ..., and 'systeminfo.html' where is stored information about the system which run the tests. These html files are transformed from xml files in the 'xmlresults' directory (at the same level) by XTest when tests are run. The 'xmlresults' directory contains only two files: 'testreport.xml' - this file has overall information about all testruns and all testbags run, and 'testreport-failures.xml', where is stores information about all test failures.

Because in the report root directory is stored only overall information, there is one or more 'testrun_${YYMMDD_HHMMSS}' subdirectories, where results from one test run are stored. One test run usually means a set of tests run from one Test instance in one CVS repository. The name of the directory is constructed as 'testrun_YYMMDD_HHMMSS', where YYMMDD is year, month and day, and HHMMSS is hour, minute and second, when tests from the given instance were started. Under each 'testrun-...' directory you can find again 'htmlresults' and 'xmlresults'. The 'htmlresults' directory is empty for the time being, but this can be changed in the future. The 'xmlresults' directory contains xml files with information about this particular testrun. There are two files: 'testrun.xml' - this file has all information about testbags run in this testrun and 'testrun-failures.xml', which stores information about all test failures occurred in this testrun.

Because one testrun can contain one or more testbags, in the testrun directory is one or more testbag subdirectories. One subdirectory stores information about test run in one testbag (as configured by test developers in the config files - more information available here ). The name of the directory is generated automatically, as 'testbag-1' to 'testbag-N', as the testbags are run (please note, the name of the directory has nothing to do with the name of the testbag defined in the config file). As the parent directories, there are 'htmlresults' and 'xmlresults' again. In the 'htmlresults' directory there is information about the run testbag ('testbag.html') and all suites run in this testbag (files in the 'suites' subdirectory). The same pattern applies to 'xmlresults' directory. In the testbag directory, there are also 'sys' and 'user' subdirectories. The 'sys' directory contains 'userdata.properties' which is a standard property file with system properties passed to the tests (this mechanism allows to use system dependent values, like JDBC URLs, filesystem paths, etc. in the tests), and subdirectory 'ide' where is stored the whole IDE userdir ('ide.zip') and 'messages.log' from the IDE's user directory. The last directory left, is 'user' subdirectory of testbag. There tests store all their working files, logs, etc.

When all required tests are run by the XTest's driver, this whole structure is packed to the single archive file (a zip file), which is named as : 'xtr-${project}-${build-id}-${department}-${test-type}-${machine-name}.zip', where:

${project} is the name of the tested project (like NetBeans 3.3, NetBeans dev JDK1.3, ...)
${build-id} is the build ID of the tested project (e.g. 20020104-1452)
${department} is the name of the department which runs the tests (e.g. QA, RE, Development, ...)
${test-type} is the type of tests run (e.g. unit, functional, performance, validation, ...)
${machine-name} is usually the domain name of the machine (e.g. mytester.netbeans.org), but it can be also some logical name known among people running tests.

The driver also copies this zip to the 'results' directory of the PES, where PES looks for these zips and process them further.

Understanding Structure of Web Pages

Because the structure described above can hold only tests for one run of Driver on one testing machine, PES needs to organize results in the following structure, which is usually stored under some directory presented to the XTest users by a web server. PES is able to maintain all the directories and html files with appropriate links automatically, although if some set of tests needs to be deleted/moved within different groups, the user has to do this task manually and run PES in reconfigure mode. The structure of the web pages maintained by PES is:

tests_web_root/
 |- index.html
 |- group-${group-1}.html
 |- ...
 |- group-${group-N}.html
 |- _xmldata/
     |- webdata.xml
     |- groupdata-${main_group}.xml
     |- ...
     |- groupdata-${group-N}.xml
 |- ${project-1}/
     |- ${department-testtype-1}.html
     |- ...
     |- ${department-testtype-N}.html
     |- old-${department-test_type-1}.html
     |- ...
     |- old-${department-testtype-N}.html
     |- matrix-${department-test_type-1}-${machine-name-1}.html
     |- ...
     |- matrix-${department-test_type-N}-${machine-name-M}.html
     |- ${build-id-1}/
          |- ${department-1}-${test_type-1}/
               |- ${machine-name-1}/
                   |- {unpacked results from results archive}
               |- .../
               |- ${machine-name-2}/
               |- .../
               |- ${test_type-N}
          |- .../
          |- ${department-N}-${test_type-N}/
     |- .../
     |- ${build-id-N}/
 |- .../
 |- ${project_name-N}/

In the structure shown above are stored all results received from testing machines. The 'tests_web_root' is any directory at your system, which is usually further exported via web server for browsing by users. This directory contains 'index.xml' file, which serves as an overall results page containing summary information about all projects from the main group. Other possible html files in this directory contain summary information for other defined project groups (named as group-${group_name}.html).

The '_xmldata' directory contains all data required by PES for result processing. From these files are generated most of the html files maintained on the server, including 'index.html' or report files for particular projects. You don't need to keep any attention to these files, since they're all maintained by PES automatically.

Because PES can organize results from several projects, several departments, including several test types, when a new archive with test results is received by the server, under 'tests_web_root' directory is created a new directory named as ${project}, if not available, where is stored information about all builds tested by all departments in this project. This directory contains several html files, like ${department-testtype}.html, where is stored information about all builds tested by given department and by given test type, old-${department-testtype}.html containing information about older builds (number of builds after a build is considered as 'old' can be defined in the PES config file) and matrix-${department-testtype}-${machine-name}.html, which contains a 'history matrix' of failures from all tests run by the given department on the particular machine (number of builds used for history matrices can be also defined in the config file).

Results from the particular archives are stored even deeper, under ${build-id}/${depatrment}-${testy-type}/${machine-name}. Variables used in this path has exactly the same meaning as the variables explaining the name of the results archive in the previous section. The only difference is, that if the directory ${machine-name} already exists and contains results and PES process another results archive which should be placed in this directory, instead of ${machine-name} is used ${machine-name}-1 and so on.

Finally, under each ${machine-name} directory are stored results taken directly from the archive. The only thing changed in the results is 'navigator' page (left frame), which is retransformed to include links to other pages (summaries, history matrices) maintained by PES.

Before Running PES

Before running publishing server, make sure you have the following:

JRE or JDK, at version 1.4.
Ant version 1.4 or greater
HTTP server and directory where results from testing machines together with summary pages will be stored.
Exported directory (via NFS for Unix machines, via SMB for Windows machines) where testing machines will store test results. Please note, this directory has to have read/write access either for users running tests on testing machines or the user running publishing server.

Building PES

PES is no longer available within the XTest's binary distribution available from xtest.netbeans.org, so you need to rather build your own version. Build PES with the help of the following steps:

check out xtest, nbbuild, core/external and xml/external from cvs.netbeans.org in the standard directory structure used for building NetBeans (i.e. nb_all/, ...)
build bootstrap of nbbuild module (in the nbbuild directory type 'ant bootstrap')
build PES by typing 'ant makedist-pes' in the xtest directory

When successfully done, you should find xtest-pes-distribution.zip file in the nb_all/xtest directory. This archive contain PES binary distribution and can be unpacked in any suitable directory.

Configuring PES

Publishing server can be configured via XML configuration file. DTD file with explaining comments for the configuration file is available here. The following lines contain a simple example of such a config file (you can use it as a template for your own configuration file).

<?xml version="1.0"?>
<PESConfig
   incomingDir="/export/incoming"
   workDir="/mytools/xtest-pes">
   <Mail smtpHost="mymailer.netbeans.org" from="pes@netbeans.org" to="pes-admin@netbeans.org"/>
   <PESWeb description="NetBeans PES Web" webroot="/myweb/tests">        
      <PESProjectGroup name="NetBeans" main="true"  description="NetBeans trunk builds"  currentBuilds="10" detailedData="10" historyMatrices="10"/>
      <PESProjectGroup name="NetBeans3.3" main="false" description="NetBeans 3.3.* builds" detailedData="0">
	 <PESProject project="NetBeans3.3.*"/>
      </PESProjectGroup>
   </PESWeb>
</PESConfig>

In this example you can see a simple configuration file, which can be used for processing results from NetBeans tests.

The main element 'PESConfig' has to contain two attributes - 'incomingDir' - where PES looks for new archives with results. The 'incomingDir' also contains two subdirectories. One is named 'replace' and is intended for archives, which should rather replace already processed results (if any), than to simply add them. This can be used for example when rerunning tests for a given build, because the previous results mostly failed since there were network problems. The other directory is called 'invalid' and it contains all archives which are evaluated as invalid - i.e. they don't contain necessary data or the file is not an archive, etc. The second attribute 'workDir' is used for setting working directory, which PES uses for it's work.

The 'PESConfig' element contains two single nested elements - 'Mail' and 'PESWeb'. The first element - 'Mail' - is for specifying required information, so PES is able to send notification emails in the case of some exceptional states. This element is not mandatory and if not specified, then simply no email notifications are sent.

The other element - 'PESWeb' - describes a single web managed by PES. It's attributes are 'description' for setting a string describing the web and 'webroot', which contains a path to the directory where all pages for this web are created/stored.

Because one website can contain results from many tested projects, PES can organize projects in groups (element 'PESProjectGroup'), where each group has its own summary page. 'PESWeb' element have to contain exactly one 'main' group (main attribute is set to true) and optionally one or more secondary groups. Each group has it's name (attribute 'name' - name is used for naming the group summary html files), description (attribute 'description' - used as a title on summary pages). Other optional attribute is 'currenBuilds' - when defined, it sets a number of builds, which are considered as 'current'. Current builds are summarized on the main page of the group, but older builds are summarized on 'old page' (main page contains link to the page with old builds). Next optional attribute is 'detailedData'. When defined, it sets a number of builds for which all data and logs from tests (like test workdirs, test logs, ide userdirs) are preserved (for older builds are such data deleted, so the web does not unnecessarily occupy disk space). Finally when defined 'historyMatrices' attribute, it sets a number of builds used for creation of history matrix. If the attribute is not defined, then no matrices are created at all.

Each 'PESProjectGroup' element which is not main ('main' attribute is set to false or not set at all) have to contain one or more nested elements called 'PESProject'. Each 'PESProject' element defines which tested projects are included in the group. 'PESProject' is usually defined by project name (attribute 'project') and can be further refined by setting additional attributes - 'department' and 'testType'. All these three attributes can also include regular expressions (with syntax used by regex package in JDK 1.4 - for more information see this page or directly JavaDoc for java.util.regex.Pattern ) to define projects included in the group. Finally, the 'PESProject' element can also define an interval of builds included in the group. For this use attributes 'fromBuild' for setting the low boundary of builds to be included and/or 'toBuild' to set the high boundary. When 'fromBuild'/'toBuild' attributes are not set, then all builds are included.

In the following example you can see a slightly more complicated configuration file, which, besides standard processing of results as in the previous example, can be used for uploading results to XTest's central database (Please note, implementation of central database is currently not open sourced):

<?xml version="1.0"?>
<PESConfig
   incomingDir="/export/incoming"
   workDir="/mytools/xtest-pes"
   loggingLevel="INFO"
   databaseUploadPath="/mytools/dbfeeder/upload"
   team="NetBeansQA">
   <Mail smtpHost="smtp.netbeans.org" from="pes@netbeans.org" to="pes-admin@netbeans.org" loggingLevel="WARNING"/>
   <PESWeb description="NetBeans PES Web" webroot="/myweb/tests" webURL="http://www.netbeans.org/download/xtest-results/">        
      <PESProjectGroup name="NetBeans" main="true" description="NetBeans trunk builds" currentBuilds="10" detailedData="10" 
                       deleteAge="20" uploadToDatabase="true"/>
      <PESProjectGroup name="NetBeans3.3" main="false" description="NetBeans 3.3.* builds" detailedData="0" uploadToDatabase="false">
	 <PESProject project="NetBeans3.3.*"/>
      </PESProjectGroup>
   </PESWeb>
</PESConfig>

In this example, to 'PESConfig' element are added three attributes. First one is not related to database - 'loggingLevel' - says what level PES uses for writing out informatino on a console (default is WARNING, usually no reason to change it, this example is for demostration only). The other two attributes are related to database. Attribuet 'databaseUploadPath' contains a path to a directory where are uploaded results to be imported to central database via dbfeeder (for more information about dbfeeder, look at this page). Attribute 'team' contains a value, which uniquely describes the team running this instance of PES among the whole QA organization (basically this can be understood as a PES ID).

Element 'Mail' have just one more attribute - again 'loggingLevel' - it is now set as WARNING, which means all log messages with level WARNING and ABOVE are sent via email. Default loggingLevel is SEVERE and there is usually no reason to change it. Please note this element is not related to central database, it is just included as a part of more complex example.

Element 'PESWeb' contains also just one more attribute - 'webURL', this attribute defines a root URL of the pages with results generated by PES. This attribute is used by central database to locate detailed results, like working directories and logs (which are not usually stored to the database).

Elements "PESProjectGroup' contain one more important attribute concerning storage to database - attribute 'uploadToDatabase', which when set to true (default value), PES will upload all results from the group to database. When set to false, PES will not upload any result from the group to database. Please note, this attribute has effect only when 'databaseUploadPath' in 'PESConfig' is set, otherwise is simply ignored.

Finally, element 'PESProjectGroup' for group 'NetBeans' contain attribute 'deleteAge', which sets a number of builds to be available in this group. When any of the build is older than a specified age, it is deleted and cannot be longer accessed. This attribute is usually used for automatic deletion of results from older builds, which are no more important. Please note this attribute is not directly related to database, but can slighlty simplify PES administration (results are kept on server for a limited amount of time, thus after that amount of time they don't occupy disk space).

When done with the configuraion file, you will also need to configure PES' shell script (available in CVS, at nb_all/xtest/bin/xtest-pes.sh), which runs periodically the Ants script. Here you will need to set several variables:

PES_HOME - directory where is your home of XTest PES
PES_CONFIG - path to your PES configuration file
JAVA_HOME - directory where JDK 1.4 is installed

Running PES

When everything is properly configured, you can run PES via supplied shell script, which is available in CVS at nb_all/xtest/bin/xtest-pes.sh (please don't forget to properly set all required variables) . The script can be run in the following ways:

'xtest-pes.sh start' - runs PES in background
'xtest-pes.sh run' - runs PES in foreground
'xtest-pes.sh stop' - stops the server (if running) in a gentle way (the server will agree on the end), so it might take even a couple of minutes to finish it's work
'xtest-pes.sh kill' - kills unconditionally the server - this command is not recommended for general use, unless this is the only way to stop it.
'xtest-pes.sh reconfigure' - reconfigures web maintained by the run PES (e.g. when some results were manually deleted/moved, configuration file has drastically changed (). This command should be run only when PES is in offline mode and it might take even several hours to reconfigure web (depends on a number of results maintained by PES).

Please note, you should not forget also run a web server to make results available for other users.