How to Do Automated Batch Testing

This section shows different approaches to running automated tests and processing the results. The newer recommended approaches appear first, and the older approaches are included later for historical purposes. See also, Automated Batch Testing.

Processing Test Results

Uploading results to Squish Test Center

This section shows how to use squishrunner, the command-line program, to generate and upload XML test results directly to Squish Test Center.

For demonstration, we will look at the results produced by running the example suite_addressbook_py test suite we used before. The test case checks whether entering some data into a simple addressbook AUT works as expected.

When using the testcenter report generator, we provide the URL of a running instance of Squish Test Center. The squishserver must be already running as well.

squishrunner --testsuite suite_addressbook_py --reportgen testcenter,http://localhost:8800/project/MyProject?token=MyToken&label=MyLabelKey1%3DMyLabelValue1&label=OS%3DLinux&batch=MyBatch

The URL provided for Squish Test Center contains additional information in its query string that determines how it is organized in the results database.

• A project name, MyProject
• Some form of authentication, which could be a username and password, or in this case, an upload token, MyToken. See Squish Test Center documentation for details.
• 0 or more label=, each can contain a key=value pair, where the = is URL-encoded as %3D. The labels encoded in the URL above are MyLabelKey1=MyLabelValue1, and OS=Linux.
• A batch name, MyBatch, which can be an arbitrary string. Branch names or Version numbers are often used here.

Project Names, Batches and Labels are concepts used to organize and select test reports in the web interface, and are explained in more detail here.

After the results are uploaded, an HTML report can be seen by navigation from a web browser to the URL of Squish Test Center.

Generating HTML Test Results

In this section we will demonstrate how to generate Squish's results directly to HTML format.

Squish's use of HTML means that test results can easily be processed and viewed in a web browser.

For demonstration we will look at HTML results produced by running the example suite_addressbook_py test suite we used before. The test case checks whether entering some data into a simple addressbook AUT works as expected.

To generate the HTML report, we run the test suite using squishrunner—of course the squishserver must already be running:

squishrunner --testsuite suite_addressbook_py --reportgen html,/tmp/results

As you probably noticed, we used the --reportgen option again and provided html as a value. The syntax is the same as we used for other report types. Besides of mentioning the report type we also specified the results directory /tmp/results where the final data will end up. If the directory already exists and contains results from the previous executions, the new execution results will be appended to the existing ones and produce an aggregated report.

Since we specified html format, squishrunner will generate the HTML report in the specified directory. You can open the index.html results file in any web browser for viewing and analyzing.

Generating XML Test Results

One important part of automating test runs is to present the test results in an accessible format so that the testing team can immediately spot any problems that have occurred. Squish can save the results of tests in an XML format that is suitable for arbitrary processing. In this section we will present a Python script that converts results in Squish's XML output format to HTML.

Squish's use of XML means that test results can easily be processed into formats suitable for other tools to use. This makes it possible to integrate the results of Squish test runs into a test management system. The processing needed to transform Squish's XML format data works the same way whether the output is HTML or some other format, so even though this section shows the transformation from XML to HTML, the concepts and approaches that are illustrated can be used for any other kind of transformation that you need.

In order to convert the Squish results XML format, we must first understand its format, as described in Processing Test Results.

To put things in concrete terms we will look at an XML test results file produced by running an example suite_py test suite that contains one test case. The test case checks whether entering some data into a simple addressbook AUT works as expected.

To generate the XML report, we run the test suite using squishrunner—of course the squishserver must already be running:

squishrunner --testsuite suite_addressbook_py --reportgen xml3.5,/tmp/results

The valid values for --reportgen depend on the Squish version, and are listed in squishrunner –reportgen: Generating Reports.

Since we specified xml3.5 format, we get an XML results file similar to the one shown below:

<?xml version="1.0" encoding="UTF-8"?>
<SquishReport version="3.4" xmlns="http://www.froglogic.com/resources/schemas/xml3">
    <test type="testsuite">
        <prolog time="2015-06-19T11:22:27+02:00">
            <name><![CDATA[suite_test]]>%lt;/name>
            <location>
                <uri><![CDATA[file:///D:/downloads/simple]]>%lt;/uri>
            </location>
        </prolog>
        <test type="testcase">
            <prolog time="2015-06-19T11:22:27+02:00">
                <name><![CDATA[tst_case1]]></name>
                <location>
                    <uri><![CDATA[x-testsuite:/tst_case1]]>%lt;/uri>
                </location>
            </prolog>
            <verification>
                <location>
                    <uri><![CDATA[x-testcase:/test.py]]></uri>
                    <lineNo><![CDATA[2]]></lineNo>
                </location>
                <scriptedVerificationResult time="2015-06-19T11:22:27+02:00" type="PASS">
                    <scriptedLocation>
                        <uri><![CDATA[x-testcase:/test.py]]>%lt;/uri>
                        <lineNo><![CDATA[2]]></lineNo>
                    </scriptedLocation>
                    <text><![CDATA[Verified]]></text>
                    <detail><![CDATA[True expression]]></detail>
                </scriptedVerificationResult>
            </verification>
            <verification>
                <location>
                    <uri><![CDATA[x-testcase:/test.py]]></uri>
                    <lineNo><![CDATA[3]]></lineNo>
                </location>
                <scriptedVerificationResult time="2015-06-19T11:22:27+02:00" type="FAIL">
                    <scriptedLocation>
                        <uri><![CDATA[x-testcase:/test.py]]></uri>
                        <lineNo><![CDATA[3]]></lineNo>
                    </scriptedLocation>
                    <text><![CDATA[Comparison]]></text>
                    <detail><![CDATA['foo' and 'goo' are not equal]]></detail>
                </scriptedVerificationResult>
            </verification>
            <epilog time="2015-06-19T11:22:27+02:00"/>
        </test>
        <epilog time="2015-06-19T11:22:27+02:00"/>
    </test>
</SquishReport>

The xml3 XML format is described in The Report Format.

Automatically Running Tests

This section shows how to automate running the tests and calling the squishxml3html.py script to convert the XML reports into HTML. It was written before Squish had an HTML report generator, or Jenkins integration, and is documented here for academic/historical purposes only.

Here is an example of using squishxml3html.py directly in a shell or command prompt:

cd squish_dir
./python2/bin/python ./examples/regressiontesting/squishxml3html.py -d /home/myuser/squish_reports_html /home/myuser/squish_xml3_report/results.xml

After executing this, the HTML report will be in /home/myuser/squish_reports_html.

For automating this, we just need to define a list of test suites and hosts and then, using a loop, the test suites are run on all hosts using squishrunner. The resulting XML files are then automatically converted into HTML using the squishxml3html.py program. The file names for the HTML result contain the date, test suite and host to ensure that they are unique.

The squishruntests.py script provides the automation we need. It reads the information it requires from a simple .ini format file, by default runtests.ini, in the same directory as the squishruntests.py script itself. Another .ini file can be specified on the command line using the -i or --ini option.

Here is an example runtests.ini file—only the SQUISHDIR and SUITES key–values are mandatory, so the others can be omitted:

# Use ~ to stand for your HOME directory---on Windows or Unix
SQUISHDIR = ~/opt/squish/bin
SUITES = ~/testing/myapp/version1/suite_main \
        ~/testing/myapp/version1.1/suite_main \
        ~/testing/myapp/version1.1/suite_extra
RESULTSDIR = ~/testing/myapp/results
HOSTS = 127.0.0.1
PRESERVE = 0
ISO = 1

The SQUISHDIR key–value is mandatory and must be set to the path where the Squish executables are found.

The SUITES key–value is mandatory and must be set to one or more space-separated suite paths (each path must contain a suite.conf file).

The RESULTSDIR key–value is optional and should be set to the path where the results XML files and HTML files must go. This directory will be created if it doesn't exist. It defaults to the current directory.

The HOSTS key–value is optional and should be set to one or more space-separated host names or IP addresses. Every test suite will be executed on every host. It defaults to one host, 127.0.0.1, the local machine and using Squish's default port number. Each host name or IP may be followed by a colon and a port number. For example, 192.0.4.67:9812.

The PRESERVE key–value is optional and defaults to 0 (don't preserve the formatting of messages). If set to 1, the format of log messages and similar is preserved in the resultant HTML files.

The ISO key–value is optional and defaults to 0 (use the locale-specific date/time format); set it to 1 to use ISO 8601 date/time format.

As the example file illustrates, long lines can be split over multiple lines by escaping newlines. Also, ~ can be used to stand for your home directory.

When running this script it is assumed that squishserver is running on all the specified hosts, and that the AUT paths are set up so that the AUTs run by the test suites can be found.

The last step would be to have the squishruntests.py script run automatically—for example, once every night—to ensure that no regressions have been introduced into the AUT. This is usually done from a CI (Continuous Integration) system such as Jenkins.

Conclusion

This section showed how to set up automated test runs using Squish's command line tools and different report generators. The approach that gives you the most flexibility is uploading results to Squish Test Center, because from there it is easy to view the results in a web browser, or download the results to another squishide.