CygWin Advanced Setup

Support for Testing Frameworks

Introduction

When tests are run from a testing framework, it is possible to measure the coverage for each test separately. To do this, Coco must know when each test begins and ends, together with other information about the test. By adding some code to the tests, this is possible.

There are two ways to do it:

  1. Using the Coco library to write data to the .csexe file.
  2. Writing to the .csexe file directly.

The next two chapters describe the use of these methods in the general case. In the chapters after that, concrete examples are given for several testing frameworks.

Use of the Coco library

Most test frameworks provide a method to run some code before each test and to run some other code after each test. They usually have also a way to get the name of the test. If this is supported by the test framework, it also supports measuring code coverage for each test separately and writing it separately to the .csexe file.

In general, the test framework will also provide a way to get the test result and possibly other information about the test. This information can also be written to the .csexe file.

All this is done by calls to the Coco library. The library exists in two versions, one for C/C++ and one for C#, with the same functions but in different syntax. For the differences in the code see the examples below.

The following library call must occur before each test:

  1. Clearing the counters with __coveragescanner_clear(). At the beginning of a test, this function sets all the internal coverage counters of Coco to zero. This prevents that the measurements are polluted by data from earlier activity of the program.

    Function description: C/C++, C#.

The next library calls are used to set information that is later written to the .csexe file by __coveragescanner_save(). They can be run before or after the test and in any order.

  1. Setting the test name with __coveragescanner_testname(). In most frameworks, there is not just a name for each test but also one for the test suite to which it belongs. Coco supports this by allowing hierarchical names: strings with several parts that are separated by slashes. CoverageBrowser can display a list of hierarchical names as a tree.

    So when the name of the test suite is known, one can create names of the form Suite/Test and use them as parameters to __coveragescanner_testname().

    Calling this function is mandatory.

    Function description: C/C++, C#.

  2. Setting the test state with __coveragescanner_teststate(). This function is usually called after a test; it sets the test result that will be written to the {.csexe} file. But it can also be called before the test, e.g. to set the result to "SKIPPED"

    Calling this function is not mandatory.

    Function description: C/C++, C#.

  3. Writing HTML annotations, if necessary. It is possible to write arbitrary other data in text form to the .csexe file. This is done with the functions __coveragescanner_clear_html_comment() and __coveragescanner_add_html_comment().

    These functions use an internal variable to which __coveragescanner_add_html_comment() adds text. The text is stored persistently and therefore needs to be cleared before new information is added. So the way to create HTML comments is first to call __coveragescanner_clear_html_comment() and then to call __coveragescanner_add_html_comment() one or more times.

    Clear function description: C/C++, C#.
    Add function description: C/C++, C#.

After the test, the data is witten and some cleanup is done with the following library calls. They must be run in exactly this order.

  1. Saving of the test data. With __coveragescanner_save(), the test name, the test result, the HTML annotations (if existing) and the content of all coverage counters is written to the .csexe file. Afterwards, all counters are set to zero.

    Function description: C/C++, C#.

  2. Reset the test name. When the test program finishes, it writes coverage data about its activity after the last test to the .csmes file. These data are still bound to the last test name used.

    So to prevent this possible error, one sets the name of the current test to the empty string, with __coveragescanner_testname("").

Examples

This section is a short summary of the way the code must look like that must be inserted into the test framework code.

Since the new code refers to functions that only exist in instrumented code, they must be guarded by conditional compilation statements. There is a symbol __COVERAGESCANNER__ that only exists when the code is instrumented, and it can be used for that purpose.

General structure in C++

The following code is typically run before each test:

#ifdef __COVERAGESCANNER__
    // Code to get TESTNAME
    __coveragescanner_clear();
    __coveragescanner_testname(TESTNAME);
#endif

And this code is run after every test:

#ifdef __COVERAGESCANNER__
    // Code to get TESTRESULT and COMMENT
    __coveragescanner_teststate(TESTRESULT);
    __coveragescanner_clear_html_comment(); // if needed
    __coveragescanner_add_html_comment(COMMENT); // if needed
    __coveragescanner_save();
    __coveragescanner_testname("");
#endif

For an concrete implementation see the CppUnit section. There, a concrete version of the first code snippet appears in the startTest() member function, and the second snippet (without using the HTML comment functions) occurs in endTest().

General structure in C#

The code for C# is at this abstraction level very similar to that in C#. The only difference is that there is another syntax for conditional statements and that the Coco library functions have a prefix, CoverageScanner.

Code that runs before every test:

#if __COVERAGESCANNER__
    // Code to get TESTNAME
    CoverageScanner.__coveragescanner_clear();
    CoverageScanner.__coveragescanner_testname(TESTNAME);
#endif

Code that runs after every test:

#if __COVERAGESCANNER__
    // Code to get TESTRESULT and COMMENT
    CoverageScanner.__coveragescanner_teststate(TESTRESULT);
    CoverageScanner.__coveragescanner_clear_html_comment(); // if needed
    CoverageScanner.__coveragescanner_add_html_comment(COMMENT); // if needed
    CoverageScanner.__coveragescanner_testname("");
    CoverageScanner.__coveragescanner_save();
#endif

Writing data directly to the .csexe file

The second method to specify the test which was executed is to write the information directly to the .csexe file. This method is especially useful when each test is a separate program and the tests are run by a script.

The .csexe file is a text file; the test information consists of additional lines of text that are appended to the file.

As before, some information must be added before a test and some afterwards. The following must be done before each test:

  1. To set the name of a test, a line of the form
    *<name of the test>

    must be appended to the .csexe file.

    The character * must be the first character of the line. Everything between it and the end of line is treated as the test name.

    Note: You can begin the line with a paragraph sign (§) instead of *. The § must be the in Latin-1 encoding and has the numerical value of 167. Since this may lead to internationalization problems, we strongly discourage its use. It is only supported for compatibility with older versions of the program.

And after the test, the following can be done:

  1. To set the status of a test, a line of the form
    !<status>

    must be appended to the .csexe file.

    The character ! must be the first character of the line. Everything between it and the end of line is treated as the test status.

    The status can be one of the following strings:

    • PASSED: the test was successfully executed.
    • FAILED: the test was not successfully passed.
    • INCIDENT: the test was not successfully executed (similar to failed).
    • CHECK_MANUALLY: it was not possible to determinate if the test was successfully executed.
    • SKIPPED: the test was skipped.
  2. To append an execution comment, insert the contents of an HTML document just after the execution of the application.

    The full comment must follow the HTML syntax, but only the content of the <body> tag is used. A minimal comment might therefore be something like <html><body>My comment</body></html>.

    There may be more than one HTML comment, and the comments and the status declaration may occur in any order, but they all must be appended to the .csexe file after the test has been executed.

Example

The following batch file executes the test First Test and sets the status of the execution to CHECK_MANUALLY.

echo *First Test >> myapp.csexe
myapp
echo "<HTML><BODY>Execution of myapp</BODY></HTML>" >> myapp.csexe
echo !CHECK_MANUALLY >> myapp.csexe

CppUnit

CppUnit is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

The following code is an example of how this can be done:

#include <cppunit/TestListener.h>
#include <cppunit/BriefTestProgressListener.h>
#include <cppunit/CompilerOutputter.h>
#include <cppunit/extensions/TestFactoryRegistry.h>
#include <cppunit/TestResult.h>
#include <cppunit/TestResultCollector.h>
#include <cppunit/TestRunner.h>

class CoverageScannerListener : public CppUnit::TestListener
{
    public:
        CoverageScannerListener() {}

        void startTest( CppUnit::Test *test )
        {
            m_testFailed = false;
#ifdef __COVERAGESCANNER__
            int pos;
            // Adjusting the name of the test to display the tests
            // in a tree view in CoverageBrowser
            std::string testname = "CppUnit/" + test->getName();
            while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos )
                testname.replace( pos, 2, "/" );

            // Reset the code coverage data to get only the code coverage
            // of the actual unit test.
            __coveragescanner_clear();
            __coveragescanner_testname( testname.c_str() ) ;
#endif
        }

        void addFailure( const CppUnit::TestFailure &failure )
        {
            m_testFailed = true;
        }

        void endTest( CppUnit::Test *test )
        {
#ifdef __COVERAGESCANNER__
            // Recording the execution state in the coverage report
            if ( m_testFailed )
                __coveragescanner_teststate( "FAILED" );
            else
                __coveragescanner_teststate( "PASSED" );

            // Saving the code coverage report of the unit test
            __coveragescanner_save();
            __coveragescanner_testname( "" );
#endif
        }

    private:
        bool m_testFailed;
        // Prevents the use of the copy constructor and operator.
        CoverageScannerListener( const CoverageScannerListener &copy );
        void operator =( const CoverageScannerListener &copy );
};

int main( int argc, char* argv[] )
{
#ifdef __COVERAGESCANNER__
    __coveragescanner_install( argv[0] );
#endif
    // Create the event manager and test controller
    CPPUNIT_NS::TestResult controller;

    // Add a listener that colllects test result
    CPPUNIT_NS::TestResultCollector result;
    controller.addListener( &result );

    // Add a listener that print dots as test run.
    CPPUNIT_NS::BriefTestProgressListener progress;
    controller.addListener( &progress );

    // Add a listener that saves the code coverage information
    CoverageScannerListener coveragescannerlistener;
    controller.addListener( &coveragescannerlistener );

    // Add the top suite to the test runner
    CPPUNIT_NS::TestRunner runner;
    runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );
    runner.run( controller );

    return result.wasSuccessful() ? 0 : 1;
}

In the example, we do the following:

  1. We write a CppUnit listener class which records the code coverage of each unit test after it is completed.

    We want to be able to run the program with and without Coco. Therefore, we use the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Coco, without the need to #include anything.

    In the listener class, CppUnitListener, we use the following member functions:

    • startTest(): This function is called before each test begins.

      In it, we compute a test name with the information provided by CppUnit and pass it to the Coco library with __coveragescanner_testname(). This is the Execution Name in CoverageBrowser.

      We call the function __coveragescanner_clear() to empty the internal database and thus make sure that the coverage of the code that was executed before this test is ignored.

    • addFailure(): This function is called after a test fails. It just sets a flag that is used by the other functions.
    • endTest(): This function is called after a test has ended.

      It uses __coveragescanner_teststate() to record the execution status (PASSED or FAILED) and then saves the code coverage report itself with __coveragescanner_save().

  2. We call __coveragescanner_install() in the main() function.
  3. We add a listener to the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
    CoverageScannerListener coveragescannerlistener;
controller.addListener( &coveragescannerlistener );

Qt Test

Qt Test is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test.
  3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.
  4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest().
  5. Compile your project with code coverage enabled.

TestCoverageObject header:

#ifndef _TEST_COVERAGE_OBJECT_H
#define _TEST_COVERAGE_OBJECT_H
#include <QObject>
class TestCoverageObject : public QObject
{
  Q_OBJECT
  public:
    virtual void initTest() {}
    virtual void cleanupTest() {}
  protected slots:
    void init() ;
    void cleanup();
};
#endif

TestCoverageObject source:

#include "testcoverageobject.h"
#include <QTest>
#include <QMetaObject>
#include <QString>

void TestCoverageObject::init()
{
#ifdef __COVERAGESCANNER__
  __coveragescanner_clear();
#endif
  initTest();
}

void TestCoverageObject::cleanup()
{
  cleanupTest();
#ifdef __COVERAGESCANNER__
  QString test_name="unittest/";
  test_name+=metaObject()->className();
  test_name+="/";
  test_name+=QTest::currentTestFunction();
  __coveragescanner_testname(test_name.toLatin1());
  if (QTest::currentTestFailed())
    __coveragescanner_teststate("FAILED");
  else
    __coveragescanner_teststate("PASSED") ;
  __coveragescanner_save();
  __coveragescanner_testname("");
#endif
}

GoogleTest

GoogleTest is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure getting only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener gives an implementation example.
  3. Add this listener to the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()).
  4. Compile the unit test using CoverageScanner.
#include <gtest/gtest.h>
#include <stdlib.h>

class CodeCoverageListener : public    ::testing::TestEventListener
{
  public:
    // Fired before any test activity starts.
    virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired before each iteration of tests starts.  There may be more than
    // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration
    // index, starting from 0.
    virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration)
    {
    }

    // Fired before environment set-up for each iteration of tests starts.
    virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after environment set-up for each iteration of tests ends.
    virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired before the test case starts.
    virtual void OnTestCaseStart(const ::testing::TestCase& test_case)
    {
    }

    // Fired before the test starts.
    virtual void OnTestStart(const ::testing::TestInfo& test_info)
    {
#ifdef __COVERAGESCANNER__
      __coveragescanner_clear();
      std::string test_name=
        std::string(test_info.test_case_name())
        + '/'
        + std::string(test_info.name());
      __coveragescanner_testname(test_name.c_str());
#endif
    }

    // Fired after a failed assertion or a SUCCESS().
    virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result)
    {
    }

    // Fired after the test ends.
    virtual void OnTestEnd(const ::testing::TestInfo& test_info)
    {
#ifdef __COVERAGESCANNER__
      __coveragescanner_teststate("UNKNOWN");
      if (test_info.result())
      {
        if (test_info.result()->Passed())
          __coveragescanner_teststate("PASSED");
        if (test_info.result()->Failed())
          __coveragescanner_teststate("FAILED");
      }
      __coveragescanner_save();
#endif
    }

    // Fired after the test case ends.
    virtual void OnTestCaseEnd(const ::testing::TestCase& test_case)
    {
    }

    // Fired before environment tear-down for each iteration of tests starts.
    virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after environment tear-down for each iteration of tests ends.
    virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after each iteration of tests finishes.
    virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration)
    {
    }

    // Fired after all test activities have ended.
    virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test)
    {
    }
} ;

int main(int argc, char **argv){

  //initialize CoverageScanner library
#ifdef __COVERAGESCANNER__
  __coveragescanner_install(argv[0]);
#endif

  ::testing::FLAGS_gtest_output = "xml";
  ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);

  ::testing::InitGoogleTest(&argc, argv);       //init google test framework

  return RUN_ALL_TESTS();       //run all tests which are listed in this project
}

CxxTest

CxxTest is a unit test framework for C++. This environment can be easily adapted to get the code coverage from each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

    In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure we get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()).

    When a test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

    Finally, all test failure members of this class must be reimplemented to record the test failures.

  3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().
  4. Compile the unit test with CoverageScanner activated.

For example:

#include <cxxtest/TestRunner.h>
#include <cxxtest/TestListener.h>
#include <cxxtest/TestTracker.h>
#include <cxxtest/ValueTraits.h>
#include <cxxtest/ValueTraits.h>
#include <cxxtest/ErrorPrinter.h>

class CoverageScannerListener : public CxxTest::ErrorPrinter
{
  public:
    CoverageScannerListener(std::ostream &o=std::cout,
                            const char *preLine = ":",
                            const char *postLine = "")
        : CxxTest::ErrorPrinter( o, preLine , postLine ) {}

    int run()
    {
      return CxxTest::ErrorPrinter::run();
    }

    void enterTest( const CxxTest::TestDescription & desc)
    {
      test_passed=true;

#ifdef __COVERAGESCANNER__
      // Adjust the name of the test to display the tests
      // in a tree view in CoverageBrowser
      std::string testname="CxxTest/";
      testname += desc.suiteName();
      testname += "/";
      testname += desc.testName();

      // Reset the code coverage data to get only the code coverage
      // of the actual unit test.
      __coveragescanner_clear();
      __coveragescanner_testname(testname.c_str());
#endif
      return CxxTest::ErrorPrinter::enterTest( desc );
    }

    void leaveTest( const CxxTest::TestDescription & desc)
    {
#ifdef __COVERAGESCANNER__
      // Record the execution state in the coverage report
      if (test_passed)
        __coveragescanner_teststate("PASSED");
      else
        __coveragescanner_teststate("FAILED");

      // Save the code coverage report of the unit test
      __coveragescanner_save();
#endif
      return CxxTest::ErrorPrinter::leaveTest( desc );
    }

    void failedTest(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedTest( file, line, expression );
    }

    void failedAssert(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed = false;
      return CxxTest::ErrorPrinter::failedAssert( file, line, expression );
    }

    void failedAssertEquals(const char *file, int line,
                            const char *xStr, const char *yStr,
                            const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertEquals( file, line,
                                                        xStr, yStr,
                                                        x, y );
    }

    void failedAssertSameData(const char *file, int line,
                              const char *xStr, const char *yStr, const char *sizeStr,
                              const void *x, const void *y, unsigned size)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertSameData( file, line,
                                                          xStr, yStr, sizeStr,
                                                          x, y, size );
    }

    void failedAssertDelta(const char *file, int line,
                           const char *xStr, const char *yStr, const char *dStr,
                           const char *x, const char *y, const char *d)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertDelta( file, line,
                                                       xStr, yStr, dStr,
                                                       x, y, d );
    }

    void failedAssertDiffers(const char *file, int line,
                             const char *xStr, const char *yStr, const char *value)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertDiffers(file, line,
                                                        xStr, yStr, value );
    }

    void failedAssertLessThan(const char *file, int line,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertLessThan(file, line,
                                                         xStr, yStr,
                                                         x, y );
    }

    void failedAssertLessThanEquals(const char *file, int line,
                                    const char *xStr, const char *yStr,
                                    const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line,
                                                                xStr, yStr,
                                                                x, y );
    }

    void failedAssertRelation(const char *file, int line, const char *relation,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation,
                                                          xStr, yStr,
                                                          x, y);
    }

    void failedAssertPredicate(const char *file, int line, const char *predicate,
                               const char *xStr, const char *x )
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate,
                                                           xStr, x);
    }

    void failedAssertThrows(const char *file, int line, const char *expression,
                            const char *type, bool otherThrown)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression,
                                                        type, otherThrown );
    }

    void failedAssertThrowsNot(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression );
    }

  private:
    bool test_passed;
};

int main()
{
#ifdef __COVERAGESCANNER__
  __coveragescanner_install(argv[0]);
#endif

  // Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()"
  return CoverageScannerListener().run();
}

boost::test

boost::test is a unit test framework for C++ which is part of the boost libraries. To adapt it to get the code coverage from each unit test:

  1. Implement a TestObserver, which derives from boost::unit_test_framework::test_observer and store as BoostTestObserver.hpp.
  2. Also implement (within the same file) a boost::test::fixture and define it to be executed at the beginning and end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file should look something like this:
            #ifndef COVERAGE_TEST_OBSERVER_INCLUDED
        #define COVERAGE_TEST_OBSERVER_INCLUDED

        #include <boost/test/framework.hpp>
        #include <boost/test/tree/observer.hpp>
        #include <boost/test/included/unit_test.hpp>

        #ifdef __COVERAGESCANNER__

        class BoostTestObserver
            : public boost::unit_test_framework::test_observer
        {
        public:
            BoostTestObserver(const char* moduleName)
                : test_observer()
                , m_testSuiteName(moduleName)
                , m_isAnyTestcaseOpen(false)
            {}

            ~BoostTestObserver()
            {
                if (m_isAnyTestcaseOpen)
                    close_current_testcase();
            }

        ///////////////////////////////////////////////////////////////////////////////
        // test suite related events:

            virtual int priority()
            {
                return 1;
            }

            virtual void test_finish()
            {
                if (m_isAnyTestcaseOpen) {
                    close_current_testcase();
                }
            }

            virtual void test_aborted()
            {
                if (m_isAnyTestcaseOpen) {
                    close_current_testcase();
                }
            }

            const char* get_testsuite_name()
            {
                return m_testSuiteName.data();
            }

            // called in the very beginning of the test suite:
            virtual void test_start(
                boost::unit_test_framework::counter_t  NumberOfTestCases)
            {
                m_currentTestNumber = -1;
                m_numberOfTestCases = NumberOfTestCases;
                __coveragescanner_install(m_testSuiteName.c_str());
            }

        ///////////////////////////////////////////////////////////////////////////////
        // test case related events:

            // called before each testcase's start
            virtual void test_unit_start(
                boost::unit_test_framework::test_unit const &unit)
            {
                // skip any calls which may being triggered by the 'TestWrap' fixture
                if(unit.full_name().find('/') == -1) {
                    return;
                }

    if (++m_currentTestNumber)
                    close_current_testcase();

    m_currentCaseHasFailed = false;
                __coveragescanner_clear();
                m_currentTestName.assign( unit.full_name().c_str());
                __coveragescanner_testname(m_currentTestName.c_str());
                m_isAnyTestcaseOpen = true;
            }

            // should be called after each testcase has finished...
            // but which definitveley, for unknown reason is NOT
            virtual void test_unit_finish(
                boost::unit_test_framework::test_unit const &unit,
                unsigned number)
            {
                close_current_testcase();
            }

// workaround for missing 'test_unit_finish' event being not triggered:
            void close_current_testcase()
            {
                if (m_isAnyTestcaseOpen) {
                    __coveragescanner_teststate(m_currentCaseHasFailed
                                               ? "FAILED"
                                               : "PASSED");
                    __coveragescanner_save();
                    m_currentTestName.append("_invalid");
                    m_isAnyTestcaseOpen = false;
                }
            }

            // called once on each FAILED or PASSED result
            virtual void assertion_result(boost::unit_test::assertion_result result)
            {
                if (result == boost::unit_test::AR_FAILED)
                    m_currentCaseHasFailed = true;
            }

        protected: // Deprecated boost event:
            virtual void assertion_result(bool result)
            {
                if (!result) m_currentCaseHasFailed = true;
            }

        ///////////////////////////////////////////////////////////////////////////////
        //  member variables:

            std::string m_testSuiteName;
            std::string m_currentTestName;
            int         m_currentTestNumber;
            int         m_numberOfTestCases;
            bool        m_verboseLogging;
            bool        m_currentCaseHasFailed;
            bool        m_isAnyTestcaseOpen;
        };

        //////////////////
        // boost fixture:
        // will be called in the very beginning and in the end of executing test
        ///////////////////////////////////////////////////////////////////////////////

        struct TestWrap
        {
            BoostTestObserver observer;

TestWrap()
            : BoostTestObserver(
                boost::unit_test::framework::master_test_suite().argv[0])
            {
                boost::unit_test::framework::register_observer(observer);
            }

           ~TestWrap()
            {
                observer.close_current_testcase();
                boost::unit_test::framework::deregister_observer(observer);
            }
        };

        ///////////////////////////////////////////////////////////////////////////////

        BOOST_GLOBAL_FIXTURE(TestWrap);

          #endif
        #endif
  3. If it is included by any of your already existing boost::test unit test .cpp files (must be done right after the mandatory BOOST_TEST_MODULE definition), it is being instantiated and initialized by boost::test automatically.
  4. Then, boost::test will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section being executed. Also, it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case's end.
  5. Additionally here is a small example which shows how to include this into your already existing boost::test modules:
    // define the name for yourBoost::Test module.
// (*this must be done BEFORE including BoostTestObserver.hpp)
#define BOOST_TEST_MODULE MyBoostTest
#include <boost/test/framework.hpp>
#include <boost/test/tree/observer.hpp>
#include <boost/test/included/unit_test.hpp>

// any of your already existing Boost::Test modules can be made
// compatible to Coco by including "BoostTestObserver.hpp"
// just after the module's BOOST_TEST_MODULE definition.
#include "BoostTestObserver.hpp"

// declare a boost test suite:
// (when "BoostTestObserver.hpp" is included, this will
// automatically instantiate a BoostTestObserver then.)
BOOST_AUTO_TEST_SUITE(BOOST_TEST_MODULE)

    // declare first test case:
    BOOST_AUTO_TEST_CASE(testA)
    {
        // code for testcase A
    }

     //...  declare any other testcases

    // last testcase:
    BOOST_AUTO_TEST_CASE(testX)
    {//...
    }

// in the end of the test, close the suite's scope
// (which at least also will unload our BoostTestObserver)
BOOST_AUTO_TEST_SUITE_END()
  6. Compile the boost test with CoverageScanner activated.

xUnit

When Coco is used with xUnit, it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test.

The tests themself need not be instrumented, but the code that is tested must be built with CoverageScanner. A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated.

Proceed as follows:

  1. Add to your xUnit test library the following code that provides a set of functions to handle the coverage information and to save the execution report (the .csexe file):
    using System;
using System.Collections.Generic;
using System.Text;
using Xunit;
using System.Reflection;
using Xunit.Sdk;
using System.Diagnostics;
using Xunit.Abstractions;

class Coverage
{
    public enum State { PASSED, FAILED, CHECK_MANUALLY, UNKNOWN };

    public static void CoverageCleanup()
    {
        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                // clear all coverage information to only get the code coverage of the current executed unit test
                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
            }
        }
    }

    public static void CoverageTextLog(String text, bool bold, bool italic)
    {
        String style_begin = "";
        String style_end = "";
        if (italic)
        {
            style_begin += "<I>";
            style_end += "</I>";
        }
        if (bold)
        {
            style_begin += "<B>";
            style_end += "</B>";
        }

        String comment = "<HTML><BODY><TT>"
                        + style_begin
                        + text.Replace("&", "&amp;")
                                .Replace("<", "&lt;")
                                .Replace(">", "&gt;")
                                .Replace("\"", "&quot;")
                                .Replace("'", "&apos;")
                                .Replace("\n", "<BR>")
                                .Replace("\r", "")
                        + style_end
                        + "</TT></BODY></HTML>"
                        ;
        CoverageHtmlLog(comment);
    }

    public static void CoverageHtmlLog(String comment)
    {
        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                coco.InvokeMember("__coveragescanner_add_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { comment });
            }
        }
    }

    public static void CoverageRecord(String csexe_filename, String testName, State result)
    {
        if (csexe_filename == "")
            return;

        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                String name = testName;
                String state = "";

                switch (result)
                {
                    case State.FAILED:
                        state = ("FAILED");
                        break;
                    case State.CHECK_MANUALLY:
                        state = ("CHECK_MANUALLY");
                        break;
                    case State.PASSED:
                        state = ("PASSED");
                        break;
                }

                name = name.Replace('.', '/');

                // set the execution state: PASSES, FAILED or CHECK_MANUALLY
                coco.InvokeMember("__coveragescanner_teststate", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { state });

                if (name.Length > 0)
                    // Test name: Namespace/Class/Testfunction
                    coco.InvokeMember("__coveragescanner_testname", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { name });

                // File name is <classname>.csexe
                coco.InvokeMember("__coveragescanner_filename", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { csexe_filename });
                // saves the code coverage data
                coco.InvokeMember("__coveragescanner_save", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                // clear all coverage information to only get the code coverage of the current executed unit test
                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
            }
        }
    }
}
  2. Add to your xUnit test library the following code that contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline:
    using System;
using System.Collections.Generic;
using System.Text;
using Xunit;
using System.Reflection;
using Xunit.Sdk;
using System.Diagnostics;
using Xunit.Abstractions;

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public class SquishCocoHooks : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo methodUnderTest)
    {
        Coverage.CoverageCleanup();
    }

    public override void After(MethodInfo methodUnderTest)
    {
        string base_name = "xUnit";
        string method_name = methodUnderTest.Name;
        string class_name = methodUnderTest.ReflectedType.FullName;

        Coverage.CoverageRecord("coverage.csexe", base_name + "/" + class_name + "/" + method_name, Coverage.State.UNKNOWN);
    }
}

public class CustomFixture : IDisposable
{
    public CustomFixture()
    {
        // Some initialization
    }

    public void Dispose()
    {
        // Some cleanup
    }
}

[CollectionDefinition("SquishCoco")]
[SquishCocoHooks]
public class SquishCocoDefinition :  ICollectionFixture<CustomFixture>
{
}
  3. Activate the collection for each test class by adding [Collection("SquishCoco")] to the definition of each test class. For example:
    [Collection("SquishCoco")]
public class Class2Tests
{
    [Fact()]
    public void dummyTest()
    {
        Assert.True(false, "Dummy test");
    }
}

After running the tests, a file coverage.csexe will be generated. It can then be imported into the corresponding coverage database (.csmes file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately.

NUnit

Coco provides an add-in for NUnit version 2.4.4 and above as sample. To install it:

  1. Build NUnitSquishCoco.dll using the Visual Studio project NUnitSquishCoco.vsproj provided in the sample directory.
  2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
  3. Start NUnit.exe and verify that the add-in NUnit \COCO is loaded.

Once installed, as soon as NUnit's test driver is executing a C# or C++ managed unit test test.dll, it generates a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application's instrumentation database with CoverageBrowser or cmcsexeimport.

Catch2

Catch2 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.

Complete code example:

*
    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
    #include <string>
    #include <deque>

    class CoverageScannerListener : public Catch::TestEventListenerBase
    {
    public:
        using TestEventListenerBase::TestEventListenerBase;

        virtual void sectionStarting( Catch::SectionInfo const& sectionInfo ) override
        {
            m_testNameHierarchy.push_back( sectionInfo.name );
    #ifdef __COVERAGESCANNER__
            // Adjusting the name of the test for the CoverageBrowser
            std::string testname = implodeTestNameHierarchy();
            // Reset the code coverage data to get only the code coverage
            // of the actual unit test.
            __coveragescanner_clear();
            __coveragescanner_testname( testname.c_str() );
    #endif
        }

        virtual void sectionEnded( Catch::SectionStats const& sectionStats ) override
        {
            m_testNameHierarchy.pop_back();
    #ifdef __COVERAGESCANNER__
            // Recording the execution state in the coverage report
            if ( sectionStats.assertions.allPassed() )
                __coveragescanner_teststate( "PASSED" );
            else
                __coveragescanner_teststate( "FAILED" );
            // Saving the code coverage report of the unit test
            __coveragescanner_save();
            __coveragescanner_testname( "" );
    #endif
        }

    private:
        std::deque<std::string> m_testNameHierarchy;

        /* Helper method which generates a hierarchical name string usable by the coveragebrowser. */
        std::string implodeTestNameHierarchy() {
            std::string fullTestName ( "Catch2" );
            std::deque<std::string>::const_iterator it = m_testNameHierarchy.cbegin();
            while ( it != m_testNameHierarchy.cend() ) {
                fullTestName += "/" + *it;
                it++;
            }
            return fullTestName;
        }
    };

    // Register the coveragescanner listener with Catch2
    CATCH_REGISTER_LISTENER( CoverageScannerListener )

    int main( int argc, char* argv[] )
    {
    #ifdef __COVERAGESCANNER__
        __coveragescanner_install( argv[0] );
    #endif
        return Catch::Session().run( argc, argv );
    }

To adapt Catch2:

  1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 provides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header:
    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
  2. Create a Catch2 listener class inheriting Catch::TestEventListenerBase which records the code coverage of each section after it is completed. Note that testcases also count as sections in Catch2, so you do not need to listen to testcase events specifically.

    In the created listener class (CoverageScannerListener), use the following member functions:

    • sectionStarting(): This function is called before each testcase and section begins to compute a test name with the information provided by Catch2 and pass it to the Coco library with __coveragescanner_testname().

      We also call the function __coveragescanner_clear() to empty the internal database and so make sure that the coverage of the code that was executed before this test is ignored.

    • sectionEnded(): This function is called after a testcase or section has ended.

      It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  3. We add this listener by adding the following Catch2 macro:
    CATCH_REGISTER_LISTENER( CoverageScannerListener )

Note: Remember to exclude any test framework sources from instrumentation (see Beyond the minimal instrumentation).

Squish

Run the GUI testing tool Squish together with Coco to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Coco features the comparison of individual executions, including the calculation of an optimal order.

General approach

The approach depicted below is based on the possibility to apply information about the name, the result, and a free-form comment to each execution (stored in .csexe files). See Test suites and Coco.

As an example we'll use Squish for Qt's addressbook on a Unix-based system and a JavaScript test script:

  1. First we'll initialize the execution data with the name of the Squish test case that is being run.
    function main()
{
    var currentAUT = currentApplicationContext();
    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
    var testCase = squishinfo.testCase;
    var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1);
    var file = File.open(execution, "a");
    file.write("*" + testExecutionName + "\n");
    file.close();
    var ctx = startApplication("addressbook");
...
  2. Insert the main test script at this point.
  3. After the main test script we'll log the result of the test for the coverage tool:
    ...
    // wait until AUT shutdown
    while (ctx.isRunning) {
        snooze(1); // increase time if not enough to dump coverage data
    }

    // test result summary and status
    var positive = test.resultCount("passes");
    var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals");
    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
    var status = negative == 0 ? "PASSED" : "FAILED";
    var file = File.open(execution, "a");
    file.write("<html><body>" + msg + "</body></html>\n");
    file.write("!" + status + "\n")
    file.close();
}

When you execute the scripts containing these steps, the Coco Execution Report loads with the test case name, status and execution summary in the execution details and comments.

Simplified for reuse

  1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
    function getExecutionPath() {
    var currentAUT = currentApplicationContext();
    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
    return execution;
}

function logTestNameToCocoReport(currentTestCase, execution) {
    var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1);
    var file = File.open(execution, "a");
    file.write("*" + testExecutionName + "\n");
    file.close();
}

function logTestResultsToCocoReport(testInfo, execution){

    var currentAUT = currentApplicationContext();

    // wait until AUT shuts down
    while (currentAUT.isRunning)
      snooze(5);

    // collect test result summary and status
    var positive = testInfo.resultCount("passes");
    var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +
        testInfo.resultCount("fatals");
    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
    var status = negative == 0 ? "PASSED" : "FAILED";

    // output results and status to Coco execution report file
    var file = File.open(execution, "a");
    file.write("<html><body>" + msg + "</body></html>\n");
    file.write("!" + status + "\n")
    file.close();
}

    A Python version of this code is:

    import re

def getExecutionPath():
    currentAUT = currentApplicationContext()
    execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name}
    return execution

def logTestNameToCocoReport(currentTestCase, execution):
    testExecutionName = re.search(r'[^\\]\w*$', currentTestCase)
    testExecutionName = testExecutionName.group(0)
    file = open(execution, "a")
    file.write("*" + testExecutionName + "\n")
    file.close()

def logTestResultsToCocoReport(testInfo, execution):
    currentAUT = currentApplicationContext()
    # wait until AUT shuts down
    while (currentAUT.isRunning):
        snooze(5)

    # collect test result summary and status
    positive = testInfo.resultCount("passes")
    negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals")
    msg = "TEST RESULTS - Passed: %(positive)s  |  Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative}
    if negative == 0:
        status = "PASSED"
    else:
        status = "FAILED"

    # output results and status to Coco execution report file
    file = open(execution, "a")
    file.write("<html><body>" + msg + "</body></html>\n")
    file.write("!" + status + "\n")
    file.close()
  2. Add the following function calls after startApplication() in the main test script:
    execution = getExecutionPath();

logTestNameToCocoReport(squishinfo.testCase, execution);

    In Python:

    execution = getExecutionPath()

logTestNameToCocoReport(squishinfo.testCase, execution)
  3. At the end of your script, after closing the AUT (for example, by clicking File > Exit), call the following function:
    logTestResultsToCocoReport(test, execution);

    In Python:

    logTestResultsToCocoReport(test, execution)
  4. If your AUT closes unexpectedly or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.

Your main test script should be similar to the following:

source(findFile("scripts","squishCocoLogging.JS"))

function main()
{
    startApplication("addressbook");
    execution = getExecutionPath();
    logTestNameToCocoReport(squishinfo.testCase, execution);

    try {
        // body of script
    }
    catch(e) {
       test.fail('An unexpected error occurred', e.message)
    }
    finally {
       logTestResultsToCocoReport(test, execution)
    }
}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():
    startApplication("addressbook")
    execution = getExecutionPath()
    logTestNameToCocoReport(squishinfo.testCase, execution)

    try:
        try:
            # body of script
        except Exception as e:
            test.fail("test failed: ", e)
    finally:
        logTestResultsToCocoReport(test,execution)

CygWin Advanced Setup

Coco v7.1.0 ©2024 The Qt Company Ltd.
Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.