Support for IDEs and toolchains Installing Coco in Docker

Support for specific test frameworks

CppUnit

CppUnit is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

The following code is an example of how this can be done:

#include <cppunit/TestListener.h>
#include <cppunit/BriefTestProgressListener.h>
#include <cppunit/CompilerOutputter.h>
#include <cppunit/extensions/TestFactoryRegistry.h>
#include <cppunit/TestResult.h>
#include <cppunit/TestResultCollector.h>
#include <cppunit/TestRunner.h>

class CoverageScannerListener : public CppUnit::TestListener
{
    public:
        CoverageScannerListener() {}

        void startTest( CppUnit::Test *test )
        {
            m_testFailed = false;
#ifdef __COVERAGESCANNER__
            int pos;
            // Adjusting the name of the test to display the tests
            // in a tree view in CoverageBrowser
            std::string testname = "CppUnit/" + test->getName();
            while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos )
                testname.replace( pos, 2, "/" );

            // Reset the code coverage data to get only the code coverage
            // of the actual unit test.
            __coveragescanner_clear();
            __coveragescanner_testname( testname.c_str() ) ;
#endif
        }

        void addFailure( const CppUnit::TestFailure &failure )
        {
            m_testFailed = true;
        }

        void endTest( CppUnit::Test *test )
        {
#ifdef __COVERAGESCANNER__
            // Recording the execution state in the coverage report
            if ( m_testFailed )
                __coveragescanner_teststate( "FAILED" );
            else
                __coveragescanner_teststate( "PASSED" );

            // Saving the code coverage report of the unit test
            __coveragescanner_save();
            __coveragescanner_testname( "" );
#endif
        }

    private:
        bool m_testFailed;
        // Prevents the use of the copy constructor and operator.
        CoverageScannerListener( const CoverageScannerListener &copy );
        void operator =( const CoverageScannerListener &copy );
};

int main( int argc, char* argv[] )
{
#ifdef __COVERAGESCANNER__
    __coveragescanner_install( argv[0] );
#endif
    // Create the event manager and test controller
    CPPUNIT_NS::TestResult controller;

    // Add a listener that colllects test result
    CPPUNIT_NS::TestResultCollector result;
    controller.addListener( &result );

    // Add a listener that print dots as test run.
    CPPUNIT_NS::BriefTestProgressListener progress;
    controller.addListener( &progress );

    // Add a listener that saves the code coverage information
    CoverageScannerListener coveragescannerlistener;
    controller.addListener( &coveragescannerlistener );

    // Add the top suite to the test runner
    CPPUNIT_NS::TestRunner runner;
    runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() );
    runner.run( controller );

    return result.wasSuccessful() ? 0 : 1;
}

In the example, we do the following:

  1. We write a CppUnit listener class which records the code coverage of each unit test after it is completed.

    We want to be able to run the program with and without Coco. Therefore, we use the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Coco, without the need to #include anything.

    In the listener class, CppUnitListener, we use the following member functions:

    • startTest(): This function is called before each test begins.

      In it, we compute a test name with the information provided by CppUnit and pass it to the Coco library with __coveragescanner_testname(). This is the Execution Name in CoverageBrowser.

      We call the function __coveragescanner_clear() to empty the internal database and thus make sure that the coverage of the code that was executed before this test is ignored.

    • addFailure(): This function is called after a test fails. It just sets a flag that is used by the other functions.
    • endTest(): This function is called after a test has ended.

      It uses __coveragescanner_teststate() to record the execution status (PASSED or FAILED) and then saves the code coverage report itself with __coveragescanner_save().

  2. We call __coveragescanner_install() in the main() function.
  3. We add a listener to the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
    CoverageScannerListener coveragescannerlistener;
controller.addListener( &coveragescannerlistener );

Qt Test

Qt Test is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test.
  3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.
  4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest().
  5. Compile your project with code coverage enabled.

TestCoverageObject header:

#ifndef _TEST_COVERAGE_OBJECT_H
#define _TEST_COVERAGE_OBJECT_H
#include <QObject>
class TestCoverageObject : public QObject
{
  Q_OBJECT
  public:
    virtual void initTest() {}
    virtual void cleanupTest() {}
  protected slots:
    void init() ;
    void cleanup();
};
#endif

TestCoverageObject source:

#include "testcoverageobject.h"
#include <QTest>
#include <QMetaObject>
#include <QString>

void TestCoverageObject::init()
{
#ifdef __COVERAGESCANNER__
  __coveragescanner_clear();
#endif
  initTest();
}

void TestCoverageObject::cleanup()
{
  cleanupTest();
#ifdef __COVERAGESCANNER__
  QString test_name="unittest/";
  test_name+=metaObject()->className();
  test_name+="/";
  test_name+=QTest::currentTestFunction();
  __coveragescanner_testname(test_name.toLatin1());
  if (QTest::currentTestFailed())
    __coveragescanner_teststate("FAILED");
  else
    __coveragescanner_teststate("PASSED") ;
  __coveragescanner_save();
  __coveragescanner_testname("");
#endif
}

GoogleTest

GoogleTest is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure getting only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener gives an implementation example.
  3. Add this listener to the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()).
  4. Compile the unit test using CoverageScanner.
#include <gtest/gtest.h>
#include <stdlib.h>

class CodeCoverageListener : public    ::testing::TestEventListener
{
  public:
    // Fired before any test activity starts.
    virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired before each iteration of tests starts.  There may be more than
    // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration
    // index, starting from 0.
    virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration)
    {
    }

    // Fired before environment set-up for each iteration of tests starts.
    virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after environment set-up for each iteration of tests ends.
    virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired before the test case starts.
    virtual void OnTestCaseStart(const ::testing::TestCase& test_case)
    {
    }

    // Fired before the test starts.
    virtual void OnTestStart(const ::testing::TestInfo& test_info)
    {
#ifdef __COVERAGESCANNER__
      __coveragescanner_clear();
      std::string test_name=
        std::string(test_info.test_case_name())
        + '/'
        + std::string(test_info.name());
      __coveragescanner_testname(test_name.c_str());
#endif
    }

    // Fired after a failed assertion or a SUCCESS().
    virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result)
    {
    }

    // Fired after the test ends.
    virtual void OnTestEnd(const ::testing::TestInfo& test_info)
    {
#ifdef __COVERAGESCANNER__
      __coveragescanner_teststate("UNKNOWN");
      if (test_info.result())
      {
        if (test_info.result()->Passed())
          __coveragescanner_teststate("PASSED");
        if (test_info.result()->Failed())
          __coveragescanner_teststate("FAILED");
      }
      __coveragescanner_save();
#endif
    }

    // Fired after the test case ends.
    virtual void OnTestCaseEnd(const ::testing::TestCase& test_case)
    {
    }

    // Fired before environment tear-down for each iteration of tests starts.
    virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after environment tear-down for each iteration of tests ends.
    virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test)
    {
    }

    // Fired after each iteration of tests finishes.
    virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration)
    {
    }

    // Fired after all test activities have ended.
    virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test)
    {
    }
} ;

int main(int argc, char **argv){

  //initialize CoverageScanner library
#ifdef __COVERAGESCANNER__
  __coveragescanner_install(argv[0]);
#endif

  ::testing::FLAGS_gtest_output = "xml";
  ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);

  ::testing::InitGoogleTest(&argc, argv);       //init google test framework

  return RUN_ALL_TESTS();       //run all tests which are listed in this project
}

CxxTest

CxxTest is a unit test framework for C++. This environment can be easily adapted to get the code coverage from each unit test:

  1. Call __coveragescanner_install() in the main() function.
  2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

    In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure we get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()).

    When a test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

    Finally, all test failure members of this class must be reimplemented to record the test failures.

  3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().
  4. Compile the unit test with CoverageScanner activated.

For example:

#include <cxxtest/TestRunner.h>
#include <cxxtest/TestListener.h>
#include <cxxtest/TestTracker.h>
#include <cxxtest/ValueTraits.h>
#include <cxxtest/ValueTraits.h>
#include <cxxtest/ErrorPrinter.h>

class CoverageScannerListener : public CxxTest::ErrorPrinter
{
  public:
    CoverageScannerListener(std::ostream &o=std::cout,
                            const char *preLine = ":",
                            const char *postLine = "")
        : CxxTest::ErrorPrinter( o, preLine , postLine ) {}

    int run()
    {
      return CxxTest::ErrorPrinter::run();
    }

    void enterTest( const CxxTest::TestDescription & desc)
    {
      test_passed=true;

#ifdef __COVERAGESCANNER__
      // Adjust the name of the test to display the tests
      // in a tree view in CoverageBrowser
      std::string testname="CxxTest/";
      testname += desc.suiteName();
      testname += "/";
      testname += desc.testName();

      // Reset the code coverage data to get only the code coverage
      // of the actual unit test.
      __coveragescanner_clear();
      __coveragescanner_testname(testname.c_str());
#endif
      return CxxTest::ErrorPrinter::enterTest( desc );
    }

    void leaveTest( const CxxTest::TestDescription & desc)
    {
#ifdef __COVERAGESCANNER__
      // Record the execution state in the coverage report
      if (test_passed)
        __coveragescanner_teststate("PASSED");
      else
        __coveragescanner_teststate("FAILED");

      // Save the code coverage report of the unit test
      __coveragescanner_save();
#endif
      return CxxTest::ErrorPrinter::leaveTest( desc );
    }

    void failedTest(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedTest( file, line, expression );
    }

    void failedAssert(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed = false;
      return CxxTest::ErrorPrinter::failedAssert( file, line, expression );
    }

    void failedAssertEquals(const char *file, int line,
                            const char *xStr, const char *yStr,
                            const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertEquals( file, line,
                                                        xStr, yStr,
                                                        x, y );
    }

    void failedAssertSameData(const char *file, int line,
                              const char *xStr, const char *yStr, const char *sizeStr,
                              const void *x, const void *y, unsigned size)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertSameData( file, line,
                                                          xStr, yStr, sizeStr,
                                                          x, y, size );
    }

    void failedAssertDelta(const char *file, int line,
                           const char *xStr, const char *yStr, const char *dStr,
                           const char *x, const char *y, const char *d)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertDelta( file, line,
                                                       xStr, yStr, dStr,
                                                       x, y, d );
    }

    void failedAssertDiffers(const char *file, int line,
                             const char *xStr, const char *yStr, const char *value)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertDiffers(file, line,
                                                        xStr, yStr, value );
    }

    void failedAssertLessThan(const char *file, int line,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertLessThan(file, line,
                                                         xStr, yStr,
                                                         x, y );
    }

    void failedAssertLessThanEquals(const char *file, int line,
                                    const char *xStr, const char *yStr,
                                    const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line,
                                                                xStr, yStr,
                                                                x, y );
    }

    void failedAssertRelation(const char *file, int line, const char *relation,
                              const char *xStr, const char *yStr,
                              const char *x, const char *y)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation,
                                                          xStr, yStr,
                                                          x, y);
    }

    void failedAssertPredicate(const char *file, int line, const char *predicate,
                               const char *xStr, const char *x )
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate,
                                                           xStr, x);
    }

    void failedAssertThrows(const char *file, int line, const char *expression,
                            const char *type, bool otherThrown)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression,
                                                        type, otherThrown );
    }

    void failedAssertThrowsNot(const char *file, int line, const char *expression)
    { // Only record that the test fails
      test_passed=false;
      return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression );
    }

  private:
    bool test_passed;
};

int main()
{
#ifdef __COVERAGESCANNER__
  __coveragescanner_install(argv[0]);
#endif

  // Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()"
  return CoverageScannerListener().run();
}

boost::test

boost::test is a unit test framework for C++ which is part of the boost libraries. To adapt it to get the code coverage from each unit test:

  1. Implement a TestObserver, which derives from boost::unit_test_framework::test_observer and store as BoostTestObserver.hpp.
  2. Also implement (within the same file) a boost::test::fixture and define it to be executed at the beginning and end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file should look something like this:
            #ifndef COVERAGE_TEST_OBSERVER_INCLUDED
        #define COVERAGE_TEST_OBSERVER_INCLUDED

        #include <boost/test/framework.hpp>
        #include <boost/test/tree/observer.hpp>
        #include <boost/test/included/unit_test.hpp>

        #ifdef __COVERAGESCANNER__

        class BoostTestObserver
            : public boost::unit_test_framework::test_observer
        {
        public:
            BoostTestObserver(const char* moduleName)
                : test_observer()
                , m_testSuiteName(moduleName)
                , m_isAnyTestcaseOpen(false)
            {}

            ~BoostTestObserver()
            {
                if (m_isAnyTestcaseOpen)
                    close_current_testcase();
            }

        ///////////////////////////////////////////////////////////////////////////////
        // test suite related events:

            virtual int priority()
            {
                return 1;
            }

            virtual void test_finish()
            {
                if (m_isAnyTestcaseOpen) {
                    close_current_testcase();
                }
            }

            virtual void test_aborted()
            {
                if (m_isAnyTestcaseOpen) {
                    close_current_testcase();
                }
            }

            const char* get_testsuite_name()
            {
                return m_testSuiteName.data();
            }

            // called in the very beginning of the test suite:
            virtual void test_start(
                boost::unit_test_framework::counter_t  NumberOfTestCases)
            {
                m_currentTestNumber = -1;
                m_numberOfTestCases = NumberOfTestCases;
                __coveragescanner_install(m_testSuiteName.c_str());
            }

        ///////////////////////////////////////////////////////////////////////////////
        // test case related events:

            // called before each testcase's start
            virtual void test_unit_start(
                boost::unit_test_framework::test_unit const &unit)
            {
                // skip any calls which may being triggered by the 'TestWrap' fixture
                if(unit.full_name().find('/') == -1) {
                    return;
                }

    if (++m_currentTestNumber)
                    close_current_testcase();

    m_currentCaseHasFailed = false;
                __coveragescanner_clear();
                m_currentTestName.assign( unit.full_name().c_str());
                __coveragescanner_testname(m_currentTestName.c_str());
                m_isAnyTestcaseOpen = true;
            }

            // should be called after each testcase has finished...
            // but which definitveley, for unknown reason is NOT
            virtual void test_unit_finish(
                boost::unit_test_framework::test_unit const &unit,
                unsigned number)
            {
                close_current_testcase();
            }

// workaround for missing 'test_unit_finish' event being not triggered:
            void close_current_testcase()
            {
                if (m_isAnyTestcaseOpen) {
                    __coveragescanner_teststate(m_currentCaseHasFailed
                                               ? "FAILED"
                                               : "PASSED");
                    __coveragescanner_save();
                    m_currentTestName.append("_invalid");
                    m_isAnyTestcaseOpen = false;
                }
            }

            // called once on each FAILED or PASSED result
            virtual void assertion_result(boost::unit_test::assertion_result result)
            {
                if (result == boost::unit_test::AR_FAILED)
                    m_currentCaseHasFailed = true;
            }

        protected: // Deprecated boost event:
            virtual void assertion_result(bool result)
            {
                if (!result) m_currentCaseHasFailed = true;
            }

        ///////////////////////////////////////////////////////////////////////////////
        //  member variables:

            std::string m_testSuiteName;
            std::string m_currentTestName;
            int         m_currentTestNumber;
            int         m_numberOfTestCases;
            bool        m_verboseLogging;
            bool        m_currentCaseHasFailed;
            bool        m_isAnyTestcaseOpen;
        };

        //////////////////
        // boost fixture:
        // will be called in the very beginning and in the end of executing test
        ///////////////////////////////////////////////////////////////////////////////

        struct TestWrap
        {
            BoostTestObserver observer;

TestWrap()
            : BoostTestObserver(
                boost::unit_test::framework::master_test_suite().argv[0])
            {
                boost::unit_test::framework::register_observer(observer);
            }

           ~TestWrap()
            {
                observer.close_current_testcase();
                boost::unit_test::framework::deregister_observer(observer);
            }
        };

        ///////////////////////////////////////////////////////////////////////////////

        BOOST_GLOBAL_FIXTURE(TestWrap);

          #endif
        #endif
  3. If it is included by any of your already existing boost::test unit test .cpp files (must be done right after the mandatory BOOST_TEST_MODULE definition), it is being instantiated and initialized by boost::test automatically.
  4. Then, boost::test will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section being executed. Also, it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case's end.
  5. Additionally here is a small example which shows how to include this into your already existing boost::test modules:
    // define the name for yourBoost::Test module.
// (*this must be done BEFORE including BoostTestObserver.hpp)
#define BOOST_TEST_MODULE MyBoostTest
#include <boost/test/framework.hpp>
#include <boost/test/tree/observer.hpp>
#include <boost/test/included/unit_test.hpp>

// any of your already existing Boost::Test modules can be made
// compatible to Coco by including "BoostTestObserver.hpp"
// just after the module's BOOST_TEST_MODULE definition.
#include "BoostTestObserver.hpp"

// declare a boost test suite:
// (when "BoostTestObserver.hpp" is included, this will
// automatically instantiate a BoostTestObserver then.)
BOOST_AUTO_TEST_SUITE(BOOST_TEST_MODULE)

    // declare first test case:
    BOOST_AUTO_TEST_CASE(testA)
    {
        // code for testcase A
    }

     //...  declare any other testcases

    // last testcase:
    BOOST_AUTO_TEST_CASE(testX)
    {//...
    }

// in the end of the test, close the suite's scope
// (which at least also will unload our BoostTestObserver)
BOOST_AUTO_TEST_SUITE_END()
  6. Compile the boost test with CoverageScanner activated.

xUnit

When Coco is used with xUnit, it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test.

The tests themself need not be instrumented, but the code that is tested must be built with CoverageScanner. A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated.

Proceed as follows:

  1. Add to your xUnit test library the following code that provides a set of functions to handle the coverage information and to save the execution report (the .csexe file):
    using System;
using System.Collections.Generic;
using System.Text;
using Xunit;
using System.Reflection;
using Xunit.Sdk;
using System.Diagnostics;
using Xunit.Abstractions;

class Coverage
{
    public enum State { PASSED, FAILED, CHECK_MANUALLY, UNKNOWN };

    public static void CoverageCleanup()
    {
        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                // clear all coverage information to only get the code coverage of the current executed unit test
                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
            }
        }
    }

    public static void CoverageTextLog(String text, bool bold, bool italic)
    {
        String style_begin = "";
        String style_end = "";
        if (italic)
        {
            style_begin += "<I>";
            style_end += "</I>";
        }
        if (bold)
        {
            style_begin += "<B>";
            style_end += "</B>";
        }

        String comment = "<HTML><BODY><TT>"
                        + style_begin
                        + text.Replace("&", "&amp;")
                                .Replace("<", "&lt;")
                                .Replace(">", "&gt;")
                                .Replace("\"", "&quot;")
                                .Replace("'", "&apos;")
                                .Replace("\n", "<BR>")
                                .Replace("\r", "")
                        + style_end
                        + "</TT></BODY></HTML>"
                        ;
        CoverageHtmlLog(comment);
    }

    public static void CoverageHtmlLog(String comment)
    {
        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                coco.InvokeMember("__coveragescanner_add_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { comment });
            }
        }
    }

    public static void CoverageRecord(String csexe_filename, String testName, State result)
    {
        if (csexe_filename == "")
            return;

        AppDomain MyDomain = AppDomain.CurrentDomain;
        Assembly[] AssembliesLoaded = MyDomain.GetAssemblies();

        foreach (Assembly a in AssembliesLoaded)
        {
            Type coco = a.GetType("CoverageScanner");
            if (coco != null)
            {
                String name = testName;
                String state = "";

                switch (result)
                {
                    case State.FAILED:
                        state = ("FAILED");
                        break;
                    case State.CHECK_MANUALLY:
                        state = ("CHECK_MANUALLY");
                        break;
                    case State.PASSED:
                        state = ("PASSED");
                        break;
                }

                name = name.Replace('.', '/');

                // set the execution state: PASSES, FAILED or CHECK_MANUALLY
                coco.InvokeMember("__coveragescanner_teststate", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { state });

                if (name.Length > 0)
                    // Test name: Namespace/Class/Testfunction
                    coco.InvokeMember("__coveragescanner_testname", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { name });

                // File name is <classname>.csexe
                coco.InvokeMember("__coveragescanner_filename", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { csexe_filename });
                // saves the code coverage data
                coco.InvokeMember("__coveragescanner_save", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                // clear all coverage information to only get the code coverage of the current executed unit test
                coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
                coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null);
            }
        }
    }
}
  2. Add to your xUnit test library the following code that contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline:
    using System;
using System.Collections.Generic;
using System.Text;
using Xunit;
using System.Reflection;
using Xunit.Sdk;
using System.Diagnostics;
using Xunit.Abstractions;

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public class SquishCocoHooks : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo methodUnderTest)
    {
        Coverage.CoverageCleanup();
    }

    public override void After(MethodInfo methodUnderTest)
    {
        string base_name = "xUnit";
        string method_name = methodUnderTest.Name;
        string class_name = methodUnderTest.ReflectedType.FullName;

        Coverage.CoverageRecord("coverage.csexe", base_name + "/" + class_name + "/" + method_name, Coverage.State.UNKNOWN);
    }
}

public class CustomFixture : IDisposable
{
    public CustomFixture()
    {
        // Some initialization
    }

    public void Dispose()
    {
        // Some cleanup
    }
}

[CollectionDefinition("SquishCoco")]
[SquishCocoHooks]
public class SquishCocoDefinition :  ICollectionFixture<CustomFixture>
{
}
  3. Activate the collection for each test class by adding [Collection("SquishCoco")] to the definition of each test class. For example:
    [Collection("SquishCoco")]
public class Class2Tests
{
    [Fact()]
    public void dummyTest()
    {
        Assert.True(false, "Dummy test");
    }
}

After running the tests, a file coverage.csexe will be generated. It can then be imported into the corresponding coverage database (.csmes file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately.

NUnit

Coco provides an add-in for NUnit version 2.4.4 and above as sample. To install it:

  1. Build NUnitSquishCoco.dll using the Visual Studio project NUnitSquishCoco.vsproj provided in the sample directory.
  2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
  3. Start NUnit.exe and verify that the add-in NUnit \COCO is loaded.

Once installed, as soon as NUnit's test driver is executing a C# or C++ managed unit test test.dll, it generates a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application's instrumentation database with CoverageBrowser or cmcsexeimport.

Catch2

Catch2 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.

Complete code example:

*
    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
    #include <string>
    #include <deque>

    class CoverageScannerListener : public Catch::TestEventListenerBase
    {
    public:
        using TestEventListenerBase::TestEventListenerBase;

        virtual void sectionStarting( Catch::SectionInfo const& sectionInfo ) override
        {
            m_testNameHierarchy.push_back( sectionInfo.name );
    #ifdef __COVERAGESCANNER__
            // Adjusting the name of the test for the CoverageBrowser
            std::string testname = implodeTestNameHierarchy();
            // Reset the code coverage data to get only the code coverage
            // of the actual unit test.
            __coveragescanner_clear();
            __coveragescanner_testname( testname.c_str() );
    #endif
        }

        virtual void sectionEnded( Catch::SectionStats const& sectionStats ) override
        {
            m_testNameHierarchy.pop_back();
    #ifdef __COVERAGESCANNER__
            // Recording the execution state in the coverage report
            if ( sectionStats.assertions.allPassed() )
                __coveragescanner_teststate( "PASSED" );
            else
                __coveragescanner_teststate( "FAILED" );
            // Saving the code coverage report of the unit test
            __coveragescanner_save();
            __coveragescanner_testname( "" );
    #endif
        }

    private:
        std::deque<std::string> m_testNameHierarchy;

        /* Helper method which generates a hierarchical name string usable by the coveragebrowser. */
        std::string implodeTestNameHierarchy() {
            std::string fullTestName ( "Catch2" );
            std::deque<std::string>::const_iterator it = m_testNameHierarchy.cbegin();
            while ( it != m_testNameHierarchy.cend() ) {
                fullTestName += "/" + *it;
                it++;
            }
            return fullTestName;
        }
    };

    // Register the coveragescanner listener with Catch2
    CATCH_REGISTER_LISTENER( CoverageScannerListener )

    int main( int argc, char* argv[] )
    {
    #ifdef __COVERAGESCANNER__
        __coveragescanner_install( argv[0] );
    #endif
        return Catch::Session().run( argc, argv );
    }

To adapt Catch2:

  1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 provides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header:
    #define CATCH_CONFIG_RUNNER
    #include "catch.hpp"
  2. Create a Catch2 listener class inheriting Catch::TestEventListenerBase which records the code coverage of each section after it is completed. Note that testcases also count as sections in Catch2, so you do not need to listen to testcase events specifically.

    In the created listener class (CoverageScannerListener), use the following member functions:

    • sectionStarting(): This function is called before each testcase and section begins to compute a test name with the information provided by Catch2 and pass it to the Coco library with __coveragescanner_testname().

      We also call the function __coveragescanner_clear() to empty the internal database and so make sure that the coverage of the code that was executed before this test is ignored.

    • sectionEnded(): This function is called after a testcase or section has ended.

      It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  3. We add this listener by adding the following Catch2 macro:
    CATCH_REGISTER_LISTENER( CoverageScannerListener )

Note: Remember to exclude any test framework sources from instrumentation (see Beyond the minimal instrumentation).

Squish

Run the GUI testing tool Squish together with Coco to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Coco features the comparison of individual executions, including the calculation of an optimal order.

General approach

The approach depicted below is based on the possibility to apply information about the name, the result, and a free-form comment to each execution (stored in .csexe files). See Execution comment, name and status.

As an example we'll use Squish for Qt's addressbook on a Unix-based system and a JavaScript test script:

  1. First we'll initialize the execution data with the name of the Squish test case that is being run.
    function main()
{
    var currentAUT = currentApplicationContext();
    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
    var testCase = squishinfo.testCase;
    var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1);
    var file = File.open(execution, "a");
    file.write("*" + testExecutionName + "\n");
    file.close();
    var ctx = startApplication("addressbook");
...
  2. Insert the main test script at this point.
  3. After the main test script we'll log the result of the test for the coverage tool:
    ...
    // wait until AUT shutdown
    while (ctx.isRunning) {
        snooze(1); // increase time if not enough to dump coverage data
    }

    // test result summary and status
    var positive = test.resultCount("passes");
    var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals");
    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
    var status = negative == 0 ? "PASSED" : "FAILED";
    var file = File.open(execution, "a");
    file.write("<html><body>" + msg + "</body></html>\n");
    file.write("!" + status + "\n")
    file.close();
}

When you execute the scripts containing these steps, the Coco Execution Report loads with the test case name, status and execution summary in the execution details and comments.

Simplified for reuse

  1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
    function getExecutionPath() {
    var currentAUT = currentApplicationContext();
    var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
    return execution;
}

function logTestNameToCocoReport(currentTestCase, execution) {
    var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1);
    var file = File.open(execution, "a");
    file.write("*" + testExecutionName + "\n");
    file.close();
}

function logTestResultsToCocoReport(testInfo, execution){

    var currentAUT = currentApplicationContext();

    // wait until AUT shuts down
    while (currentAUT.isRunning)
      snooze(5);

    // collect test result summary and status
    var positive = testInfo.resultCount("passes");
    var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +
        testInfo.resultCount("fatals");
    var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
    var status = negative == 0 ? "PASSED" : "FAILED";

    // output results and status to Coco execution report file
    var file = File.open(execution, "a");
    file.write("<html><body>" + msg + "</body></html>\n");
    file.write("!" + status + "\n")
    file.close();
}

    A Python version of this code is:

    import re

def getExecutionPath():
    currentAUT = currentApplicationContext()
    execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name}
    return execution

def logTestNameToCocoReport(currentTestCase, execution):
    testExecutionName = re.search(r'[^\\]\w*$', currentTestCase)
    testExecutionName = testExecutionName.group(0)
    file = open(execution, "a")
    file.write("*" + testExecutionName + "\n")
    file.close()

def logTestResultsToCocoReport(testInfo, execution):
    currentAUT = currentApplicationContext()
    # wait until AUT shuts down
    while (currentAUT.isRunning):
        snooze(5)

    # collect test result summary and status
    positive = testInfo.resultCount("passes")
    negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals")
    msg = "TEST RESULTS - Passed: %(positive)s  |  Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative}
    if negative == 0:
        status = "PASSED"
    else:
        status = "FAILED"

    # output results and status to Coco execution report file
    file = open(execution, "a")
    file.write("<html><body>" + msg + "</body></html>\n")
    file.write("!" + status + "\n")
    file.close()
  2. Add the following function calls after startApplication() in the main test script:
    execution = getExecutionPath();

logTestNameToCocoReport(squishinfo.testCase, execution);

    In Python:

    execution = getExecutionPath()

logTestNameToCocoReport(squishinfo.testCase, execution)
  3. At the end of your script, after closing the AUT (for example, by clicking File > Exit), call the following function:
    logTestResultsToCocoReport(test, execution);

    In Python:

    logTestResultsToCocoReport(test, execution)
  4. If your AUT closes unexpectedly or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.

Your main test script should be similar to the following:

source(findFile("scripts","squishCocoLogging.JS"))

function main()
{
    startApplication("addressbook");
    execution = getExecutionPath();
    logTestNameToCocoReport(squishinfo.testCase, execution);

    try {
        // body of script
    }
    catch(e) {
       test.fail('An unexpected error occurred', e.message)
    }
    finally {
       logTestResultsToCocoReport(test, execution)
    }
}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():
    startApplication("addressbook")
    execution = getExecutionPath()
    logTestNameToCocoReport(squishinfo.testCase, execution)

    try:
        try:
            # body of script
        except Exception as e:
            test.fail("test failed: ", e)
    finally:
        logTestResultsToCocoReport(test,execution)

Support for IDEs and toolchains Installing Coco in Docker

Coco v7.0.0 ©2023 The Qt Company Ltd.
Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.