Support for Testing Frameworks
Introduction
When tests are run from a testing framework, it is possible to measure the coverage for each test separately. To do this, Coco must know when each test begins and ends, together with other information about the test. By adding some code to the tests, this is possible.
There are two ways to do it:
- Using the Coco library to write data to the
.csexe
file. - Writing to the
.csexe
file directly.
The next two chapters describe the use of these methods in the general case. In the chapters after that, concrete examples are given for several testing frameworks.
Use of the Coco library
Most test frameworks provide a method to run some code before each test and to run some other code after each test. They usually have also a way to get the name of the test. If this is supported by the test framework, it also supports measuring code coverage for each test separately and writing it separately to the .csexe file.
In general, the test framework will also provide a way to get the test result and possibly other information about the test. This information can also be written to the .csexe
file.
All this is done by calls to the Coco library. The library exists in two versions, one for C/C++ and one for C#, with the same functions but in different syntax. For the differences in the code see the examples below.
The following library call must occur before each test:
- Clearing the counters with
__coveragescanner_clear()
. At the beginning of a test, this function sets all the internal coverage counters of Coco to zero. This prevents that the measurements are polluted by data from earlier activity of the program.
The next library calls are used to set information that is later written to the .csexe
file by __coveragescanner_save()
. They can be run before or after the test and in any order.
- Setting the test name with
__coveragescanner_testname()
. In most frameworks, there is not just a name for each test but also one for the test suite to which it belongs. Coco supports this by allowing hierarchical names: strings with several parts that are separated by slashes. CoverageBrowser can display a list of hierarchical names as a tree.So when the name of the test suite is known, one can create names of the form
Suite/Test
and use them as parameters to__coveragescanner_testname()
.Calling this function is mandatory.
- Setting the test state with
__coveragescanner_teststate()
. This function is usually called after a test; it sets the test result that will be written to the {.csexe} file. But it can also be called before the test, e.g. to set the result to"SKIPPED"
Calling this function is not mandatory.
- Writing HTML annotations, if necessary. It is possible to write arbitrary other data in text form to the
.csexe
file. This is done with the functions__coveragescanner_clear_html_comment()
and__coveragescanner_add_html_comment()
.These functions use an internal variable to which
__coveragescanner_add_html_comment()
adds text. The text is stored persistently and therefore needs to be cleared before new information is added. So the way to create HTML comments is first to call__coveragescanner_clear_html_comment()
and then to call__coveragescanner_add_html_comment()
one or more times.Clear function description: C/C++, C#.
Add function description: C/C++, C#.
After the test, the data is witten and some cleanup is done with the following library calls. They must be run in exactly this order.
- Saving of the test data. With
__coveragescanner_save()
, the test name, the test result, the HTML annotations (if existing) and the content of all coverage counters is written to the.csexe
file. Afterwards, all counters are set to zero. - Reset the test name. When the test program finishes, it writes coverage data about its activity after the last test to the
.csmes
file. These data are still bound to the last test name used.So to prevent this possible error, one sets the name of the current test to the empty string, with
__coveragescanner_testname("")
.
Examples
This section is a short summary of the way the code must look like that must be inserted into the test framework code.
Since the new code refers to functions that only exist in instrumented code, they must be guarded by conditional compilation statements. There is a symbol __COVERAGESCANNER__
that only exists when the code is instrumented, and it can be used for that purpose.
General structure in C++
The following code is typically run before each test:
#ifdef __COVERAGESCANNER__ // Code to get TESTNAME __coveragescanner_clear(); __coveragescanner_testname(TESTNAME); #endif
And this code is run after every test:
#ifdef __COVERAGESCANNER__ // Code to get TESTRESULT and COMMENT __coveragescanner_teststate(TESTRESULT); __coveragescanner_clear_html_comment(); // if needed __coveragescanner_add_html_comment(COMMENT); // if needed __coveragescanner_save(); __coveragescanner_testname(""); #endif
For an concrete implementation see the CppUnit section. There, a concrete version of the first code snippet appears in the startTest()
member function, and the second snippet (without using the HTML comment functions) occurs in endTest()
.
General structure in C#
The code for C# is at this abstraction level very similar to that in C#. The only difference is that there is another syntax for conditional statements and that the Coco library functions have a prefix, CoverageScanner
.
Code that runs before every test:
#if __COVERAGESCANNER__ // Code to get TESTNAME CoverageScanner.__coveragescanner_clear(); CoverageScanner.__coveragescanner_testname(TESTNAME); #endif
Code that runs after every test:
#if __COVERAGESCANNER__ // Code to get TESTRESULT and COMMENT CoverageScanner.__coveragescanner_teststate(TESTRESULT); CoverageScanner.__coveragescanner_clear_html_comment(); // if needed CoverageScanner.__coveragescanner_add_html_comment(COMMENT); // if needed CoverageScanner.__coveragescanner_testname(""); CoverageScanner.__coveragescanner_save(); #endif
Writing data directly to the .csexe
file
The second method to specify the test which was executed is to write the information directly to the .csexe
file. This method is especially useful when each test is a separate program and the tests are run by a script.
The .csexe
file is a text file; the test information consists of additional lines of text that are appended to the file.
As before, some information must be added before a test and some afterwards. The following must be done before each test:
- To set the name of a test, a line of the form
*<name of the test>
must be appended to the
.csexe
file.The character
*
must be the first character of the line. Everything between it and the end of line is treated as the test name.Note: You can begin the line with a paragraph sign (
§
) instead of*
. The§
must be the in Latin-1 encoding and has the numerical value of 167. Since this may lead to internationalization problems, we strongly discourage its use. It is only supported for compatibility with older versions of the program.
And after the test, the following can be done:
- To set the status of a test, a line of the form
!<status>
must be appended to the
.csexe
file.The character
!
must be the first character of the line. Everything between it and the end of line is treated as the test status.The status can be one of the following strings:
PASSED
: the test was successfully executed.FAILED
: the test was not successfully passed.INCIDENT
: the test was not successfully executed (similar to failed).CHECK_MANUALLY
: it was not possible to determinate if the test was successfully executed.SKIPPED
: the test was skipped.
- To append an execution comment, insert the contents of an HTML document just after the execution of the application.
The full comment must follow the HTML syntax, but only the content of the
<body>
tag is used. A minimal comment might therefore be something like<html><body>My comment</body></html>
.There may be more than one HTML comment, and the comments and the status declaration may occur in any order, but they all must be appended to the
.csexe
file after the test has been executed.
Example
The following batch file executes the test First Test
and sets the status of the execution to CHECK_MANUALLY
.
echo *First Test >> myapp.csexe myapp echo "<HTML><BODY>Execution of myapp</BODY></HTML>" >> myapp.csexe echo !CHECK_MANUALLY >> myapp.csexe
CppUnit
CppUnit is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.
The following code is an example of how this can be done:
#include <cppunit/TestListener.h> #include <cppunit/BriefTestProgressListener.h> #include <cppunit/CompilerOutputter.h> #include <cppunit/extensions/TestFactoryRegistry.h> #include <cppunit/TestResult.h> #include <cppunit/TestResultCollector.h> #include <cppunit/TestRunner.h> class CoverageScannerListener : public CppUnit::TestListener { public: CoverageScannerListener() {} void startTest( CppUnit::Test *test ) { m_testFailed = false; #ifdef __COVERAGESCANNER__ int pos; // Adjusting the name of the test to display the tests // in a tree view in CoverageBrowser std::string testname = "CppUnit/" + test->getName(); while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos ) testname.replace( pos, 2, "/" ); // Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname( testname.c_str() ) ; #endif } void addFailure( const CppUnit::TestFailure &failure ) { m_testFailed = true; } void endTest( CppUnit::Test *test ) { #ifdef __COVERAGESCANNER__ // Recording the execution state in the coverage report if ( m_testFailed ) __coveragescanner_teststate( "FAILED" ); else __coveragescanner_teststate( "PASSED" ); // Saving the code coverage report of the unit test __coveragescanner_save(); __coveragescanner_testname( "" ); #endif } private: bool m_testFailed; // Prevents the use of the copy constructor and operator. CoverageScannerListener( const CoverageScannerListener © ); void operator =( const CoverageScannerListener © ); }; int main( int argc, char* argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_install( argv[0] ); #endif // Create the event manager and test controller CPPUNIT_NS::TestResult controller; // Add a listener that colllects test result CPPUNIT_NS::TestResultCollector result; controller.addListener( &result ); // Add a listener that print dots as test run. CPPUNIT_NS::BriefTestProgressListener progress; controller.addListener( &progress ); // Add a listener that saves the code coverage information CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener ); // Add the top suite to the test runner CPPUNIT_NS::TestRunner runner; runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() ); runner.run( controller ); return result.wasSuccessful() ? 0 : 1; }
In the example, we do the following:
- We write a CppUnit listener class which records the code coverage of each unit test after it is completed.
We want to be able to run the program with and without Coco. Therefore, we use the macro
__COVERAGESCANNER__
for conditional compilation. The macro is defined in every file that is instrumented by Coco, without the need to#include
anything.In the listener class,
CppUnitListener
, we use the following member functions:startTest()
: This function is called before each test begins.In it, we compute a test name with the information provided by CppUnit and pass it to the Coco library with
__coveragescanner_testname()
. This is the Execution Name in CoverageBrowser.We call the function
__coveragescanner_clear()
to empty the internal database and thus make sure that the coverage of the code that was executed before this test is ignored.addFailure()
: This function is called after a test fails. It just sets a flag that is used by the other functions.endTest()
: This function is called after a test has ended.It uses
__coveragescanner_teststate()
to record the execution status (PASSED
orFAILED
) and then saves the code coverage report itself with__coveragescanner_save()
.
- We call
__coveragescanner_install()
in themain()
function. - We add a listener to the test manager of CppUnit, the class
CPPUNIT_NS::TestResult
. In the example above, this is done by the following lines:CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener );
Qt Test
Qt Test is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test:
- Call
__coveragescanner_install()
in themain()
function. - Write a subclass of
QObject
, namedTestCoverageObject
. It must record the code coverage at the end of every unit test. - Instead of inheriting from
QObject
, let all your test cases inherit fromTestCoverageObject
. - The
TestCoverageObject
class provides its owninit()
andcleanup()
slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them toinitTest()
andcleanupTest()
. - Compile your project with code coverage enabled.
TestCoverageObject
header:
#ifndef _TEST_COVERAGE_OBJECT_H #define _TEST_COVERAGE_OBJECT_H #include <QObject> class TestCoverageObject : public QObject { Q_OBJECT public: virtual void initTest() {} virtual void cleanupTest() {} protected slots: void init() ; void cleanup(); }; #endif
TestCoverageObject
source:
#include "testcoverageobject.h" #include <QTest> #include <QMetaObject> #include <QString> void TestCoverageObject::init() { #ifdef __COVERAGESCANNER__ __coveragescanner_clear(); #endif initTest(); } void TestCoverageObject::cleanup() { cleanupTest(); #ifdef __COVERAGESCANNER__ QString test_name="unittest/"; test_name+=metaObject()->className(); test_name+="/"; test_name+=QTest::currentTestFunction(); __coveragescanner_testname(test_name.toLatin1()); if (QTest::currentTestFailed()) __coveragescanner_teststate("FAILED"); else __coveragescanner_teststate("PASSED") ; __coveragescanner_save(); __coveragescanner_testname(""); #endif }
GoogleTest
GoogleTest is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test:
- Call
__coveragescanner_install()
in themain()
function. - Write a
TestEventListener
class which records the code coverage report upon every unit test completion. The listener should set the name (using__coveragescanner_testname()
) and clear the instrumentation (using__coveragescanner_clear()
) before executing a test item (class memberstartTest()
) to ensure getting only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using__coveragescanner_teststate()
and__coveragescanner_save()
) in the class memberendTest()
. The classCodeCoverageListener
gives an implementation example. - Add this listener to the
Append
function of the GoogleTest listener (function::testing::UnitTest::GetInstance()->listeners().Append()
). - Compile the unit test using CoverageScanner.
#include <gtest/gtest.h> #include <stdlib.h> class CodeCoverageListener : public ::testing::TestEventListener { public: // Fired before any test activity starts. virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test) { } // Fired before each iteration of tests starts. There may be more than // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration // index, starting from 0. virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration) { } // Fired before environment set-up for each iteration of tests starts. virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test) { } // Fired after environment set-up for each iteration of tests ends. virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test) { } // Fired before the test case starts. virtual void OnTestCaseStart(const ::testing::TestCase& test_case) { } // Fired before the test starts. virtual void OnTestStart(const ::testing::TestInfo& test_info) { #ifdef __COVERAGESCANNER__ __coveragescanner_clear(); std::string test_name= std::string(test_info.test_case_name()) + '/' + std::string(test_info.name()); __coveragescanner_testname(test_name.c_str()); #endif } // Fired after a failed assertion or a SUCCESS(). virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result) { } // Fired after the test ends. virtual void OnTestEnd(const ::testing::TestInfo& test_info) { #ifdef __COVERAGESCANNER__ __coveragescanner_teststate("UNKNOWN"); if (test_info.result()) { if (test_info.result()->Passed()) __coveragescanner_teststate("PASSED"); if (test_info.result()->Failed()) __coveragescanner_teststate("FAILED"); } __coveragescanner_save(); #endif } // Fired after the test case ends. virtual void OnTestCaseEnd(const ::testing::TestCase& test_case) { } // Fired before environment tear-down for each iteration of tests starts. virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test) { } // Fired after environment tear-down for each iteration of tests ends. virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test) { } // Fired after each iteration of tests finishes. virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration) { } // Fired after all test activities have ended. virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test) { } } ; int main(int argc, char **argv){ //initialize CoverageScanner library #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif ::testing::FLAGS_gtest_output = "xml"; ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener); ::testing::InitGoogleTest(&argc, argv); //init google test framework return RUN_ALL_TESTS(); //run all tests which are listed in this project }
CxxTest
CxxTest is a unit test framework for C++. This environment can be easily adapted to get the code coverage from each unit test:
- Call
__coveragescanner_install()
in themain()
function. - Create a CxxTest
TestListener
classCoverageScannerListener
by subclassing an existing listener.In the example below this is
ErrorPrinter
. It will record the code coverage report upon every unit test completion. To ensure we get only the coverage data of the concerned test, the listener should set the name with__coveragescanner_testname()
and clear the instrumentation with__coveragescanner_clear()
before executing a test item (class memberenterTest()
).When a test item is executed, the instrumentation and the execution status should be saved in the member function
leaveTest()
with__coveragescanner_teststate()
and__coveragescanner_save()
.Finally, all test failure members of this class must be reimplemented to record the test failures.
- In the
main()
function call therun()
function ofCoverageScannerListener
instead ofCxxTest::ErrorPrinter().run()
. - Compile the unit test with CoverageScanner activated.
For example:
#include <cxxtest/TestRunner.h> #include <cxxtest/TestListener.h> #include <cxxtest/TestTracker.h> #include <cxxtest/ValueTraits.h> #include <cxxtest/ValueTraits.h> #include <cxxtest/ErrorPrinter.h> class CoverageScannerListener : public CxxTest::ErrorPrinter { public: CoverageScannerListener(std::ostream &o=std::cout, const char *preLine = ":", const char *postLine = "") : CxxTest::ErrorPrinter( o, preLine , postLine ) {} int run() { return CxxTest::ErrorPrinter::run(); } void enterTest( const CxxTest::TestDescription & desc) { test_passed=true; #ifdef __COVERAGESCANNER__ // Adjust the name of the test to display the tests // in a tree view in CoverageBrowser std::string testname="CxxTest/"; testname += desc.suiteName(); testname += "/"; testname += desc.testName(); // Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname(testname.c_str()); #endif return CxxTest::ErrorPrinter::enterTest( desc ); } void leaveTest( const CxxTest::TestDescription & desc) { #ifdef __COVERAGESCANNER__ // Record the execution state in the coverage report if (test_passed) __coveragescanner_teststate("PASSED"); else __coveragescanner_teststate("FAILED"); // Save the code coverage report of the unit test __coveragescanner_save(); #endif return CxxTest::ErrorPrinter::leaveTest( desc ); } void failedTest(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedTest( file, line, expression ); } void failedAssert(const char *file, int line, const char *expression) { // Only record that the test fails test_passed = false; return CxxTest::ErrorPrinter::failedAssert( file, line, expression ); } void failedAssertEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertEquals( file, line, xStr, yStr, x, y ); } void failedAssertSameData(const char *file, int line, const char *xStr, const char *yStr, const char *sizeStr, const void *x, const void *y, unsigned size) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size ); } void failedAssertDelta(const char *file, int line, const char *xStr, const char *yStr, const char *dStr, const char *x, const char *y, const char *d) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d ); } void failedAssertDiffers(const char *file, int line, const char *xStr, const char *yStr, const char *value) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDiffers(file, line, xStr, yStr, value ); } void failedAssertLessThan(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThan(file, line, xStr, yStr, x, y ); } void failedAssertLessThanEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line, xStr, yStr, x, y ); } void failedAssertRelation(const char *file, int line, const char *relation, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation, xStr, yStr, x, y); } void failedAssertPredicate(const char *file, int line, const char *predicate, const char *xStr, const char *x ) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate, xStr, x); } void failedAssertThrows(const char *file, int line, const char *expression, const char *type, bool otherThrown) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression, type, otherThrown ); } void failedAssertThrowsNot(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression ); } private: bool test_passed; }; int main() { #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif // Use "CoverageScannerListener().run()" instead of "CxxTest::ErrorPrinter().run()" return CoverageScannerListener().run(); }
boost::test
boost::test is a unit test framework for C++ which is part of the boost libraries. To adapt it to get the code coverage from each unit test:
- Implement a
TestObserver
, which derives fromboost::unit_test_framework::test_observer
and store asBoostTestObserver.hpp
. - Also implement (within the same file) a
boost::test::fixture
and define it to be executed at the beginning and end of each executedboost::test
by:BOOST_GLOBAL_FIXTURE(FixtureName)
. The file should look something like this:#ifndef COVERAGE_TEST_OBSERVER_INCLUDED #define COVERAGE_TEST_OBSERVER_INCLUDED #include <boost/test/framework.hpp> #include <boost/test/tree/observer.hpp> #include <boost/test/included/unit_test.hpp> #ifdef __COVERAGESCANNER__ class BoostTestObserver : public boost::unit_test_framework::test_observer { public: BoostTestObserver(const char* moduleName) : test_observer() , m_testSuiteName(moduleName) , m_isAnyTestcaseOpen(false) {} ~BoostTestObserver() { if (m_isAnyTestcaseOpen) close_current_testcase(); } /////////////////////////////////////////////////////////////////////////////// // test suite related events: virtual int priority() { return 1; } virtual void test_finish() { if (m_isAnyTestcaseOpen) { close_current_testcase(); } } virtual void test_aborted() { if (m_isAnyTestcaseOpen) { close_current_testcase(); } } const char* get_testsuite_name() { return m_testSuiteName.data(); } // called in the very beginning of the test suite: virtual void test_start( boost::unit_test_framework::counter_t NumberOfTestCases) { m_currentTestNumber = -1; m_numberOfTestCases = NumberOfTestCases; __coveragescanner_install(m_testSuiteName.c_str()); } /////////////////////////////////////////////////////////////////////////////// // test case related events: // called before each testcase's start virtual void test_unit_start( boost::unit_test_framework::test_unit const &unit) { // skip any calls which may being triggered by the 'TestWrap' fixture if(unit.full_name().find('/') == -1) { return; } if (++m_currentTestNumber) close_current_testcase(); m_currentCaseHasFailed = false; __coveragescanner_clear(); m_currentTestName.assign( unit.full_name().c_str()); __coveragescanner_testname(m_currentTestName.c_str()); m_isAnyTestcaseOpen = true; } // should be called after each testcase has finished... // but which definitveley, for unknown reason is NOT virtual void test_unit_finish( boost::unit_test_framework::test_unit const &unit, unsigned number) { close_current_testcase(); } // workaround for missing 'test_unit_finish' event being not triggered: void close_current_testcase() { if (m_isAnyTestcaseOpen) { __coveragescanner_teststate(m_currentCaseHasFailed ? "FAILED" : "PASSED"); __coveragescanner_save(); m_currentTestName.append("_invalid"); m_isAnyTestcaseOpen = false; } } // called once on each FAILED or PASSED result virtual void assertion_result(boost::unit_test::assertion_result result) { if (result == boost::unit_test::AR_FAILED) m_currentCaseHasFailed = true; } protected: // Deprecated boost event: virtual void assertion_result(bool result) { if (!result) m_currentCaseHasFailed = true; } /////////////////////////////////////////////////////////////////////////////// // member variables: std::string m_testSuiteName; std::string m_currentTestName; int m_currentTestNumber; int m_numberOfTestCases; bool m_verboseLogging; bool m_currentCaseHasFailed; bool m_isAnyTestcaseOpen; }; ////////////////// // boost fixture: // will be called in the very beginning and in the end of executing test /////////////////////////////////////////////////////////////////////////////// struct TestWrap { BoostTestObserver observer; TestWrap() : BoostTestObserver( boost::unit_test::framework::master_test_suite().argv[0]) { boost::unit_test::framework::register_observer(observer); } ~TestWrap() { observer.close_current_testcase(); boost::unit_test::framework::deregister_observer(observer); } }; /////////////////////////////////////////////////////////////////////////////// BOOST_GLOBAL_FIXTURE(TestWrap); #endif #endif
- If it is included by any of your already existing
boost::test
unit test.cpp
files (must be done right after the mandatoryBOOST_TEST_MODULE
definition), it is being instantiated and initialized byboost::test
automatically. - Then,
boost::test
will trigger a__coveragescanner_install()
call just in the beginning, and__coveragescanner_clear()
and__coveragescanner_testname()
calls for each test case section being executed. Also, it triggers__coveragescanner_teststate()
and__coveragescanner_save()
on each test case's end. - Additionally here is a small example which shows how to include this into your already existing
boost::test
modules:// define the name for yourBoost::Test module. // (*this must be done BEFORE including BoostTestObserver.hpp) #define BOOST_TEST_MODULE MyBoostTest #include <boost/test/framework.hpp> #include <boost/test/tree/observer.hpp> #include <boost/test/included/unit_test.hpp> // any of your already existing Boost::Test modules can be made // compatible to Coco by including "BoostTestObserver.hpp" // just after the module's BOOST_TEST_MODULE definition. #include "BoostTestObserver.hpp" // declare a boost test suite: // (when "BoostTestObserver.hpp" is included, this will // automatically instantiate a BoostTestObserver then.) BOOST_AUTO_TEST_SUITE(BOOST_TEST_MODULE) // declare first test case: BOOST_AUTO_TEST_CASE(testA) { // code for testcase A } //... declare any other testcases // last testcase: BOOST_AUTO_TEST_CASE(testX) {//... } // in the end of the test, close the suite's scope // (which at least also will unload our BoostTestObserver) BOOST_AUTO_TEST_SUITE_END()
- Compile the boost test with CoverageScanner activated.
xUnit
When Coco is used with xUnit, it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test.
The tests themself need not be instrumented, but the code that is tested must be built with CoverageScanner. A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated.
Proceed as follows:
- Add to your xUnit test library the following code that provides a set of functions to handle the coverage information and to save the execution report (the
.csexe
file):using System; using System.Collections.Generic; using System.Text; using Xunit; using System.Reflection; using Xunit.Sdk; using System.Diagnostics; using Xunit.Abstractions; class Coverage { public enum State { PASSED, FAILED, CHECK_MANUALLY, UNKNOWN }; public static void CoverageCleanup() { AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { // clear all coverage information to only get the code coverage of the current executed unit test coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null); coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null); } } } public static void CoverageTextLog(String text, bool bold, bool italic) { String style_begin = ""; String style_end = ""; if (italic) { style_begin += "<I>"; style_end += "</I>"; } if (bold) { style_begin += "<B>"; style_end += "</B>"; } String comment = "<HTML><BODY><TT>" + style_begin + text.Replace("&", "&") .Replace("<", "<") .Replace(">", ">") .Replace("\"", """) .Replace("'", "'") .Replace("\n", "<BR>") .Replace("\r", "") + style_end + "</TT></BODY></HTML>" ; CoverageHtmlLog(comment); } public static void CoverageHtmlLog(String comment) { AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { coco.InvokeMember("__coveragescanner_add_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { comment }); } } } public static void CoverageRecord(String csexe_filename, String testName, State result) { if (csexe_filename == "") return; AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { String name = testName; String state = ""; switch (result) { case State.FAILED: state = ("FAILED"); break; case State.CHECK_MANUALLY: state = ("CHECK_MANUALLY"); break; case State.PASSED: state = ("PASSED"); break; } name = name.Replace('.', '/'); // set the execution state: PASSES, FAILED or CHECK_MANUALLY coco.InvokeMember("__coveragescanner_teststate", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { state }); if (name.Length > 0) // Test name: Namespace/Class/Testfunction coco.InvokeMember("__coveragescanner_testname", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { name }); // File name is <classname>.csexe coco.InvokeMember("__coveragescanner_filename", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { csexe_filename }); // saves the code coverage data coco.InvokeMember("__coveragescanner_save", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null); // clear all coverage information to only get the code coverage of the current executed unit test coco.InvokeMember("__coveragescanner_clear", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null); coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, null); } } } }
- Add to your xUnit test library the following code that contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline:
using System; using System.Collections.Generic; using System.Text; using Xunit; using System.Reflection; using Xunit.Sdk; using System.Diagnostics; using Xunit.Abstractions; [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = false, Inherited = true)] public class SquishCocoHooks : BeforeAfterTestAttribute { public override void Before(MethodInfo methodUnderTest) { Coverage.CoverageCleanup(); } public override void After(MethodInfo methodUnderTest) { string base_name = "xUnit"; string method_name = methodUnderTest.Name; string class_name = methodUnderTest.ReflectedType.FullName; Coverage.CoverageRecord("coverage.csexe", base_name + "/" + class_name + "/" + method_name, Coverage.State.UNKNOWN); } } public class CustomFixture : IDisposable { public CustomFixture() { // Some initialization } public void Dispose() { // Some cleanup } } [CollectionDefinition("SquishCoco")] [SquishCocoHooks] public class SquishCocoDefinition : ICollectionFixture<CustomFixture> { }
- Activate the collection for each test class by adding
[Collection("SquishCoco")]
to the definition of each test class. For example:[Collection("SquishCoco")] public class Class2Tests { [Fact()] public void dummyTest() { Assert.True(false, "Dummy test"); } }
After running the tests, a file coverage.csexe
will be generated. It can then be imported into the corresponding coverage database (.csmes
file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately.
NUnit
Coco provides an add-in for NUnit version 2.4.4 and above as sample. To install it:
- Build
NUnitSquishCoco.dll
using the Visual Studio projectNUnitSquishCoco.vsproj
provided in the sample directory. - Copy
NUnitSquishCoco.dll
to theaddins
folder located in thebin
folder whereNUnit.exe
can be found. - Start
NUnit.exe
and verify that the add-inNUnit \COCO
is loaded.
Once installed, as soon as NUnit's test driver is executing a C# or C++ managed unit test test.dll
, it generates a code coverage execution report test.dll.csexe
automatically if test.dll
is instrumented with Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application's instrumentation database with CoverageBrowser or cmcsexeimport.
Catch2
Catch2 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.
Complete code example:
* #define CATCH_CONFIG_RUNNER #include "catch.hpp" #include <string> #include <deque> class CoverageScannerListener : public Catch::TestEventListenerBase { public: using TestEventListenerBase::TestEventListenerBase; virtual void sectionStarting( Catch::SectionInfo const& sectionInfo ) override { m_testNameHierarchy.push_back( sectionInfo.name ); #ifdef __COVERAGESCANNER__ // Adjusting the name of the test for the CoverageBrowser std::string testname = implodeTestNameHierarchy(); // Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname( testname.c_str() ); #endif } virtual void sectionEnded( Catch::SectionStats const& sectionStats ) override { m_testNameHierarchy.pop_back(); #ifdef __COVERAGESCANNER__ // Recording the execution state in the coverage report if ( sectionStats.assertions.allPassed() ) __coveragescanner_teststate( "PASSED" ); else __coveragescanner_teststate( "FAILED" ); // Saving the code coverage report of the unit test __coveragescanner_save(); __coveragescanner_testname( "" ); #endif } private: std::deque<std::string> m_testNameHierarchy; /* Helper method which generates a hierarchical name string usable by the coveragebrowser. */ std::string implodeTestNameHierarchy() { std::string fullTestName ( "Catch2" ); std::deque<std::string>::const_iterator it = m_testNameHierarchy.cbegin(); while ( it != m_testNameHierarchy.cend() ) { fullTestName += "/" + *it; it++; } return fullTestName; } }; // Register the coveragescanner listener with Catch2 CATCH_REGISTER_LISTENER( CoverageScannerListener ) int main( int argc, char* argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_install( argv[0] ); #endif return Catch::Session().run( argc, argv ); }
To adapt Catch2:
- Call
__coveragescanner_install()
andCatch::Session.run()
in themain()
function. Catch2 provides a predefinedmain()
, which needs to be disregarded by defining onlyCATCH_CONFIG_RUNNER
before including the Catch2 header:#define CATCH_CONFIG_RUNNER #include "catch.hpp"
- Create a Catch2 listener class inheriting
Catch::TestEventListenerBase
which records the code coverage of each section after it is completed. Note that testcases also count as sections in Catch2, so you do not need to listen to testcase events specifically.In the created listener class (
CoverageScannerListener
), use the following member functions:sectionStarting()
: This function is called before each testcase and section begins to compute a test name with the information provided by Catch2 and pass it to the Coco library with__coveragescanner_testname()
.We also call the function
__coveragescanner_clear()
to empty the internal database and so make sure that the coverage of the code that was executed before this test is ignored.sectionEnded()
: This function is called after a testcase or section has ended.It uses
__coveragescanner_teststate()
to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with__coveragescanner_save()
.
- We add this listener by adding the following Catch2 macro:
CATCH_REGISTER_LISTENER( CoverageScannerListener )
Note: Remember to exclude any test framework sources from instrumentation (see Beyond the minimal instrumentation).
Squish
Run the GUI testing tool Squish together with Coco to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Coco features the comparison of individual executions, including the calculation of an optimal order.
General approach
The approach depicted below is based on the possibility to apply information about the name, the result, and a free-form comment to each execution (stored in .csexe
files). See Test suites and Coco.
As an example we'll use Squish for Qt's addressbook
on a Unix-based system and a JavaScript test script:
- First we'll initialize the execution data with the name of the Squish test case that is being run.
function main() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" var testCase = squishinfo.testCase; var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1); var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); var ctx = startApplication("addressbook"); ...
- Insert the main test script at this point.
- After the main test script we'll log the result of the test for the coverage tool:
... // wait until AUT shutdown while (ctx.isRunning) { snooze(1); // increase time if not enough to dump coverage data } // test result summary and status var positive = test.resultCount("passes"); var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED"; var file = File.open(execution, "a"); file.write("<html><body>" + msg + "</body></html>\n"); file.write("!" + status + "\n") file.close(); }
When you execute the scripts containing these steps, the Coco Execution Report loads with the test case name, status and execution summary in the execution details and comments.
Simplified for reuse
- Create a file called
squishCocoLogging.js
inTest Suite Resources
with the following functions:function getExecutionPath() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" return execution; } function logTestNameToCocoReport(currentTestCase, execution) { var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1); var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); } function logTestResultsToCocoReport(testInfo, execution){ var currentAUT = currentApplicationContext(); // wait until AUT shuts down while (currentAUT.isRunning) snooze(5); // collect test result summary and status var positive = testInfo.resultCount("passes"); var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED"; // output results and status to Coco execution report file var file = File.open(execution, "a"); file.write("<html><body>" + msg + "</body></html>\n"); file.write("!" + status + "\n") file.close(); }
A Python version of this code is:
import re def getExecutionPath(): currentAUT = currentApplicationContext() execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name} return execution def logTestNameToCocoReport(currentTestCase, execution): testExecutionName = re.search(r'[^\\]\w*$', currentTestCase) testExecutionName = testExecutionName.group(0) file = open(execution, "a") file.write("*" + testExecutionName + "\n") file.close() def logTestResultsToCocoReport(testInfo, execution): currentAUT = currentApplicationContext() # wait until AUT shuts down while (currentAUT.isRunning): snooze(5) # collect test result summary and status positive = testInfo.resultCount("passes") negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals") msg = "TEST RESULTS - Passed: %(positive)s | Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative} if negative == 0: status = "PASSED" else: status = "FAILED" # output results and status to Coco execution report file file = open(execution, "a") file.write("<html><body>" + msg + "</body></html>\n") file.write("!" + status + "\n") file.close()
- Add the following function calls after
startApplication()
in the main test script:execution = getExecutionPath(); logTestNameToCocoReport(squishinfo.testCase, execution);
In Python:
execution = getExecutionPath() logTestNameToCocoReport(squishinfo.testCase, execution)
- At the end of your script, after closing the AUT (for example, by clicking File > Exit), call the following function:
logTestResultsToCocoReport(test, execution);
In Python:
logTestResultsToCocoReport(test, execution)
- If your AUT closes unexpectedly or a script error occurs, incorporating a
try
,catch
,finally
ensures your results still output to the Coco report file.
Your main test script should be similar to the following:
source(findFile("scripts","squishCocoLogging.JS")) function main() { startApplication("addressbook"); execution = getExecutionPath(); logTestNameToCocoReport(squishinfo.testCase, execution); try { // body of script } catch(e) { test.fail('An unexpected error occurred', e.message) } finally { logTestResultsToCocoReport(test, execution) } }
Python version:
source(findFile("scripts","squishCocoLogging.py")) def main(): startApplication("addressbook") execution = getExecutionPath() logTestNameToCocoReport(squishinfo.testCase, execution) try: try: # body of script except Exception as e: test.fail("test failed: ", e) finally: logTestResultsToCocoReport(test,execution)
Coco v7.2.0 ©2024 The Qt Company Ltd.
Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property
of their respective owners.