6.4.9. Analysis Rule Testing¶

6.4.9.1. Source Test File Format¶

Tests can be specified as ordinary C/C++ source files with the extensions .cpp or .c. The files are annotated with special comments to control the test setup, test execution and to specify the expected results. These special comments contain directives followed by a colon or slash.

In general a test is setup by describing some prerequisites, compilation of the test case into an IR (setup phase), running tools to test (e.g. the analysis tool to test a specific rule), and specifying the expected results as Python code or patterns to be matched against the tool’s output.

The following example demonstrates several of the key features:

// language: C++14
// platform: windows or linux
/* test:
   analysis(['Group-Name'], excludes=['/opt/third-party-library/*'])
 */

class SomeClass;
//^ stdout:4: required: First description (Rule Group-Name)

#ifdef __APPLE__
class SomeOtherClass;
//^ stdout(regex=True):4: required: (First|Second) description \(Rule Group\-Name\)
#endif

int some_function(); // label: od
int some_function(int i);
//^ stdout:4: required: Second description (Rule Group-Name), $(od)

namespace some_namespace {
// note: this is a comment and will be ignored
}

/* setup:
   def my_function():
     return False
 */

// test: assert my_function()

If directives should apply to several test files, the corresponding comments can be put in a file called default-test-config.h next to the individual test cases or one directory above. These directives are interpreted as if they appeared at the beginning of every test file.

Declarative Comments¶

The following directives are supported. We describe them grouped by their purpose. The first group of directives is used to describe prerequisites for the test to be run:

// language:: Give a minimal language version that is needed for this test to run (e.g. C++14). If the --language switch was specified on the command line, the test is skipped if the command line specified an older language version than this comment. If the command-line does not specify any language version, then this comment has the additional effect of adding -std=... to the compiler invocation. If the user specified a language version on the command line, the user is responsible for configuring the compiler to use that language version (e.g. via CAFECC_OPTIONS environment variable).
// max_language:: Give a maximum language version where this test should run. The test is skipped if a newer language version is given.
// platform:: A boolean expression (using Python syntax) of platforms where this test can run. If given, the test will only execute when the platform condition is satisfied, and will be skipped otherwise. See also: the perform_tests --platforms command-line argument.
// xfail: reason or // xfail(condition): reason: A reason why this test is expected to fail. In the second form, the test is only expected to fail on platforms that match the provided condition. E.g., xfail(not windows) means that the test is expected to fail on Windows.
// axivion_version: 0.1.2.3: The version of Axivion Suite that can verify this test file
// todo: comment or // note: comment: Prevents processing “comment” as a directive.
*.json / *.xml / *.py: Everything given after these directives is pasted verbatim into the files with the respective name in a temporary configuration directory. This directory is automatically inserted in BAUHAUS_CONFIG as the most-local configuration layer. This means a file called axivion_config.json will be automatically picked up by the axivion_analysis tool.
// enable_all_pp_sections: True: If set to True, patterns and test directives within all blocks guarded by preprocessor directives are evaluated. If omitted or set to False, only patterns and test directives from active guarded preprocessor blocks are evaluated.
// allow_duplicate_messages: True: If set to True, tests may produce the same output multiple times. If omitted or set to False, the test will fail if any output is encountered more than once.

Setup Phase¶

To setup a test case, the following directives can be used. You can either execute arbitrary Python code or setup special files.

// setup: python_code()

Arbitrary Python code to be executed before the actual tests run.

Python variables and functions defined in the setup directive will be visible in subsequent setup directives, and in all test directives within the same file.

Python code in the setup phase usually does one of the following:

mutate the BAUHAUS_CONFIG variable (of type list[str]) in order to add additional configuration layers

mutate the CAFECC_OPTIONS variable (of type list[str]) in order to pass additional options to influence the automatic compilation of the test case at the end up the setup phase.

explicitly call predefined functions cafeCC() and/or irlink() in order to compile the test case. Explicit compilation allows building test cases involving multiple translation units. An explicit call to cafeCC() will disable the implicit compilation at the end up the setup phase.

Note that you can use multi-line comments /* setup: ... */ in order to execute more complex Python code that does not fit on a single line.

// error #123: description

By default, the test runner will verify that source test cases compile without errors. If invalid code is used intentionally, it should be marked with an error directive. This will cause the test runner to expect an error with the number 123 in the line associated with the comment. You can use a question mark “// error? #123: description” to allow a compiler error without requiring it to be present. This is useful when the error depends on the compiler mode.

The error comment supports the ^ syntax for associating the error with a previous line instead of the current line (see documentation of // test for more details).

To ignore all occurrences of a specific error, you can use:

#pragma diag_suppress 123

Test Phase¶

If correctly setup, expected results and test commands can be specified with the following commands.

// test: python_code()

Give some arbitrary Python code to be executed. Use assert to test a specific condition. Python variables and functions defined in the test directive will be visible in subsequent test directives within the same file.

Within the test phase, the Python variable ir_graph can be used to access the IR graph that was compiled by the setup phase.

The Python variable node is set to the outermost physical IR node that has its source position in the same line as associated with the comment. The test comment supports the ^ syntax for associating the comment with a previous line instead of the current line. By using multiple ^^, the comment can be associated with code multiple lines above.

int a;
//^  test: assert node.Name == 'a'
//^^ test: assert node.is_of_type('Global_Variable_Definition')

This test will succeed: both comments are associated with the line declaring the variable, so node refers to the variable definition IR node.

Hint

You can use // test: breakpoint() to enter an interactive debugging session during the test run, where you can inspect node and ir_graph.

// label: name

Mark the corresponding line with a label. These are simple strings that are available in the stdout / stderr directive via special substitution syntax or via the global variable labels in the test directive. The label comment supports the ^ syntax for associating the label with a previous line instead of the current line.

// stdout: expected_output or // stderr: expected_output

Specify a pattern to be matched against the tool’s output. The output is expected to start with filename:line: followed by the provided expected_output. Any whitespace immediately following stdout: / stderr: is ignored and not considered part of the expected output. Similar to the // test: comments, carets (^) can be used to associate the comment with a previous line instead of the current line.

Within the expected output, the syntax $(some_label) can be used to refer to labels. Labels will be substituted with filename:line from the location associated with the label definition.

Example with multiple violations associated with the same line:

for (int x = 0; x < 10 && some_effect(); x++)
//^   stdout:26: required: Right operand of '&&' should be essentially Boolean. [bool && int32_t] (Rule MisraC2012-10.1)
//^^  stdout:47: required: Call in loop condition potentially causes side-effect [some_effect()] (Rule MisraC2012-14.2), $(some_label):5: Write global_var
//^^^ stdout:52: warning: line should not end with whitespace (Rule CodingStyle-NoTrailingWhitespace)
{
}

It is also possible to programmatically add expected output lines:

// test: TESTCASE.add_expected_stdout('abc')
// test: TESTCASE.add_expected_stderr('def')

When the provided string contains several lines, these are added separately; common leading indentation is removed, empty lines are ignored and the order of the lines is not significant. Each line is expected as-is, without the automatic prefix of the current line and without expansion of labels.

// stdout(regex=True): expected_output_regex or // stderr(regex=True): expected_output_regex

When the expected output is not known exactly (e.g., due to implementation-defined values or types), it is possible to use regular expressions by adding the regex=True qualifier:

//^ stdout(regex=True):1: required: Number of macro definitions in this unit is higher than translation limit \(limit: 1024, actual value: \d+\) \(Rule MisraC\-1\.1\)

As with the non-regex stdout directive, the output is expected to start with filename:line:. Note that special characters have to be escaped for parts of the expected output that should be matched literally.

An alternative syntax of the form // stdout/expected_output_regex/ is still supported but deprecated.

// stdout(wildcard="..."): expected_output or // stderr(wildcard="..."): expected_output

For the same reasons outlined for regex=True above it can be useful to ignore parts of a message. This can be accomplished by adding a wildcard="PLACEHOLDER" qualifier. Any number of characters (or the empty string) will be matched wherever PLACEHOLDER appears in the expected output:

//^ stdout(wildcard="..."):5: required: Forbidden characters used. [...] (Rule AutosarC++17_03-A2.2.1)

As with the other stdout directives, the output is expected to start with filename:line:.

Source tests involving multiple files¶

For more advanced usages, the following directives can be used. This is a scenario where you have one main test file and several additional ones. This is for example useful to test the interaction of several source files.

// primary_file: otherfile.cpp

With this directive, you mark the current file as a secondary one. This file will not be used as a test case, but the execution will be delegated to the file given as a relative path.

// include: header.h

By default, perform_tests will only interpret comments in the primary file as test directives. The include comment causes the test runner to also process comments from the named file.

Typically, a test case that uses C #include "header.h" will additionally use // include: header.h so that the header can contribute labels and expected output.

A test case that compiles multiple .cpp units will use // include: file2.cpp so that all units can use test comments.

Here is an example for a test case spanning multiple files:

unit1.cpp:

// setup: cafeCC(['unit1.cpp', 'unit2.cpp'])
// note: Process testrunner comments in other files:
// include: header.h
// include: unit2.cpp
// test: analysis()

#include "header.h"
void file1() {}

unit2.cpp:

// primary_file: unit1.cpp

#include "header.h"
void file2() {}

header.h:

// primary_file: unit1.cpp

void file1();
void file2();

Predefined variables¶

TESTCASE.ir_filename: str:

The file name where perform_tests expects the IR at the end of the setup phase.

BAUHAUS_CONFIG: list[str]:

Controls the BAUHAUS_CONFIG environment variable for any subprocesses started. The initial value depends on the environment in which perform_tests was started. The test setup can then insert additional configuration layers.

CAFECC_OPTIONS: list[str]:

Additional options passed to the cafeCC compiler. Can be used to pass options such as -std=c++17. The initial value of this variable can be controlled via the CAFECC_OPTIONS environment variable when starting perform_tests.

ENV: dict[str, str]:

The environment variables that are used when perform_tests starts subprocesses.

Note that by default, perform_tests adds these variables to the environment:

TEST_SOURCE_DIR: the directory that contains the primary file of the test case
TEST_EXEC_DIR: a temporary directory for the test execution.

Predefined functions¶

In // setup and // test comments, the following functions are available:

def run(args: list, capture_stdout=True, capture_stderr=False, **kwargs)

Run a subprocess. The test fails if the process returns with an exit code != 0. If stdout/stderr are captured, the output of the process is added to the “actual output” that will be compared with the output expected by // stdout comments.

The additional arguments (kwargs) are forwarded to the subprocess.run call.

Note that by default, the subprocess will run with the working directory set to a temporary directory (TEST_EXEC_DIR). Use the environment variable TEST_SOURCE_DIR if the command needs to access the test source files.

def cafeCC(args: list)

Explicitly run cafeCC with the specified command-line options.

If an argument is a relative path to a file existing relative to the test source directory, perform_tests will convert it to an absolute path before passing it to the compiler. All other relative paths are interpreted relative to the TEST_EXEC_DIR temporary directory.
```
// setup: cafeCC(['unit1.cpp', '-shared', '-o', 'first.dll'])
```
In this cafeCC invocation, unit1.cpp is read from the TEST_SOURCE_DIR, but first.dll is stored in the TEST_EXEC_DIR.
If the output filename is not explicitly specified with -o, the IR will be written to TESTCASE.ir_filename.
The -B option is automatically passed to the compiler.
Any options listed in the global CAFECC_OPTIONS environment variable will be passed to the compiler.

An explicit cafeCC() invocation will disable the implicit compilation step at the end of the setup phase.

def irlink(args: list, ir_file=None)

Run the linker with the given command-line arguments. The ir_file parameter specifies the output file name. If ir_file is not given, the output IR is written to TESTCASE.ir_filename.

def analysis(rules=None, *, ...)

Run axivion_analysis.

Parameters accepted by this function correspond to the axivion_analysis command-line options:

rules: Select rules to execute. If this parameter is omitted, all rules active in the configuration will be executed.

excludes: list[str]: Corresponds to --exclude

includes: list[str]: Corresponds to --include

brief: bool = True: Corresponds to --brief (single-line-output)

system: bool = True: Corresponds to --system

Additionally, a parameter config is supported with which you can name a configuration file to add to BAUHAUS_CONFIG. This is useful when the test not only executes a certain check, but also requires to configure that check in a configuration file embedded into the test case. For example:

/* rule_config.py:
import axivion.config
analysis = axivion.config.get_analysis()
analysis['MisraC2012-22.9'].errno_readers  = ['perror', 'check_errno']
*/
/* test: analysis(['MisraC2012-22.9', 'StaticSemanticAnalysis'],
                  config='rule_config.py')
*/

6.4.9.2. The Test Driver perform_tests¶

The tool perform_tests can be used to execute Source Test Cases.

perform_tests [-h] [-d] [-v] [--no_summary]
    [--language LANGUAGE] tst_path [tst_path ...]

The tool is a self-describing command-line tool, but for a quick overview we have provided some of its most important options here:

Option	Meaning
`-v`	Emit verbose output helpful for diagnosing tests.
`-d`	Emit lots of information helpful when debugging tests.
`--language`	Only run tests compatible with the specified language version.
`--platforms`	Specify target platform features (influences which tests will run).
`--no_summary`	Don’t output summary at end of test run.
`--keep_tempdir`	Don’t clean up the tmp directory at end of test run.
`tst_path..`	The tests to be executed one after the other.

The tool can execute one test or multiple tests in a row. In order to get a first impression you might want to execute the following command, assuming that you are interested in the Misra-C:2012 tests and are using C99, i.e. no C11 in your project:

perform_tests --language C99 misra/c2012/misrac2012-7.*

Most notably, the --language switch will avoid any tests from being run, that require any C11 features which, if you haven’t enabled C11 in your project configuration, will most likely fail for reasons not very easy to spot. If you omit the --language option, all tests will be executed which will be counterproductive, especially if for example, you aren’t using C++ at all. You can also just specify --language C or --language C++ in order to run all C or all C++ tests, which is more important for the HIS and the errorchecks directories as they contain tests for both languages at once.

Note that specifying the --language switch will disable a feature of perform_tests where it automatically adds the test’s expected language version to the compiler invocation. Instead you are expected to configure the compiler yourself to use a language version matching the --language switch. You can do this by setting the CAFECC_OPTIONS environment variable.

In addition to the language filter, perform_tests also has a target platform filter: tests can declare that they require a specific target platform feature, e.g. posix or stdc_atomic. Such tests will only run when the target platform feature is enabled in the --platforms switch. For example perform_tests --platforms linux,gnu,stdc_atomic will run tests requiring one of these three platform identifiers, but will skip tests requiring one of the other platform identifiers (e.g. posix tests will not run). If the --platforms command-line switch is not specified, the set of target platforms will be auto-detected from the compiler configuration. The set of available platforms and the set of active-by-default platforms is displayed in the perform_tests --help output.

As a summary, perform_tests reports the number of tests that failed (i.e., the test output was not as specified in expected output), the number of tests that run successfully (i. e., the test output matched the expected output), and tests with erroneous execution or failed setup. Some tests may also be not applicable on a certain platform, these are reported in addition.

The output of the example use of perform_tests to run the tests for Misra-C:2012, rules 7.1 to 7.4, might look as follows:

perform_tests --language C99 misra/c2012/misrac2012-7.*
Test "MisraC2012-7.1" passed
Test "MisraC2012-7.2" passed
Test "MisraC2012-7.3" passed
Test "MisraC2012-7.4" passed
--------------------------------------------------------------------------------
Test Summary:
--------------------------------------------------------------------------------
Count of Tests passed:.................. 4
Count of Tests failed:.................. 0
Count of Tests with execution problems:. 0
Count of Tests with failed setup:....... 0
Count of n/a tests:..................... 1
--------------------------------------------------------------------------------

For the sake of simplicity, the outputs from the tests are not printed by default and only the test results are printed.

When developing tests, the command line switches -v and -d (in particular, when used in combination) can be used for checking intermediate results.

6.4.9.3. The Qualification Kit Executor axivion_qkit¶

The Axivion Qualification Kits (see Section Axivion Qualification Kits User Guide) can be executed using a dedicated tool axivion_qkit.

axivion_qkit [--dry_run] [--in_process] [--coverage COVERAGE] [--pytest_all] [<further-pytest-options>] SOURCE_TEST_FILE_OR_DIR [SOURCE_TEST_FILE_OR_DIR ...]

axivion_qkit’s primary purpose is to execute all given source test cases, similarly to perform_tests. But in addition to only executing the given test cases, it performs additional checks, especially when instructed to perform a coverage check for a whole rule group (see --coverage).

For every rule in the given rule group, check if it is enabled in the current configuration.
For every rule in the given rule group, check if at least one test case passed (i.e., not skipped).
If at least one of the test cases of the given rule group reqires a semantic analysis engine, check if a semantic analysis engine is enabledi n the current configuration.

axivion_qkit also offers to generate a qualification report as PDF file (see --report_directory) that lists all of the primary test case results grouped by the analysis rule that was (or would have been, in case of skipped test cases) tested as well as the result of the aforementioned coverage results.

Note

It is implemented as part of a pytest plugin and the tool axivion_qkit itself is only a thin wrapper that calls pytest with the appropriate command line arguments to load and execute the plugin. Therefore, other orthogonal pytest features like selecting test cases by pattern using -k or by marker using -m are supported as well as parallel test execution using pytest-xdist’s -n. axivion_qkit passes all command line options that it does not understand as-is to pytest.

Note

As a side-effect of passing all options to pytest, the option --help does not only cause pytest’s own help message to be printed but also the complete output of pytest --help.

This is a quick overview of the most important axivion_qkit-specific options. For other options we refer to the help message, the perform_tests documentation, and the pytest documentation:

Option	Meaning
`--coverage`	Create a coverage report for given rule group (multiple comma-separated values possible)
`--report_directory`	Directory to write coverage report PDF and related files to.
`--report_overwrite`	If not set, tool will abort early if coverage report exists.
`--pytest_all`	If set, all pytest test cases collected will be executed instead of only source testcases.
`SOURCE_TEST_FILE_OR_DIR`	File or directory name to collect test cases from.

Axivion Suite 7.12.0

Navigation