6.4.9. Analysis Rule Testing

6.4.9.1. Source Test File Format

Tests can be specified as ordinary source files with the extensions .c, .cpp, .cu (CUDA), .cs (C#), or .rs (Rust). The files are annotated with special comments to control the test setup, test execution, and to specify the expected results. These special comments contain directives followed by a colon or slash. For the specifics of C# and Rust tests, see C# and Rust Test Files.

In general, a test is set up by describing prerequisites, compiling the test case into an IR (setup phase), running the tools being tested (e.g., the analysis tool to test a specific rule), and specifying the expected results as Python code or patterns to be matched against the tool’s output.

The following example demonstrates several of the key features:

// language: C++14
// platform: windows or linux
/* test:
   analysis(['Group-Name'], excludes=['/opt/third-party-library/*'])
 */

class SomeClass;
//^ stdout:4: required: First description (Rule Group-Name)

#ifdef __APPLE__
class SomeOtherClass;
//^ stdout(regex=True):4: required: (First|Second) description \(Rule Group\-Name\)
#endif

int some_function(); // label: od
int some_function(int i);
//^ stdout:4: required: Second description (Rule Group-Name), $(od)

namespace some_namespace {
// note: this is a comment and will be ignored
}

/* setup:
   def my_function():
     return False
 */

// test: assert my_function()

If directives should apply to several test files, the corresponding comments can be placed in a file called default-test-config.h next to the individual test cases or one directory above. These directives are interpreted as if they appeared at the beginning of every test file.

Declarative Comments

The following directives are supported, grouped by their purpose. The first group of directives describes the prerequisites for running the test:

// language:

Specify the minimum language version required for the test (e.g., C++14). If the --language switch is specified on the command line, the test is skipped if the specified version is older than the version in this comment. If the command line does not specify a language version, this comment adds -std=... to the compiler invocation. If a language version is specified on the command line, the user is responsible for configuring the compiler to use that version (e.g., via the CAFECC_OPTIONS environment variable).

// max_language:

Specify the maximum language version for which this test should run. The test is skipped if a newer language version is given.

// platform:

A boolean expression (using Python syntax) specifying the platforms where this test can run. If provided, the test executes only when the platform condition is satisfied; otherwise, it is skipped. See also: the perform_tests --platforms command-line argument.

// xfail: reason or
// xfail(condition): reason

The reason why this test is expected to fail. In the second form, the test is expected to fail only on platforms that match the provided condition. For example, xfail(not windows) means that the test is expected to fail on all platforms except Windows. perform_tests ignores this directive. It takes effect when the tests run via pytest, e.g., with axivion_test or axivion_qkit, where a test that passes despite an xfail is counted as a failure.

// skip: reason

Skips the test for the given reason. Like xfail, this directive only takes effect when the tests run via pytest. perform_tests ignores it.

// axivion_version: 0.1.2.3

The Axivion Suite version that can verify this test file. bauhaus_version is an alias. If the installed version differs, perform_tests aborts and a test run via pytest fails, unless --no_version_check is specified.

// isolated: True

If set to True, the test runs in a separate process.

// markers: name ...

A space-separated list of pytest markers to add to the test (e.g., for selecting tests using the pytest -m option).

// todo: comment or // note: comment

Prevents the “comment” from being processed as a directive.

*.json / *.xml / *.py

All content following these directives is pasted verbatim into files of the respective names within a temporary configuration directory. This directory is automatically inserted into BAUHAUS_CONFIG as the most local configuration layer. Consequently, a file named axivion_config.json will be automatically picked up by the axivion_analysis tool.

// enable_all_pp_sections: True

If set to True, patterns and test directives within all blocks guarded by preprocessor directives are evaluated. If omitted or set to False, only patterns and test directives from active preprocessor blocks are evaluated.

// allow_duplicate_messages: True

If set to True, tests may produce the same output multiple times. If omitted or set to False, the test will fail if any output is encountered more than once.

// parameter: name = [value, ...]

Declares a test parameter, expanding the test file into multiple test cases: the file is run once for each value, with the Python variable name bound to that value. This is useful when the same source code should be tested in several configurations.

Setup Phase

To set up a test case, the following directives can be used. You can either execute arbitrary Python code or set up special files.

// setup: python_code()

Arbitrary Python code to be executed before the tests run.

Python variables and functions defined in a setup directive are visible in subsequent setup directives and in all test directives within the same file.

Python code in the setup phase usually does one of the following:

  • Mutate the BAUHAUS_CONFIG variable (of type list[str]) to add additional configuration layers.

  • Mutate the CAFECC_OPTIONS variable (of type list[str]) to pass additional options that influence the automatic compilation of the test case at the end of the setup phase.

  • Explicitly call the predefined functions cafeCC() and/or irlink() to compile the test case. Explicit compilation allows building test cases involving multiple translation units. An explicit call to cafeCC() disables the implicit compilation at the end of the setup phase.

Multi-line comments /* setup: ... */ can be used to execute more complex Python code that does not fit on a single line.

// error #123: description

By default, the test runner verifies that source test cases compile without errors. If invalid code is used intentionally, it should be marked with an error directive. This causes the test runner to expect an error with the number 123 on the line associated with the comment. You can use a question mark (// error? #123: description) to allow a compiler error without requiring it to be present. This is useful when the error depends on the compiler mode.

The error comment supports the ^ syntax for associating the error with a previous line instead of the current line (see documentation of // test for more details).

To ignore all occurrences of a specific error, you can use:

#pragma diag_suppress 123

C# and Rust Test Files

A .cs test file is compiled with csharp2rfg instead of cafeCC. Each compiler diagnostic, including warnings, must be declared using a comment of the form // error CS0103: description or // warning CS1030: description, which supports the ^ syntax. A column can be added after the diagnostic number, as in // error CS0103:9: description.

A .rs test file is analyzed as the only target of a generated Cargo.toml. The following directives control the manifest and the analysis:

// edition: 2021

The Rust edition (2015, 2018, 2021 or 2024). Defaults to 2024.

// feature: name = [...]

Adds the given line to the [features] section of the generated manifest.

// manifest: path

Uses the given Cargo.toml, relative to the directory of the test file, instead of generating one.

// rustlib: discover

Discovers the Rust sysroot and loads the sources of the Rust standard library. Without this directive, they are not loaded.

A Cargo.toml can also be given inline with a cargo.toml: directive, like the configuration files described above. It cannot be combined with the edition, feature and manifest directives.

Test Phase

Once correctly set up, expected results and test actions can be specified with the following directives.

// test: python_code()

Provide arbitrary Python code to be executed. Use assert to test a specific condition. Python variables and functions defined in the test directive will be visible in subsequent test directives within the same file.

Within the test phase, the Python variable ir_graph can be used to access the IR graph that was compiled by the setup phase.

The Python variable node refers to the outermost physical IR node whose source position is on the same line associated with the comment. The test comment supports the ^ syntax for associating the comment with a previous line instead of the current line. By using multiple ^^, the comment can be associated with code multiple lines above.

int a;
//^  test: assert node.Name == 'a'
//^^ test: assert isinstance(node, ir.Physical.Global_Variable_Definition)

This test will succeed: both comments are associated with the line declaring the variable, so node refers to the variable definition IR node.

Hint

You can use // test: breakpoint() to enter an interactive debugging session during the test run, where you can inspect node and ir_graph.

// label: name

Mark the corresponding line with a label. These are simple strings that are available in the stdout / stderr directive via special substitution syntax or via the global variable labels in the test directive. The label comment supports the ^ syntax for associating the label with a previous line instead of the current line.

// stdout: expected_output or
// stderr: expected_output

Specify a pattern to be matched against the tool’s output. The output is expected to start with filename:line: followed by the provided expected_output. Any whitespace immediately following stdout: / stderr: is ignored and not considered part of the expected output. Similar to the // test: comments, carets (^) can be used to associate the comment with a previous line instead of the current line.

Within the expected output, the syntax $(some_label) can be used to refer to labels. Labels will be substituted with filename:line from the location associated with the label definition. Similarly, $(^), $(^^), etc. refer to the line the comment is associated with. The number of carets must match the comment’s own (//^ → $(^), //^^ → $(^^)). To refer to any other line, define a label.

Example with three violations associated with the same line, one of them referring to the declaration of f through the label f-def:

extern int f(int i, int j); // label: f-def

int i, j = 0;
for (i = 0; i < 10 && f(i, j); i++)
//^   stdout:24: required: Right operand of '&&' should be essentially Boolean. [bool && int32_t] (Rule MisraC2012-10.1)
//^^  stdout:24: advisory: Parentheses should be used to avoid dependence on precedence rules [left operand of &&] (Rule MisraC2012-12.1)
//^^^ stdout:28: required: Call in loop condition potentially causes side-effect [f()] (Rule MisraC2012-14.2), $(f-def):12: Unknown effect [f()]
{
}

Example with two violations associated with the same line, each referring to that line and to two other lines:

char* p = malloc(4); // label: alloc_p
char* q = malloc(4); // label: alloc_q
free(p); // label: free_p
free(q); // label: free_q
strcpy(p, q);
//^  stdout:11: error: Dynamic memory possibly used after it was previously released [heap memory allocated at $(alloc_p):21] (Rule FaultDetection-UseAfterFree), $(alloc_p):21: calling malloc, $(free_p):9: calling free, $(^):12: calling and passing as argument [p]
//^^ stdout:11: error: Dynamic memory possibly used after it was previously released [heap memory allocated at $(alloc_q):21] (Rule FaultDetection-UseAfterFree), $(alloc_q):21: calling malloc, $(free_q):9: calling free, $(^^):15: calling and passing as argument [q]

Here, both $(^) and $(^^) refer to the strcpy(p, q); line: the first comment sits one line below it and uses one caret (//^ … $(^)), while the second sits two lines below and uses two (//^^ … $(^^)).

It is also possible to programmatically add expected output lines:

// test: TESTCASE.add_expected_stdout('abc')
// test: TESTCASE.add_expected_stderr('def')

When the provided string contains several lines, these are added separately; common leading indentation is removed, empty lines are ignored and the order of the lines is not significant. Each line is expected as-is, without the automatic prefix of the current line and without expansion of labels.

// stdout(regex=True): expected_output_regex or
// stderr(regex=True): expected_output_regex

When the expected output is not known exactly (e.g., due to implementation-defined values or types), it is possible to use regular expressions by adding the regex=True qualifier:

//^ stdout(regex=True):1: required: Number of macro definitions in this unit is higher than translation limit \(limit: 1024, actual value: \d+\) \(Rule MisraC\-1\.1\)

As with the non-regex stdout directive, the output is expected to start with filename:line:. Note that special characters must be escaped for parts of the expected output that should be matched literally.

An alternative syntax of the form // stdout/expected_output_regex/ is still supported but deprecated.

// stdout(wildcard="..."): expected_output or
// stderr(wildcard="..."): expected_output

For the same reasons as regex=True, it may be useful to ignore parts of a message. This can be accomplished by adding a wildcard="PLACEHOLDER" qualifier. Any number of characters (or the empty string) will be matched wherever PLACEHOLDER appears in the expected output:

//^ stdout(wildcard="..."):5: required: Forbidden characters used. [...] (Rule AutosarC++17_03-A2.2.1)

As with the other stdout directives, the output is expected to start with filename:line:.

Source tests involving multiple files

For more advanced usage, the following directives can be used. This is useful for scenarios where you have one main test file and several additional ones, for example, to test the interaction of several source files.

// primary_file: otherfile.cpp

With this directive, you mark the current file as a secondary one. This file will not be treated as a test case; instead, execution will be delegated to the file specified by the relative path.

// include: header.h

By default, perform_tests will only interpret comments in the primary file as test directives. The include comment causes the test runner to also process comments from the named file.

Typically, a test case that uses C #include "header.h" will additionally use // include: header.h so that the header can contribute labels and expected output.

A test case that compiles multiple .cpp units will use // include: file2.cpp so that all units can use test comments.

Here is an example of a test case spanning multiple files:

unit1.cpp:

// setup: cafeCC(['unit1.cpp', 'unit2.cpp'])
// note: Process testrunner comments in other files:
// include: header.h
// include: unit2.cpp
// test: analysis()

#include "header.h"
void file1() {}

unit2.cpp:

// primary_file: unit1.cpp

#include "header.h"
void file2() {}

header.h:

// primary_file: unit1.cpp

void file1();
void file2();

Predefined variables

TESTCASE.ir_filename: str:

The filename where perform_tests expects the IR at the end of the setup phase.

BAUHAUS_CONFIG: list[str]:

Controls the BAUHAUS_CONFIG environment variable for any subprocesses started. The initial value depends on the environment in which perform_tests was started. The test setup can then insert additional configuration layers.

CAFECC_OPTIONS: list[str]:

Additional options passed to the cafeCC compiler, such as -std=c++17. The initial value of this variable can be controlled via the CAFECC_OPTIONS environment variable when starting perform_tests.

ENV: dict[str, str]:

The environment variables used when perform_tests starts subprocesses.

Note that by default, perform_tests adds these variables to the environment:
  • TEST_SOURCE_DIR: the directory containing the primary file of the test case

  • TEST_EXEC_DIR: a temporary directory for the test execution.

Predefined functions

In // setup and // test comments, the following functions are available:

def run(args: list, capture_stdout=True, capture_stderr=False, **kwargs)

Run a subprocess. The test fails if the process returns with an exit code != 0. If stdout/stderr are captured, the process output is added to the “actual output,” which is compared with the output expected by // stdout comments.

The additional arguments (kwargs) are forwarded to the subprocess.run call.

Note that by default, the subprocess will run with the working directory set to a temporary directory (TEST_EXEC_DIR). Use the environment variable TEST_SOURCE_DIR if the command needs to access the test source files.

def cafeCC(args: list)

Explicitly run cafeCC with the specified command-line options.

  • If an argument is a relative path to a file that exists relative to the test source directory, perform_tests converts it to an absolute path before passing it to the compiler. All other relative paths are interpreted relative to the TEST_EXEC_DIR temporary directory.

    // setup: cafeCC(['unit1.cpp', '-shared', '-o', 'first.dll'])
    

    In this cafeCC invocation, unit1.cpp is read from the TEST_SOURCE_DIR, but first.dll is stored in the TEST_EXEC_DIR.

  • If the output filename is not explicitly specified with -o, the IR will be written to TESTCASE.ir_filename.

  • The -B option is automatically passed to the compiler.

  • Any options listed in the global CAFECC_OPTIONS environment variable will be passed to the compiler.

An explicit cafeCC() invocation will disable the implicit compilation step at the end of the setup phase.

def irlink(args: list, ir_file=None)

Run the linker with the given command-line arguments. The ir_file parameter specifies the output filename. If ir_file is not given, the output IR is written to TESTCASE.ir_filename.

def analysis(rules=None, *, ...)

Run axivion_analysis.

Parameters accepted by this function correspond to the axivion_analysis command-line options:

rules: Select rules to execute. If this parameter is omitted, all rules active in the configuration will be executed.

excludes: list[str]: Corresponds to --exclude

includes: list[str]: Corresponds to --include

brief: bool = True: Corresponds to --brief (single-line-output)

system: bool = True: Corresponds to --system

Additionally, a config parameter is supported, allowing you to specify a configuration file to add to BAUHAUS_CONFIG. This is useful when a test not only executes a certain check but also requires configuring that check via a configuration file embedded in the test case. For example:

/* rule_config.py:
import axivion.config
analysis = axivion.config.get_analysis()
analysis['MisraC2012-22.9'].errno_readers  = ['perror', 'check_errno']
*/
/* test: analysis(['MisraC2012-22.9', 'StaticSemanticAnalysis'],
                  config='rule_config.py')
*/

6.4.9.2. The Test Driver perform_tests

The tool perform_tests can be used to execute Source Test Cases.

perform_tests [-h] [-d] [-v] [--no_summary] [--dry_run] [--in_process]
    [--language LANGUAGE] [--tempdir TEMPDIR] [--keep_tempdir]
    [--no_version_check] [--platforms PLATFORMS] [--no_platform_check]
    [--exitfirst] [--maxfail MAXFAIL]
    file_or_dir [file_or_dir ...]

The tool is self-describing, but its options are provided here for a quick overview:

Option

Meaning

-v

Emit verbose output helpful for diagnosing tests.

-d

Emit detailed information helpful for debugging tests.

--no_summary

Do not output the summary at the end of the test run.

--dry_run

Do not run tests; only count and list the tests that match the language and platform filters.

--in_process

Execute the tests in the same process, e.g., to attach a debugger. This is unsafe when running several tests.

--language

Only run tests compatible with the specified language version.

--tempdir

Use the given directory instead of an automatic temporary one.

--keep_tempdir

Do not clean up the temporary directory at the end of the test run.

--no_version_check

Do not check whether the axivion_version of a test matches the installed version.

--platforms

Specify target platform features (influences which tests will run).

--no_platform_check

Run tests even if they are not applicable to the target platform. --no_prerequisite_check is an alias.

--exitfirst

Stop after the first failed test. -x is an alias.

--maxfail

Stop after the given number of failed tests.

file_or_dir..

The test files to execute sequentially. Directories are searched recursively for test files, and glob patterns are expanded.

The tool can execute a single test or multiple tests sequentially. To get a first impression, you may want to execute the following command, assuming that you are interested in the Misra-C:2012 tests and are using C99 (i.e., no C11) in your project:

perform_tests --language C99 misra/c2012/misrac2012-7.*

Notably, the --language switch prevents tests that require C11 features from running. This is useful if C11 is not enabled in your project configuration, as such tests would likely fail for reasons that are difficult to diagnose. Omitting the --language option will cause all tests to be executed, which may be counterproductive – especially if you are not using C++ at all. You can also specify --language C or --language C++ to run all tests for a specific language. This is particularly useful for the HIS and errorchecks directories, as they contain tests for both languages.

Note that specifying a specific language version with the --language switch (e.g., C99, rather than just C or C++) disables the feature where perform_tests automatically adds the test’s expected language version to the compiler invocation. Instead, you must configure the compiler to use a language version that matches the --language switch. You can do this by setting the CAFECC_OPTIONS environment variable.

In addition to the language filter, perform_tests includes a target platform filter; tests can declare a requirement for a specific target platform feature, such as posix or stdc_atomic. These tests only run when the corresponding target platform feature is enabled via the --platforms switch. For example, perform_tests --platforms linux,gnu,stdc_atomic will run tests requiring any of these three platform identifiers, but will skip tests requiring others (e.g., posix tests will not run). If the --platforms switch is not specified, the target platforms are auto-detected from the compiler configuration. The available and default-active platforms are displayed in the perform_tests --help output.

In summary, perform_tests reports the number of tests that failed (i.e., the output did not match the expected output), the number of tests that passed (i.e., the output matched the expected output), and tests with execution errors or failed setup. Tests that are not applicable to a specific platform are also reported.

The output from the example of using perform_tests to run the Misra-C:2012 tests for rules 7.1 to 7.4 may look like this:

perform_tests --language C99 misra/c2012/misrac2012-7.*
Test "MisraC2012-7.1" passed
Test "MisraC2012-7.2" passed
Test "MisraC2012-7.3" passed
Test "MisraC2012-7.4" passed
--------------------------------------------------------------------------------
Test Summary:
--------------------------------------------------------------------------------
Count of Tests passed:.................. 4
Count of Tests failed:.................. 0
Count of Tests with execution problems:. 0
Count of Tests with failed setup:....... 0
Count of n/a tests:..................... 1
--------------------------------------------------------------------------------

For simplicity, test outputs are not printed by default; only the test results are displayed.

When developing tests, the -v and -d switches (especially when used together) can be used to check intermediate results.

6.4.9.3. The pytest-Based Test Driver axivion_test

The axivion_test tool executes source test cases using pytest.

axivion_test [--dry_run] [--in_process] [--pytest_all] [<further-pytest-options>]
             SOURCE_TEST_FILE_OR_DIR [SOURCE_TEST_FILE_OR_DIR ...]

Unlike perform_tests, it respects the xfail and skip directives. It accepts the following perform_tests options: --language, --platforms, --no_platform_check, --tempdir, --keep_tempdir, and --no_version_check, and passes all other unrecognized options to pytest. For example, the following command runs all tests in a directory in parallel using the -n option from pytest-xdist:

axivion_test -n auto misra/c2012/

Option

Meaning

--dry_run

Only collect the tests, alias for pytest --collect-only.

--in_process

Disable pytest-xdist and execute the tests in the same process, e.g., to attach a debugger.

--pytest_all

Execute all collected pytest tests instead of only source test cases.

6.4.9.4. The Qualification Kit Executor axivion_qkit

The Axivion Qualification Kits (see Section Axivion Qualification Kits User Guide) can be executed using the dedicated axivion_qkit tool.

axivion_qkit [--dry_run] [--in_process] [--coverage COVERAGE] [--report_kind {generic,do330}]
             [--report_directory REPORT_DIRECTORY] [--report_overwrite] [--add_rulegroup_coverage_to_junit_xml]
             [--pytest_all] [<further-pytest-options>] SOURCE_TEST_FILE_OR_DIR [SOURCE_TEST_FILE_OR_DIR ...]

The primary purpose of axivion_qkit is to execute all specified source test cases, similar to perform_tests. In addition to executing test cases, it performs additional checks, particularly when instructed to perform a coverage check for an entire rule group (see --coverage).

  • Check if every rule in the given rule group is enabled in the current configuration.

  • Check if at least one test case passed (i.e., was not skipped) for every rule in the given rule group.

  • If any test case in the given rule group requires a semantic analysis engine, it checks if that engine is enabled in the current configuration.

axivion_qkit can also generate a qualification report as a PDF file (see --report_directory). This report lists all primary test case results, grouped by the analysis rule that was tested (or would have been tested, in the case of skipped test cases), along with the results of the coverage checks mentioned above.

Note

It is implemented as a pytest plugin; the axivion_qkit tool is a thin wrapper that calls pytest with the appropriate command-line arguments to load and execute the plugin. Consequently, other pytest features – such as selecting test cases by pattern using -k, by marker using -m, or parallel execution using the pytest-xdist -n option – are also supported. axivion_qkit passes all unrecognized command-line options directly to pytest.

Note

As a side effect of passing all options to pytest, the --help option prints both axivion_qkit’s own help message and the complete output of pytest --help.

This is a brief overview of the most important axivion_qkit-specific options. For further details, refer to the help message, the perform_tests documentation, and the pytest documentation:

Option

Meaning

--coverage

Create a coverage report for the specified rule group (multiple comma-separated values are possible).

--report_kind

The type of PDF report to create: generic (default) or do330. The do330 option additionally creates a report on the execution environment and configuration to be referenced by the DO-330 Tool Configuration Index, and requires --report_directory.

--report_directory

Directory in which to save the coverage report PDF and related files.

--report_overwrite

If not set, the tool will abort early if the coverage report already exists.

--add_rulegroup_coverage_to_junit_xml

If set, the results of the rule group coverage check are added as test cases to the JUnit XML file generated by pytest’s --junitxml.

--pytest_all

If set, all collected pytest test cases will be executed instead of only the source test cases.

SOURCE_TEST_FILE_OR_DIR

The file or directory name from which to collect test cases.