6.4.9. Analysis Rule Testing¶
6.4.9.1. Source Test File Format¶
Tests can be specified as ordinary source files with the extensions .c,
.cpp, .cu (CUDA), .cs (C#), or .rs
(Rust). The files are annotated with special comments to control the test setup,
test execution, and to specify the expected results. These special comments contain
directives followed by a colon or slash. For the specifics of C# and Rust tests, see
C# and Rust Test Files.
In general, a test is set up by describing prerequisites,
compiling the test case into an IR (setup phase),
running the tools being tested (e.g., the analysis tool to test a specific rule), and specifying
the expected results as Python code or patterns to be matched against the tool’s output.
The following example demonstrates several of the key features:
// language: C++14
// platform: windows or linux
/* test:
analysis(['Group-Name'], excludes=['/opt/third-party-library/*'])
*/
class SomeClass;
//^ stdout:4: required: First description (Rule Group-Name)
#ifdef __APPLE__
class SomeOtherClass;
//^ stdout(regex=True):4: required: (First|Second) description \(Rule Group\-Name\)
#endif
int some_function(); // label: od
int some_function(int i);
//^ stdout:4: required: Second description (Rule Group-Name), $(od)
namespace some_namespace {
// note: this is a comment and will be ignored
}
/* setup:
def my_function():
return False
*/
// test: assert my_function()
If directives should apply to several test files, the corresponding comments
can be placed in a file called default-test-config.h next to the individual test cases
or one directory above.
These directives are interpreted as if they appeared at the beginning of every test file.
Declarative Comments¶
The following directives are supported, grouped by their purpose. The first group of directives describes the prerequisites for running the test:
// language:Specify the minimum language version required for the test (e.g., C++14). If the
--languageswitch is specified on the command line, the test is skipped if the specified version is older than the version in this comment. If the command line does not specify a language version, this comment adds-std=...to the compiler invocation. If a language version is specified on the command line, the user is responsible for configuring the compiler to use that version (e.g., via theCAFECC_OPTIONSenvironment variable).// max_language:Specify the maximum language version for which this test should run. The test is skipped if a newer language version is given.
// platform:A boolean expression (using Python syntax) specifying the platforms where this test can run. If provided, the test executes only when the platform condition is satisfied; otherwise, it is skipped. See also: the
perform_tests --platformscommand-line argument.// xfail: reasonor
// xfail(condition): reasonThe reason why this test is expected to fail. In the second form, the test is expected to fail only on platforms that match the provided condition. For example,
xfail(not windows)means that the test is expected to fail on all platforms except Windows.perform_testsignores this directive. It takes effect when the tests run viapytest, e.g., withaxivion_testoraxivion_qkit, where a test that passes despite anxfailis counted as a failure.// skip: reasonSkips the test for the given reason. Like
xfail, this directive only takes effect when the tests run viapytest.perform_testsignores it.// axivion_version: 0.1.2.3The Axivion Suite version that can verify this test file.
bauhaus_versionis an alias. If the installed version differs,perform_testsaborts and a test run viapytestfails, unless--no_version_checkis specified.// isolated: TrueIf set to
True, the test runs in a separate process.// markers: name ...A space-separated list of
pytestmarkers to add to the test (e.g., for selecting tests using thepytest-moption).// todo: commentor// note: commentPrevents the “comment” from being processed as a directive.
*.json/*.xml/*.pyAll content following these directives is pasted verbatim into files of the respective names within a temporary configuration directory. This directory is automatically inserted into
BAUHAUS_CONFIGas the most local configuration layer. Consequently, a file namedaxivion_config.jsonwill be automatically picked up by theaxivion_analysistool.// enable_all_pp_sections: TrueIf set to
True, patterns andtestdirectives within all blocks guarded by preprocessor directives are evaluated. If omitted or set toFalse, only patterns andtestdirectives from active preprocessor blocks are evaluated.// allow_duplicate_messages: TrueIf set to
True, tests may produce the same output multiple times. If omitted or set toFalse, the test will fail if any output is encountered more than once.// parameter: name = [value, ...]Declares a test parameter, expanding the test file into multiple test cases: the file is run once for each value, with the Python variable
namebound to that value. This is useful when the same source code should be tested in several configurations.
Setup Phase¶
To set up a test case, the following directives can be used. You can either execute arbitrary Python code or set up special files.
// setup: python_code()Arbitrary Python code to be executed before the tests run.
Python variables and functions defined in a
setupdirective are visible in subsequentsetupdirectives and in alltestdirectives within the same file.Python code in the setup phase usually does one of the following:
Mutate the
BAUHAUS_CONFIGvariable (of typelist[str]) to add additional configuration layers.Mutate the
CAFECC_OPTIONSvariable (of typelist[str]) to pass additional options that influence the automatic compilation of the test case at the end of the setup phase.Explicitly call the predefined functions
cafeCC()and/orirlink()to compile the test case. Explicit compilation allows building test cases involving multiple translation units. An explicit call tocafeCC()disables the implicit compilation at the end of the setup phase.
Multi-line comments
/* setup: ... */can be used to execute more complex Python code that does not fit on a single line.// error #123: descriptionBy default, the test runner verifies that source test cases compile without errors. If invalid code is used intentionally, it should be marked with an
errordirective. This causes the test runner to expect an error with the number 123 on the line associated with the comment. You can use a question mark (// error? #123: description) to allow a compiler error without requiring it to be present. This is useful when the error depends on the compiler mode.The
errorcomment supports the^syntax for associating the error with a previous line instead of the current line (see documentation of// testfor more details).To ignore all occurrences of a specific error, you can use:
#pragma diag_suppress 123
C# and Rust Test Files¶
A .cs test file is compiled with csharp2rfg instead of cafeCC.
Each compiler diagnostic, including warnings, must be declared using a comment of the
form // error CS0103: description or // warning CS1030: description, which
supports the ^ syntax. A column can be added after the diagnostic number, as in
// error CS0103:9: description.
A .rs test file is analyzed as the only target of a generated Cargo.toml.
The following directives control the manifest and the analysis:
// edition: 2021The Rust edition (
2015,2018,2021or2024). Defaults to2024.// feature: name = [...]Adds the given line to the
[features]section of the generated manifest.// manifest: pathUses the given
Cargo.toml, relative to the directory of the test file, instead of generating one.// rustlib: discoverDiscovers the Rust sysroot and loads the sources of the Rust standard library. Without this directive, they are not loaded.
A Cargo.toml can also be given inline with a cargo.toml: directive, like the
configuration files described above. It cannot be combined with the edition,
feature and manifest directives.
Test Phase¶
Once correctly set up, expected results and test actions can be specified with the following directives.
// test: python_code()Provide arbitrary Python code to be executed. Use
assertto test a specific condition. Python variables and functions defined in thetestdirective will be visible in subsequenttestdirectives within the same file.Within the test phase, the Python variable
ir_graphcan be used to access the IR graph that was compiled by the setup phase.The Python variable
noderefers to the outermost physical IR node whose source position is on the same line associated with the comment. Thetestcomment supports the^syntax for associating the comment with a previous line instead of the current line. By using multiple^^, the comment can be associated with code multiple lines above.int a; //^ test: assert node.Name == 'a' //^^ test: assert isinstance(node, ir.Physical.Global_Variable_Definition)
This test will succeed: both comments are associated with the line declaring the variable, so
noderefers to the variable definition IR node.Hint
You can use
// test: breakpoint()to enter an interactive debugging session during the test run, where you can inspectnodeandir_graph.// label: nameMark the corresponding line with a label. These are simple strings that are available in the
stdout/stderrdirective via special substitution syntax or via the global variablelabelsin thetestdirective. Thelabelcomment supports the^syntax for associating the label with a previous line instead of the current line.// stdout: expected_outputor
// stderr: expected_outputSpecify a pattern to be matched against the tool’s output. The output is expected to start with
filename:line:followed by the providedexpected_output. Any whitespace immediately followingstdout:/stderr:is ignored and not considered part of the expected output. Similar to the// test:comments, carets (^) can be used to associate the comment with a previous line instead of the current line.Within the expected output, the syntax
$(some_label)can be used to refer to labels. Labels will be substituted withfilename:linefrom the location associated with the label definition. Similarly,$(^),$(^^), etc. refer to the line the comment is associated with. The number of carets must match the comment’s own (//^→$(^),//^^→$(^^)). To refer to any other line, define a label.Example with three violations associated with the same line, one of them referring to the declaration of
fthrough the labelf-def:extern int f(int i, int j); // label: f-def int i, j = 0; for (i = 0; i < 10 && f(i, j); i++) //^ stdout:24: required: Right operand of '&&' should be essentially Boolean. [bool && int32_t] (Rule MisraC2012-10.1) //^^ stdout:24: advisory: Parentheses should be used to avoid dependence on precedence rules [left operand of &&] (Rule MisraC2012-12.1) //^^^ stdout:28: required: Call in loop condition potentially causes side-effect [f()] (Rule MisraC2012-14.2), $(f-def):12: Unknown effect [f()] { }
Example with two violations associated with the same line, each referring to that line and to two other lines:
char* p = malloc(4); // label: alloc_p char* q = malloc(4); // label: alloc_q free(p); // label: free_p free(q); // label: free_q strcpy(p, q); //^ stdout:11: error: Dynamic memory possibly used after it was previously released [heap memory allocated at $(alloc_p):21] (Rule FaultDetection-UseAfterFree), $(alloc_p):21: calling malloc, $(free_p):9: calling free, $(^):12: calling and passing as argument [p] //^^ stdout:11: error: Dynamic memory possibly used after it was previously released [heap memory allocated at $(alloc_q):21] (Rule FaultDetection-UseAfterFree), $(alloc_q):21: calling malloc, $(free_q):9: calling free, $(^^):15: calling and passing as argument [q]
Here, both
$(^)and$(^^)refer to thestrcpy(p, q);line: the first comment sits one line below it and uses one caret (//^ … $(^)), while the second sits two lines below and uses two (//^^ … $(^^)).It is also possible to programmatically add expected output lines:
// test: TESTCASE.add_expected_stdout('abc') // test: TESTCASE.add_expected_stderr('def')
When the provided string contains several lines, these are added separately; common leading indentation is removed, empty lines are ignored and the order of the lines is not significant. Each line is expected as-is, without the automatic prefix of the current line and without expansion of labels.
// stdout(regex=True): expected_output_regexor
// stderr(regex=True): expected_output_regexWhen the expected output is not known exactly (e.g., due to implementation-defined values or types), it is possible to use regular expressions by adding the
regex=Truequalifier://^ stdout(regex=True):1: required: Number of macro definitions in this unit is higher than translation limit \(limit: 1024, actual value: \d+\) \(Rule MisraC\-1\.1\)As with the non-regex
stdoutdirective, the output is expected to start withfilename:line:. Note that special characters must be escaped for parts of the expected output that should be matched literally.An alternative syntax of the form
// stdout/expected_output_regex/is still supported but deprecated.// stdout(wildcard="..."): expected_outputor
// stderr(wildcard="..."): expected_outputFor the same reasons as
regex=True, it may be useful to ignore parts of a message. This can be accomplished by adding awildcard="PLACEHOLDER"qualifier. Any number of characters (or the empty string) will be matched whereverPLACEHOLDERappears in the expected output://^ stdout(wildcard="..."):5: required: Forbidden characters used. [...] (Rule AutosarC++17_03-A2.2.1)As with the other
stdoutdirectives, the output is expected to start withfilename:line:.
Source tests involving multiple files¶
For more advanced usage, the following directives can be used. This is useful for scenarios where you have one main test file and several additional ones, for example, to test the interaction of several source files.
// primary_file: otherfile.cppWith this directive, you mark the current file as a secondary one. This file will not be treated as a test case; instead, execution will be delegated to the file specified by the relative path.
// include: header.hBy default,
perform_testswill only interpret comments in the primary file as test directives. Theincludecomment causes the test runner to also process comments from the named file.Typically, a test case that uses C
#include "header.h"will additionally use// include: header.hso that the header can contribute labels and expected output.A test case that compiles multiple
.cppunits will use// include: file2.cppso that all units can use test comments.
Here is an example of a test case spanning multiple files:
unit1.cpp:
// setup: cafeCC(['unit1.cpp', 'unit2.cpp']) // note: Process testrunner comments in other files: // include: header.h // include: unit2.cpp // test: analysis() #include "header.h" void file1() {}
unit2.cpp:
// primary_file: unit1.cpp #include "header.h" void file2() {}
header.h:
// primary_file: unit1.cpp void file1(); void file2();
Predefined variables¶
TESTCASE.ir_filename: str:The filename where
perform_testsexpects the IR at the end of the setup phase.BAUHAUS_CONFIG: list[str]:Controls the
BAUHAUS_CONFIGenvironment variable for any subprocesses started. The initial value depends on the environment in whichperform_testswas started. The test setup can then insert additional configuration layers.CAFECC_OPTIONS: list[str]:Additional options passed to the
cafeCCcompiler, such as-std=c++17. The initial value of this variable can be controlled via theCAFECC_OPTIONSenvironment variable when startingperform_tests.ENV: dict[str, str]:The environment variables used when
perform_testsstarts subprocesses.- Note that by default,
perform_testsadds these variables to the environment: TEST_SOURCE_DIR: the directory containing the primary file of the test caseTEST_EXEC_DIR: a temporary directory for the test execution.
- Note that by default,
Predefined functions¶
In // setup and // test comments, the following functions are available:
def run(args: list, capture_stdout=True, capture_stderr=False, **kwargs)Run a subprocess. The test fails if the process returns with an exit code != 0. If stdout/stderr are captured, the process output is added to the “actual output,” which is compared with the output expected by
// stdoutcomments.The additional arguments (
kwargs) are forwarded to thesubprocess.runcall.Note that by default, the subprocess will run with the working directory set to a temporary directory (
TEST_EXEC_DIR). Use the environment variableTEST_SOURCE_DIRif the command needs to access the test source files.def cafeCC(args: list)Explicitly run cafeCC with the specified command-line options.
If an argument is a relative path to a file that exists relative to the test source directory,
perform_testsconverts it to an absolute path before passing it to the compiler. All other relative paths are interpreted relative to theTEST_EXEC_DIRtemporary directory.// setup: cafeCC(['unit1.cpp', '-shared', '-o', 'first.dll'])In this
cafeCCinvocation,unit1.cppis read from theTEST_SOURCE_DIR, butfirst.dllis stored in theTEST_EXEC_DIR.If the output filename is not explicitly specified with -o, the IR will be written to
TESTCASE.ir_filename.The
-Boption is automatically passed to the compiler.Any options listed in the global
CAFECC_OPTIONSenvironment variable will be passed to the compiler.
An explicit
cafeCC()invocation will disable the implicit compilation step at the end of the setup phase.def irlink(args: list, ir_file=None)Run the linker with the given command-line arguments. The
ir_fileparameter specifies the output filename. Ifir_fileis not given, the output IR is written toTESTCASE.ir_filename.def analysis(rules=None, *, ...)Run
axivion_analysis.Parameters accepted by this function correspond to the
axivion_analysiscommand-line options:rules: Select rules to execute. If this parameter is omitted, all rules active in the configuration will be executed.excludes: list[str]: Corresponds to--excludeincludes: list[str]: Corresponds to--includebrief: bool = True: Corresponds to--brief(single-line-output)system: bool = True: Corresponds to--systemAdditionally, a
configparameter is supported, allowing you to specify a configuration file to add toBAUHAUS_CONFIG. This is useful when a test not only executes a certain check but also requires configuring that check via a configuration file embedded in the test case. For example:/* rule_config.py: import axivion.config analysis = axivion.config.get_analysis() analysis['MisraC2012-22.9'].errno_readers = ['perror', 'check_errno'] */ /* test: analysis(['MisraC2012-22.9', 'StaticSemanticAnalysis'], config='rule_config.py') */
6.4.9.2. The Test Driver perform_tests¶
The tool perform_tests can be used to execute Source Test Cases.
perform_tests [-h] [-d] [-v] [--no_summary] [--dry_run] [--in_process]
[--language LANGUAGE] [--tempdir TEMPDIR] [--keep_tempdir]
[--no_version_check] [--platforms PLATFORMS] [--no_platform_check]
[--exitfirst] [--maxfail MAXFAIL]
file_or_dir [file_or_dir ...]
The tool is self-describing, but its options are provided here for a quick overview:
Option |
Meaning |
|---|---|
|
Emit verbose output helpful for diagnosing tests. |
|
Emit detailed information helpful for debugging tests. |
|
Do not output the summary at the end of the test run. |
|
Do not run tests; only count and list the tests that match the language and platform filters. |
|
Execute the tests in the same process, e.g., to attach a debugger. This is unsafe when running several tests. |
|
Only run tests compatible with the specified language version. |
|
Use the given directory instead of an automatic temporary one. |
|
Do not clean up the temporary directory at the end of the test run. |
|
Do not check whether the |
|
Specify target platform features (influences which tests will run). |
|
Run tests even if they are not applicable to the target platform.
|
|
Stop after the first failed test. |
|
Stop after the given number of failed tests. |
|
The test files to execute sequentially. Directories are searched recursively for test files, and glob patterns are expanded. |
The tool can execute a single test or multiple tests sequentially. To get a first impression, you may want to execute the following command, assuming that you are interested in the Misra-C:2012 tests and are using C99 (i.e., no C11) in your project:
perform_tests --language C99 misra/c2012/misrac2012-7.*
Notably, the --language switch prevents tests that require C11 features from running. This is useful if C11 is not enabled in your project
configuration, as such tests would likely fail for reasons that are difficult to diagnose. Omitting the --language option will cause all tests to be executed, which may be counterproductive – especially if you are not using C++ at all. You can also specify --language C or --language C++ to run all tests for a specific language. This is particularly useful for the HIS and errorchecks directories, as they contain tests for both languages.
Note that specifying a specific language version with the --language switch (e.g.,
C99, rather than just C or C++) disables the feature where perform_tests automatically adds the test’s expected language version to
the compiler invocation. Instead, you must configure the compiler to use a language version that matches the --language switch.
You can do this by setting the CAFECC_OPTIONS environment variable.
In addition to the language filter, perform_tests includes a target platform filter; tests can declare a requirement for a specific target platform feature, such as posix or stdc_atomic. These tests only run when the corresponding target platform feature is enabled via the --platforms switch. For example, perform_tests --platforms linux,gnu,stdc_atomic will run tests requiring any of these three platform identifiers, but will skip tests requiring others (e.g., posix tests will not run).
If the --platforms switch is not specified, the target
platforms are auto-detected from the compiler configuration.
The available and default-active platforms are displayed
in the perform_tests --help output.
In summary, perform_tests reports the number of tests that failed (i.e., the output did not match the expected output), the number of tests that passed (i.e., the output matched the expected output), and tests with execution errors or failed setup. Tests that are not applicable to a specific platform are also reported.
The output from the example of using perform_tests to run the Misra-C:2012 tests for rules 7.1 to 7.4 may look like this:
perform_tests --language C99 misra/c2012/misrac2012-7.*
Test "MisraC2012-7.1" passed
Test "MisraC2012-7.2" passed
Test "MisraC2012-7.3" passed
Test "MisraC2012-7.4" passed
--------------------------------------------------------------------------------
Test Summary:
--------------------------------------------------------------------------------
Count of Tests passed:.................. 4
Count of Tests failed:.................. 0
Count of Tests with execution problems:. 0
Count of Tests with failed setup:....... 0
Count of n/a tests:..................... 1
--------------------------------------------------------------------------------
For simplicity, test outputs are not printed by default; only the test results are displayed.
When developing tests, the -v and -d switches (especially when used together) can be used to check intermediate results.
6.4.9.3. The pytest-Based Test Driver axivion_test¶
The axivion_test tool executes source test cases using pytest.
axivion_test [--dry_run] [--in_process] [--pytest_all] [<further-pytest-options>]
SOURCE_TEST_FILE_OR_DIR [SOURCE_TEST_FILE_OR_DIR ...]
Unlike perform_tests, it respects the xfail and skip directives. It
accepts the following perform_tests options: --language,
--platforms, --no_platform_check, --tempdir,
--keep_tempdir, and --no_version_check, and passes all other
unrecognized options to pytest. For example, the following command runs
all tests in a directory in parallel using the -n option from pytest-xdist:
axivion_test -n auto misra/c2012/
Option |
Meaning |
|---|---|
|
Only collect the tests, alias for |
|
Disable |
|
Execute all collected |
6.4.9.4. The Qualification Kit Executor axivion_qkit¶
The Axivion Qualification Kits (see Section Axivion Qualification Kits User Guide) can be executed using the dedicated axivion_qkit tool.
axivion_qkit [--dry_run] [--in_process] [--coverage COVERAGE] [--report_kind {generic,do330}]
[--report_directory REPORT_DIRECTORY] [--report_overwrite] [--add_rulegroup_coverage_to_junit_xml]
[--pytest_all] [<further-pytest-options>] SOURCE_TEST_FILE_OR_DIR [SOURCE_TEST_FILE_OR_DIR ...]
The primary purpose of axivion_qkit is to execute all specified source test cases, similar to perform_tests. In addition to executing test cases, it performs additional checks, particularly when instructed to perform a coverage check for an entire rule group (see --coverage).
Check if every rule in the given rule group is enabled in the current configuration.
Check if at least one test case passed (i.e., was not skipped) for every rule in the given rule group.
If any test case in the given rule group requires a semantic analysis engine, it checks if that engine is enabled in the current configuration.
axivion_qkit can also generate a qualification report as a PDF file (see --report_directory). This report lists all primary test case results, grouped by the analysis rule that was tested (or would have been tested, in the case of skipped test cases), along with the results of the coverage checks mentioned above.
Note
It is implemented as a pytest plugin; the axivion_qkit tool is a thin wrapper that calls pytest with the appropriate command-line arguments to load and execute the plugin. Consequently, other pytest features – such as selecting test cases by pattern using -k, by marker using -m, or parallel execution using the pytest-xdist -n option – are also supported.
axivion_qkit passes all unrecognized command-line options directly to pytest.
Note
As a side effect of passing all options to pytest, the --help option prints both axivion_qkit’s own help message and the complete output of pytest --help.
This is a brief overview of the most important axivion_qkit-specific options. For further details, refer to the help message, the perform_tests documentation, and the pytest documentation:
Option |
Meaning |
|---|---|
|
Create a coverage report for the specified rule group (multiple comma-separated values are possible). |
|
The type of PDF report to create: |
|
Directory in which to save the coverage report PDF and related files. |
|
If not set, the tool will abort early if the coverage report already exists. |
|
If set, the results of the rule group coverage check are added as test cases to the JUnit XML file generated by pytest’s |
|
If set, all collected pytest test cases will be executed instead of only the source test cases. |
|
The file or directory name from which to collect test cases. |