4.6.8. Axivion Qualification Kits User Guide

4.6.8.1. Introduction

Here we explain how the Axivion QKit C/C++, the Axivion QKit for CUDA and the Axivion QKit for AV are used in order to demonstrate that checks performed by Axivion Suite are correctly configured and are functioning as expected.

The Axivion QKit C/C++ focuses on HIS Metrics, MISRA-C:2012, MISRA-C++:2008, and other checks as defined in section Axivion QKit C/C++ Contents. The Axivion QKit for AV focuses on rules related to architecture and RFG verification / modification. The Axivion QKit for CUDA focuses on NVIDIA CUDA C++ guidelines and MISRA C++ 2023 rules on CUDA code.

Please refer to the safety standards you have to apply regarding when and how to use the various qualification kits.

4.6.8.2. Prerequisites

In order to use the qualification kits, at least the following prerequisites must be met.

  1. Install and set up Axivion Suite properly. Carefully consider the Axivion Reference Guide in particular with regards to MISRA and HIS configuration. Make sure to understand the size_t concepts of your environment and how to configure the code analysers accordingly.

  2. If function pointers are used in your code and these shall be considered for metrics, you have to set up the rule StaticSemanticAnalysis the callgraph metrics in your project configuration so that the tools take pointers into account. The option required for StaticSemanticAnalysis is semantics.advanced.treat_uncalled_functions_as_alive, for the callgraph metrics use consider_dynamic_calls.

  3. Configure Axivion Suite for the use with the target compiler and project settings. The qualification kits can only produce reliable results if the configuration to the respective compiler and project environment is free from errors. For example,

    • The environment variable BAUHAUS_CONFIG must be set to the directory layers where the project configuration(s) is/are located.

    • The compiler profile used must be set up properly (e.g. predefined macros, standard include paths and type sizes must be configured correctly).

    • There must be no error messages during code analysis. To detect that kind of messages, activate the stylecheck rules Generic-NoCompilerWarnings.

    • Avoid warning messages during code analysis. To detect that kind of messages, activate the stylecheck rules Generic-NoCompilerWarnings.

    • There must be no linker errors. To detect that kind of messages, activate the stylecheck rules Generic-NoLinkerWarnings.

    • Note that, in addition to the configuration files in BAUHAUS_CONFIG, there can be switches in your build system (e. g., makefiles) passed to your compiler and cafeCC, respectively, that influence the tools’ behavior. For example, choosing a certain language standard for C or C++ must be considered -std=c++14. Make sure the tests provided in the qualification kit are aligned with your compiler switches. For this purpose, the environment variable CAFECC_OPTIONS can be used to pass additional options to the cafeCC frontend during test execution.

  4. The computing machinery running the tests inside the qualification kits must be as close to the production environment as possible. Ideally, the tests run on exactly the same machine. To name a few important things,

    • The version of Axivion Suite must be identical.

    • The target compiler settings during the analysis build and the header files of the compiler must be the same.

    • The operating systems of the host platform must be the same.

    • User and access rights should be the same.

  5. The version of the qualification kits must match the version of Axivion Suite. This is ensured by the test driver for the tests included, but users are responsible for ensuring it for their own tests.

  6. If you want to execute tests for the IBM Rhapsody importer in the Axivion QKit for AV, you must set the environment variable RHAPSODY_CL_PATH. RHAPSODY_CL_PATH must be set to the path to the RhapsodyCL.exe. If RHAPSODY_CL_PATH is not set, the tests for the IBM Rhapsody importer will not be executed.

Each time a relevant part of the tool chain changes, tests must be re-run to ensure that the configuration is still valid.

4.6.8.3. Limitations

Notice that the problems addressed by several of the checks covered in the qualification kits are undecidable. This means that only approximations can be calculated by the checks, which can result in missing and incorrect notifications. For example, as a compromise to get results in a reasonable amount of time, the current analysis does not distinguish array elements, nor does it track integer values through pointer-indirect expressions. While the qualification kits for the undecidable cases demonstrate that the Axivion Suite is able to detect instances of these problems, no guarantee can be made that all such instances are detected. A manual check of the program under analysis is required for a definitive assessment.

4.6.8.4. Axivion QKit C/C++ Contents

The Axivion QKit C/C++ comes with a number of self-contained test files all having one of the file suffixes .cpp, .c, or .h. See Section Source Test File Format for a description of the file format.

The main task for using the Axivion QKit C/C++ will be to select and adjust the correct set of test files for you to run in order to test the Axivion Suite against your compiler and setup.

In the following you can see the directory structure of the Axivion QKit C/C++:

autosar/
17_03/

test files for AutosarC++17_03

17_10/

test files for AutosarC++17_10

18_03/

test files for AutosarC++18_03

18_10/

test files for AutosarC++18_10

19_03/

test files for AutosarC++19_03

cert/
c/

rule group directories containing test files for CertC

cpp/

rule group directories containing test files for CertC++

cwe/

test files for Common Weakness Enumeration

errorchecks/

test files for errorchecks divided into subfolders

his/

test files for HIS metrics

misra/
c2012/

test files for MisraC2012

c2019/

test files for MisraC2019

c2023/

test files for MisraC2023

c2025/

test files for MisraC2025

cpp/

test files for MisraC++2008

cpp2023/

test files for MisraC++:2023

secure_coding/

test files for ISO/IEC TS 17961:2013 C Secure Coding Rules

4.6.8.5. Axivion QKit for CUDA Contents

The Axivion QKit for CUDA comes with a number of self-contained test files all having one of the file suffixes .cpp, .c, or .h. See Section Source Test File Format for a description of the file format.

Just as with the Axivion QKit C/C++, you must select and adjust the correct set of test files to run in order to test the Axivion Suite against your compiler and setup.

In the following you can see the directory structure of the Axivion QKit for CUDA:

cuda/

test files for NVIDIA CUDA C++ Guidelines for robust and safety critical programming and test files for CUDA security checks

misra_on_cuda/

test files for MisraC++:2023 applied to CUDA C++ code

4.6.8.6. Axivion QKit for AV Contents

The Axivion QKit for AV comes with a number of self-contained test files all having one of the file suffixes .cpp, .c, or .h. See Section Source Test File Format for a description of the file format.

Just as with the Axivion QKit C/C++, you must select and adjust the correct set of test files to run in order to test the Axivion Suite against your compiler and setup.

In the following you can see the directory structure of the Axivion QKit for AV:

Architecture_Analyses/

test files for architecture analyses, sanity checks, and model transformation

Dependencies/

test files for enhancing the RFG with types and access information

Exporters/

test files for GXL exports and RFG storage

Graph_Based_Analyses/

test files for additional RFG-based analyses, such as cycle detection, dead code detection, or RFG schema checks

Importers/

test files for importers targeting various sources, such as Enterprise Architect or PlantUML

Legacy/

test files for legacy rules

Transformations/

test files for graph transformations

The directories containing test files contain more tests than you will want to execute. We recommend executing only the tests for one of the subdirectories at once. axivion_qkit assists you (in a limited way) in further excluding tests that won’t work with your C or C++ compiler, however, as stated in Section Limitations you will still need to look at the individual test results in order to know for sure, whether a positive test is a result of a bug in the Axivion Suite or test-case assumptions not being met by your compiler configuration. See Section Tasks on the general test exclusion process.

For details on the tool axivion_qkit and its test file formats, see The Qualification Kit Executor axivion_qkit and Source Test File Format respectively.

Instead of axivion_qkit, the Axivion Qualification Kits can in principle also be executed with the tool perform_tests. Note, however, that only axivion_qkit provides meta-testing features like rule group test coverage, and pdf report generation.

4.6.8.7. Tasks

The tests delivered as a part of the Axivion QKit C/C++, Axivion QKit for CUDA or the Axivion QKit for AV serve as a blueprint. In most cases, some tests will need to be adapted to the target environment to reflect the specific setup in the user’s settings.

Proceed as follows:

  1. Configure your project and point the environment variable BAUHAUS_CONFIG to the configuration directory layers.

  2. Run the tests by executing axivion_qkit and check which tests produce errors or failure messages. If all or many test executions look weird, try to locate and eliminate the cause.

  3. Rerun the failed tests with verbose and debug options and check the failed tests:

    • Check if a precondition explained here is not met. If so, meet the precondition and rerun the tests.

    • Most test cases are self-contained, but some need the environment (e.g. headers of an installed compiler). The configuration of cafeCC and other tools may also interfere with test execution. Check if and why your local installation interferes with test execution.

    • Was the build step of the test successful? (If not, check the test area and try to reproduce/fix build errors manually.)

    • Does the configuration of the test (in the temporary test area) match the configuration of the project?

    • Is there additional unexpected output that results from header files in unusual locations? (Adapt the remove pattern accordingly.)

    • Do not change the test’s code without being absolutely sure what you do. If you apply changes to the test cases, make sure by use of appropriate means that you are able to identify later on why and how you changed them.

    • Report to Axivion (axivion.support@qt.io). We will assist you in pinning down the problem.

  4. Iterate until all tests pass.