4.8. Risk Ranking

4.8.1. Introduction

An issue is not equally risky throughout the code. The risk depends on the likelihood that the issue is real, the severity of the consequences, the quality of the surrounding tests, and the amount of external knowledge available about the code. Regulations such as the EU Cyber Resilience Act (CRA) require manufacturers to assess the cybersecurity risks of their products and to base their handling of vulnerabilities on that assessment. The risk ranking provides the data for this part of the assessment: it records, for every issue, the factors that contribute to its risk and combines them into an overall risk score.

Each issue has five numeric columns – the rank factors – which the Dashboard and IDE plugins can display alongside the issue. Each value lies between 0.0 and 1.0. By default, a higher value indicates higher risk, although the contributor options described below can change this behavior. The columns do not replace a risk assessment of the product; they make the assumptions behind it explicit and reproducible, since each factor is computed by a configured rule rather than judged case by case.

The columns are generic. The content, label, and computation of each column are defined in the configuration. The default configuration fills them as follows:

Column

Default label

Default content

Factor0

Risk

The overall score, aggregated from the Confidence and Impact columns. Visible by default.

Factor1

Confidence

How certain the analysis is that the issue is real.

Factor2

Impact

How severe the issue is, derived from the severity of the rule that reported it.

Factor3 and Factor4 remain empty until entries are added, such as for test coverage or code age. The column labels are shown in the Dashboard; see Table Columns. Only the Risk column is visible by default; other columns can be enabled via Show columns in the table menu.

4.8.2. Configuration

The configuration is found below the rule group RiskRanking in the analysis configuration. The group is an ordered list of entries, configured like the Architecture group: see Graphical Configuration Interface. Each active entry fills one column in the order of the list: the first entry fills Factor0, the second Factor1, and so on. The name of an entry is the label of its column. There can be a maximum of five active entries, one for each column.

The default list consists of the entries Risk, Confidence and Impact. To add an entry, drag a contributor from Available actions into the list and give it a name, which becomes the label of its column. To change the order of the columns, reorder the entries; to empty a column, deactivate its entry.

Each entry has a show_by_default option that makes its column visible in the Dashboard issue table without enabling it in the table menu. The options unknown_severity_score, unmatched_criticality, and missing_value provide a score for issues where no information is available; if these are unset, the entry provides no value and the column remains empty for that issue.

Note

The column values are not taken from an analysis result file. When issues are imported, columns are recomputed using the configuration of the importing analysis (e.g., using its own coverage reports). Only the confidence of an issue is retained from the risk ranking data in the file.

The following example adds a column to rank issues by the age of the code in which they occur. Any other metric is configured the same way. In the configuration GUI:

  1. Activate the rule Metric-VCS.Age.

  2. Drag RiskRanking-Metric from Available actions of RiskRanking behind the last entry and name it Code Age.

  3. Set metric of Code Age to Metric.VCS.Age, min_value to 0 and max_value to 2000.

Since the new entry is fourth in the list, it fills Factor3 and is labeled Code Age.

4.8.3. Contributors

These are the actions used to create the entries in the list.

RiskRanking-AnalysisConfidence

Uses the confidence reported by the analysis for the issue. The default entry Confidence is created from it. See Analysis confidence.

RiskRanking-Severity

Scores the severity of the rule that reported the issue. The default entry Impact is created from it. The option severity_scores maps each severity to a score. It supports the obligation levels of MISRA and AUTOSAR, the severities of CERT and CWE, and the diagnostic levels of compiler frontends; adjust the scores to match the standards used for the project. Issues of types that do not carry a severity – such as clones, cycles, dead entities, and architecture violations – receive the unknown_severity_score.

RiskRanking-FileCriticality

Scores the files containing an issue based on patterns provided in criticality, for example: {"**": 0.2, "src/crypto/**": 1.0}. The most specific matching pattern takes precedence. An issue that relates several places, such as a clone pair, is as critical as the most critical file it involves. Files that do not match any pattern are assigned the unmatched_criticality score.

RiskRanking-Metric

Scores a metric of the code containing the issue. Set metric to the RFG node attribute of the metric (e.g., Metric.Fan.In.Call) to rank issues higher when more code depends on the containing routine. The rule that computes the metric must be active. A value of min_value scores 0.0 and max_value scores 1.0, using a linear scale. A max_value below min_value inverts the scale. Issues for which no value is found score missing_value.

RiskRanking-Coverage

Scores the issue by test coverage; see Coverage.

RiskRanking-CocoCoverage

Scores the issue by the test coverage measured by Coco; see Coco coverage.

RiskRanking-Callback

Scores the issue using a Python function specified in the callback option of a Python configuration file. The function receives the finding, the IR graph (or None if no IR exists), and the RFG, returning a value between 0.0 and 1.0, or None if no value should be assigned.

RiskRanking-FromOtherRankingFactors

Computes an overall score from other entries. The default entry Risk is created from it. The weights option specifies the entries to be combined and assigns a weight to each for the weighted mean. The aggregation option selects the combination method: minimum, maximum, weighted_mean (default), or product. Only entries with a value for the issue are included. Each named entry must be active and cannot be an aggregate itself; weights must be non-negative.

4.8.4. Analysis confidence

Confidence indicates how certain the analysis is that a reported issue is not a false positive. Checks that determine exactly whether code violates a rule report issues with full confidence. Other checks rely on approximations. For example, a null pointer dereference is certain if the pointer was just compared against null, but only possible if the pointer is a parameter with unknown properties; confidence distinguishes these two cases.

Confidence differs from severity, which describes how severe an issue would be if it were real. It also differs from the likelihood assigned by coding standards such as CERT, which refers to how likely the code is to be exploitable.

Most checks use one of three levels:

definite

The analysis proves the issue without approximation. Score: 1.0.

probable

The issue is based on an analysis approximation but is still considered likely. For semantic checks, this means the issue survived the heuristics used to filter out unlikely over-approximations. Score: 0.66.

improbable

The analysis over-approximates and considers the issue unlikely, yet reports it anyway. Score: 0.33.

Style checks specify confidence via the confidence option, though individual messages within a check may specify their own. Rules consisting of multiple parts report the confidence of the specific part that generated the issue. Checks that do not specify confidence report none; issues from such checks have no value in the confidence column. The confidence option can be set in the configuration to one of the three levels to override a check’s default. The analysis may also compute its own score between 0.0 and 1.0 for an issue; this is not available as a configuration value.

4.8.5. Coverage

The RiskRanking-Coverage contributor reads coverage reports and scores each issue based on how often the code at its location was executed. It reads reports in the lcov tracefile format and the Cobertura XML format. Reports are matched to issue files using the longest common trailing path, allowing reports from different checkouts to be applied.

coverage_files

The reports to read.

metric

The kind of coverage: line, branch, function or mcdc. Both formats carry line and branch coverage. Function coverage is exact in lcov and derived from the first line of a method in Cobertura. MC/DC coverage is only available in lcov tracefiles written by lcov 2.2 or later.

scores_high

untested (default) assigns high scores to issues in rarely executed code, ranking them above issues in tested code because tests have not yet verified the functionality of that code. frequently_executed does the opposite, ranking issues in frequently executed code higher, as they are more likely to occur during operation.

saturation_hits

The execution count at which code is considered fully tested, using a logarithmic scale. The default value of 1 only distinguishes between executed and non-executed code.

missing_value

The score for issues in locations not covered by reports, such as lines without code or non-instrumented files.

4.8.6. Coco coverage

The RiskRanking-CocoCoverage contributor scores each issue based on test coverage measured by Coco. It reads the Cobertura XML reports that cmreport writes from an instrumentation database with imported executions, for example:

cmreport --csmes=app.csmes --cobertura=coverage.xml --cobertura-source=/path/to/sources \
    --cobertura-manual-validation-hits=1000 --coverage-mcdc

--cobertura-manual-validation-hits gives manually validated code an execution count. Set this to at least saturation_hits so that such code is considered fully tested. Leave out -T, which replaces the execution counts with test counts.

The options are those of RiskRanking-Coverage, except for metric:

metric

line (default) scores based on line execution frequency; function scores based on function call frequency. coverage_level scores based on the proportion of covered outcomes on the line, using the coverage level specified when cmreport wrote the report: statement block, decision, condition (the cmreport default), MC/DC, or multiple condition coverage. Reports do not record their coverage level; therefore, each level requires its own report and entry.

4.8.7. Metrics as risk indicators

The RiskRanking-Metric contributor can utilize any metric from the metrics guide; the choice of metrics depends on what the project considers risky. Different metrics indicate different types of risk. Each metric is assigned its own entry and column, and entries created via RiskRanking-FromOtherRankingFactors can combine them:

  • Fan-in (e.g., Metric.Fan.In.Call) measures how much code depends on a routine. Issues in widely used code can affect many callers, thereby increasing the impact.

  • Complexity (e.g., Metric.McCabe_Complexity) or size (e.g., Metric.Lines.LOC) measures how difficult a routine is to understand. Issues in complex code are harder to fix correctly and more likely to conceal further problems.

  • Churn, the number of authors, and the age of the code indicate stability and familiarity. Frequently changed code, code touched by many authors, and very new code have had less time to be proven through use.

The version control history is available as the metrics Metric.VCS.Age, Metric.VCS.Authors and Metric.VCS.Churn, computed by the rules Metric-VCS.Age, Metric-VCS.Authors and Metric-VCS.Churn. These are only available when the analysis runs in the CI, and the project must use a supported version control system. Churn is only available for Git.

Custom metrics imported via Metric-ImportCSV can be used in the same manner.