4.8. Risk Ranking¶
4.8.1. Introduction¶
An issue is not equally risky throughout the code. The risk depends on the likelihood that the issue is real, the severity of the consequences, the quality of the surrounding tests, and the amount of external knowledge available about the code. Regulations such as the EU Cyber Resilience Act (CRA) require manufacturers to assess the cybersecurity risks of their products and to base their handling of vulnerabilities on that assessment. The risk ranking provides the data for this part of the assessment: it records, for every issue, the factors that contribute to its risk and combines them into an overall risk score.
Each issue has five numeric columns – the rank factors – which the Dashboard and IDE plugins can display alongside the issue. Each value lies between 0.0 and 1.0. By default, a higher value indicates higher risk, although the contributor options described below can change this behavior. The columns do not replace a risk assessment of the product; they make the assumptions behind it explicit and reproducible, since each factor is computed by a configured rule rather than judged case by case.
The columns are generic. The content, label, and computation of each column are defined in the configuration. The default configuration fills them as follows:
Column |
Default label |
Default content |
|---|---|---|
|
Risk |
The overall score, aggregated from the Confidence and Impact columns. Visible by default. |
|
Confidence |
How certain the analysis is that the issue is real. |
|
Impact |
How severe the issue is, derived from the severity of the rule that reported it. |
Factor3 and Factor4 remain empty until entries are added, such as for test coverage or code age.
The column labels are shown in the Dashboard; see Table Columns.
Only the Risk column is visible by default; other columns can be enabled via Show columns in the table menu.
4.8.2. Configuration¶
The configuration is found below the rule group RiskRanking in the analysis configuration.
The group is an ordered list of entries, configured like the Architecture group: see Graphical Configuration Interface.
Each active entry fills one column in the order of the list: the first entry fills Factor0, the second Factor1, and so on.
The name of an entry is the label of its column.
There can be a maximum of five active entries, one for each column.
The default list consists of the entries Risk, Confidence and Impact.
To add an entry, drag a contributor from Available actions into the list and give it a name, which becomes the label of its column.
To change the order of the columns, reorder the entries; to empty a column, deactivate its entry.
Each entry has a show_by_default option that makes its column visible in the Dashboard issue table without enabling it in the table menu.
The options unknown_severity_score, unmatched_criticality, and missing_value provide a score for issues where no information is available; if these are unset, the entry provides no value and the column remains empty for that issue.
Note
The column values are not taken from an analysis result file. When issues are imported, columns are recomputed using the configuration of the importing analysis (e.g., using its own coverage reports). Only the confidence of an issue is retained from the risk ranking data in the file.
The following example adds a column to rank issues by the age of the code in which they occur. Any other metric is configured the same way. In the configuration GUI:
Activate the rule Metric-VCS.Age.
Drag RiskRanking-Metric from
Available actionsof RiskRanking behind the last entry and name itCode Age.Set
metricofCode AgetoMetric.VCS.Age,min_valueto0andmax_valueto2000.
Since the new entry is fourth in the list, it fills Factor3 and is labeled Code Age.
4.8.3. Contributors¶
These are the actions used to create the entries in the list.
- RiskRanking-AnalysisConfidence
Uses the confidence reported by the analysis for the issue. The default entry
Confidenceis created from it. See Analysis confidence.- RiskRanking-Severity
Scores the severity of the rule that reported the issue. The default entry
Impactis created from it. The optionseverity_scoresmaps each severity to a score. It supports the obligation levels of MISRA and AUTOSAR, the severities of CERT and CWE, and the diagnostic levels of compiler frontends; adjust the scores to match the standards used for the project. Issues of types that do not carry a severity – such as clones, cycles, dead entities, and architecture violations – receive theunknown_severity_score.- RiskRanking-FileCriticality
Scores the files containing an issue based on patterns provided in
criticality, for example:{"**": 0.2, "src/crypto/**": 1.0}. The most specific matching pattern takes precedence. An issue that relates several places, such as a clone pair, is as critical as the most critical file it involves. Files that do not match any pattern are assigned theunmatched_criticalityscore.- RiskRanking-Metric
Scores a metric of the code containing the issue. Set
metricto the RFG node attribute of the metric (e.g.,Metric.Fan.In.Call) to rank issues higher when more code depends on the containing routine. The rule that computes the metric must be active. A value ofmin_valuescores 0.0 andmax_valuescores 1.0, using a linear scale. Amax_valuebelowmin_valueinverts the scale. Issues for which no value is found scoremissing_value.- RiskRanking-Coverage
Scores the issue by test coverage; see Coverage.
- RiskRanking-CocoCoverage
Scores the issue by the test coverage measured by Coco; see Coco coverage.
- RiskRanking-Callback
Scores the issue using a Python function specified in the
callbackoption of a Python configuration file. The function receives the finding, the IR graph (orNoneif no IR exists), and the RFG, returning a value between 0.0 and 1.0, orNoneif no value should be assigned.- RiskRanking-FromOtherRankingFactors
Computes an overall score from other entries. The default entry
Riskis created from it. Theweightsoption specifies the entries to be combined and assigns a weight to each for the weighted mean. Theaggregationoption selects the combination method:minimum,maximum,weighted_mean(default), orproduct. Only entries with a value for the issue are included. Each named entry must be active and cannot be an aggregate itself; weights must be non-negative.
4.8.4. Analysis confidence¶
Confidence indicates how certain the analysis is that a reported issue is not a false positive. Checks that determine exactly whether code violates a rule report issues with full confidence. Other checks rely on approximations. For example, a null pointer dereference is certain if the pointer was just compared against null, but only possible if the pointer is a parameter with unknown properties; confidence distinguishes these two cases.
Confidence differs from severity, which describes how severe an issue would be if it were real. It also differs from the likelihood assigned by coding standards such as CERT, which refers to how likely the code is to be exploitable.
Most checks use one of three levels:
definiteThe analysis proves the issue without approximation. Score: 1.0.
probableThe issue is based on an analysis approximation but is still considered likely. For semantic checks, this means the issue survived the heuristics used to filter out unlikely over-approximations. Score: 0.66.
improbableThe analysis over-approximates and considers the issue unlikely, yet reports it anyway. Score: 0.33.
Style checks specify confidence via the confidence option, though individual messages within a check may specify their own.
Rules consisting of multiple parts report the confidence of the specific part that generated the issue.
Checks that do not specify confidence report none; issues from such checks have no value in the confidence column.
The confidence option can be set in the configuration to one of the three levels to override a check’s default.
The analysis may also compute its own score between 0.0 and 1.0 for an issue; this is not available as a configuration value.
4.8.5. Coverage¶
The RiskRanking-Coverage contributor reads coverage reports and scores each issue based on how often the code at its location was executed. It reads reports in the lcov tracefile format and the Cobertura XML format. Reports are matched to issue files using the longest common trailing path, allowing reports from different checkouts to be applied.
coverage_filesThe reports to read.
metricThe kind of coverage:
line,branch,functionormcdc. Both formats carry line and branch coverage. Function coverage is exact in lcov and derived from the first line of a method in Cobertura. MC/DC coverage is only available in lcov tracefiles written by lcov 2.2 or later.scores_highuntested(default) assigns high scores to issues in rarely executed code, ranking them above issues in tested code because tests have not yet verified the functionality of that code.frequently_executeddoes the opposite, ranking issues in frequently executed code higher, as they are more likely to occur during operation.saturation_hitsThe execution count at which code is considered fully tested, using a logarithmic scale. The default value of 1 only distinguishes between executed and non-executed code.
missing_valueThe score for issues in locations not covered by reports, such as lines without code or non-instrumented files.
4.8.6. Coco coverage¶
The RiskRanking-CocoCoverage contributor scores each issue based on test coverage measured by Coco.
It reads the Cobertura XML reports that cmreport writes from an instrumentation database with imported executions, for example:
cmreport --csmes=app.csmes --cobertura=coverage.xml --cobertura-source=/path/to/sources \
--cobertura-manual-validation-hits=1000 --coverage-mcdc
--cobertura-manual-validation-hits gives manually validated code an execution count.
Set this to at least saturation_hits so that such code is considered fully tested.
Leave out -T, which replaces the execution counts with test counts.
The options are those of RiskRanking-Coverage, except for metric:
metricline(default) scores based on line execution frequency;functionscores based on function call frequency.coverage_levelscores based on the proportion of covered outcomes on the line, using the coverage level specified whencmreportwrote the report: statement block, decision, condition (thecmreportdefault), MC/DC, or multiple condition coverage. Reports do not record their coverage level; therefore, each level requires its own report and entry.
4.8.7. Metrics as risk indicators¶
The RiskRanking-Metric contributor can utilize any metric from the metrics guide; the choice of metrics depends on what the project considers risky. Different metrics indicate different types of risk. Each metric is assigned its own entry and column, and entries created via RiskRanking-FromOtherRankingFactors can combine them:
Fan-in (e.g.,
Metric.Fan.In.Call) measures how much code depends on a routine. Issues in widely used code can affect many callers, thereby increasing the impact.Complexity (e.g.,
Metric.McCabe_Complexity) or size (e.g.,Metric.Lines.LOC) measures how difficult a routine is to understand. Issues in complex code are harder to fix correctly and more likely to conceal further problems.Churn, the number of authors, and the age of the code indicate stability and familiarity. Frequently changed code, code touched by many authors, and very new code have had less time to be proven through use.
The version control history is available as the metrics Metric.VCS.Age, Metric.VCS.Authors and Metric.VCS.Churn, computed by the rules Metric-VCS.Age, Metric-VCS.Authors and Metric-VCS.Churn.
These are only available when the analysis runs in the CI, and the project must use a supported version control system.
Churn is only available for Git.
Custom metrics imported via Metric-ImportCSV can be used in the same manner.