6.4.8. Generating review comments with annotate_pr¶
Note
Starting with 7.11.0, you can also annotate pull requests as part of the Continuous Integration by adding an AnnotatePR rule in PostCIActions.
The documentation below assumes that you are using annotate_pr as a script, but the options are essentially the same for both methods.
The differences are as follows.
The script prepends a checkbox to each reported issue unless --no_checkbox is given, whereas the rule does so only if general_options.markdown_checkbox is set.
Instead of --exit_code_if_issues_exist, the rule has the option general_options.fail_if_issues_exist.
A rule failure changes the exit code of axivion_ci only if abort_on_error is set; the exit code is then determined by failure_exit_code.
Note that while using it as a rule is generally easier and preferable (e.g., due to GUI configuration and integration into the existing axivion_ci run), the command-line version may be necessary in some cases.
For example, under some configurations (such as the warning and error report types on GitHub), annotate_pr must be run within that platform’s CI framework for the comments to be processed correctly, even if you run Axivion’s general continuous integration on a different platform.
This script is intended to be used as part of a CI/CD pipeline.
It generates reports based on the issues found by the Axivion Suite and posts them to one of the supported platforms (see below).
Reports are either posted as comments to a pull request (if --report_as_one_comment is used) or as inline code review annotations.
Note that parameters can also be passed via environment variables, but command-line options always take precedence.
Use annotate_pr $PLATFORM --help to view all available options for each platform, where $PLATFORM is one of the platforms listed below.
- Supported platforms:
Bitbucket Cloud (
bitbucket_cloud)Bitbucket Server (
bitbucket_server)Gerrit (
gerrit)Gitea (
gitea)GitHub Free / Pro / Team / Enterprise Cloud (
github)GitHub Enterprise Server (
githubwith the--enterprise_serveroption)GitLab (
gitlab)
You can also pass dry_run as the platform.
Instead of posting detected issues to a platform, this will simply output them directly.
The Azure DevOps platform is not supported by annotate_pr,
but similar functionality is provided by the Azure Devops Integration.
6.4.8.1. Example¶
Note
In all examples, we assume that issues are retrieved from the Dashboard, which is authenticated using environment variables not specified here (see Retrieving the issues for details).
Additionally, we use HelloWorld as the name of the Dashboard project.
For example, you may want to post issues to a Bitbucket Cloud pull request.
By default, issues are retrieved from the most recent version of the Dashboard project relative to the previous one.
For Bitbucket, issues are marked as inline report annotations.
To do this for pull request 42 at Bitbucket project foo/bar, you would use:
annotate_pr bitbucket_cloud --dashboard_project_name HelloWorld --server https://api.bitbucket.org/2.0 --token my-access-token --repository foo/bar --pull_request 42
Additionally, you may want to include issues added in earlier revisions.
The command would look like this (substituting [...] for the options from the first example):
annotate_pr bitbucket_cloud [...] --start_revision 0
Alternatively, you can also include issues that are not part of the changes in the pull request.
In that case, it may make sense to post these as a single comment rather than as individual annotations.
Otherwise, Bitbucket does not display annotations outside of the changed lines and only includes them in the report count; annotate_pr will flag this with a warning unless --no_annotations_warning is provided.
On other platforms, the script may fail with an error, as it cannot leave annotations outside of the changed lines.
The complete command would be:
annotate_pr bitbucket_cloud [...] --keep_issues all --report_as_one_comment
Finally, you may want to report all issues inside the diff as annotations and all issues outside the diff as comments (since annotations cannot be placed outside the diff).
To achieve this, run annotate_pr twice: once for issues inside the diff and
once for issues outside the diff:
annotate_pr bitbucket_cloud [...] --keep_issues inside_diff --report_type ANNOTATIONS
annotate_pr bitbucket_cloud [...] --keep_issues outside_diff --report_type COMMENTS
For full details on available configuration options, refer to annotate_pr $PLATFORM --help.
6.4.8.2. Retrieving the issues¶
There are four ways to retrieve issues from the Axivion Suite: using the Dashboard API, a database file (.db), an AR file (.ar), or a JSON file containing an issue table retrieved from the Dashboard API (.json).
Retrieving from the Dashboard¶
- To access the Dashboard, one of the following must be set (in addition to
--dashboard_urland--dashboard_project_name):[*] --dashboard_api_token: Uses an authentication token for the Dashboard API.--dashboard_db_token: Uses a database uid string as credentials to access the project associated with the uid. Can be obtained by executingcidbman metadata list path/to/project.db.--dashboard_usernameand--dashboard_password: Uses a username/password combination for the Dashboard.
Retrieving from a database file¶
Alternatively, you can pass in a database containing results directly.
You can do this by passing the path to a .db file using the --analysis_results_database option.
Retrieving from an AR file¶
Similarly, you can pass a binary AR file generated by axivion_analysis (using --output_binary) directly via annotate_pr $PLATFORM [...] --analysis_results_binary results.ar, where results.ar is the AR file containing analysis findings.
This option does not require a Dashboard to be part of the workflow.
Caveat If you load issues from a database file or from an AR file, you may see findings on the git platform that are semantically unrelated to the current pull request.
This can happen if an older, but still existing, issue has the same source line as a current issue.
This occurs because annotate_pr checks whether the source location of every issue loaded from the files lies within the git changeset for the current PR.
For example, assume the following code snippet is in the current git main branch:
int function_name(int X, int Y) {
return X + Y;
}
Assume the configured rule set specifies a rule requiring every function parameter to start with a lowercase letter. In this case, the analysis of the main branch will produce findings for both parameters X and Y because they are uppercase. If a developer then creates a pull request changing the first parameter from X to x, the AR or database file will still contain a finding for the second parameter Y. Since both parameters are on the same source line, the Y finding will be propagated to the git platform, even though it was not introduced in this pull request.
Retrieving from a JSON file¶
In some use cases, you may want to use data from the Dashboard but either prevent annotate_pr from accessing it directly or modify/filter it before passing it to annotate_pr (see Filtering).
In such cases, you can use the --dashboard_issues_json parameter, which accepts a JSON file containing a single IssueTable retrieved from the Dashboard API.
You can pass this parameter multiple times to include multiple IssueTables.
A minor limitation of this approach is that, as of version 7.10.0, the returned IssueTable does not include the “kind” (e.g., style violation, dead code) of the issues by default. Hence, you need to include the additional kind column when querying the Dashboard. [†] To do this, use the columns query parameter (see Issue Lists API) to explicitly specify the kind column. Note that you must also specify every other column you wish to include; once a column is explicitly specified, the default set of columns is no longer included (see Issue Table Columns for an overview). We will amend the Dashboard API in a future version to include the kind column by default.
For example, to retrieve style violations in this manner and pass them to annotate_pr:
curl --header "Authorization: AxToken $DASHBOARD_API_TOKEN" "$DASHBOARD_URL/api/projects/$DASHBOARD_PROJECT_NAME/issues?kind=SV&columns=id,state,errorNumber,message,entity,path,line,provider,severity,justification,kind" >issues.json
annotate_pr [...] --dashboard_issues_json issues.json
Alternatively, if you are manually constructing, merging, or modifying this JSON, you can add the kind column yourself. Its possible values are SV (Style Violation), AV (Architecture Violation), MV (Metric Violation), CL (Clone), DE (Dead Code), and CY (Cyclic Dependency).
6.4.8.3. Filtering¶
In some circumstances, you may want to run annotate_pr with different parameters depending on the issues being processed.
For example, when using GitHub, high-severity issues could be posted as the “comment” report type, while lower-severity issues are reported as warnings.
Another common use case is to handle issues inside the changed lines of a pull request differently from issues outside those lines (e.g., some platforms only allow comments on lines within the diff, requiring other issues to be reported differently).
To address these scenarios, call annotate_pr multiple times, filtering the issues for each call.
This allows you to specify different parameters for each subset of issues.
Below are concrete examples of how to implement the two scenarios mentioned above.
Filtering inside/outside changed lines¶
As previously mentioned, some platforms restrict certain report types for issues located outside the changed lines of a pull request. In this example, we assume GitLab is being used.
GitLab only allows notes (comments) to be posted for locations inside the diff. To report issues outside the diff as well, you must either use the code_quality report type or the --report_as_one_comment flag, which collects these issues and summarizes them in a single comment posted to the merge request.
Using the latter option looks like this:
# Post issues inside the changed lines as notes...
annotate_pr gitlab [...] --keep_issues inside_diff --report_type discussion_notes
# ... and summarize issues outside the changed lines as a single comment.
annotate_pr gitlab [...] --keep_issues outside_diff --report_as_one_comment
Filtering by severity¶
In this example, we configure annotate_pr (for GitHub) to:
report severities “error” and “high” as review comments (so that they are immediately visible and must be addressed before the PR can be merged), and
report all other severities as CI warnings.
To achieve this, use the JSON import format described earlier.
First, query all issues from the Dashboard (for simplicity, only style violations), then partition them into one set containing only “error” or “high” severities and another set containing all others.
Finally, call annotate_pr once for each set:
curl --header "Authorization: AxToken $DASHBOARD_API_TOKEN" "$DASHBOARD_URL/api/projects/$DASHBOARD_PROJECT_NAME/issues?kind=SV&columns=id,state,errorNumber,message,entity,path,line,provider,severity,justification,kind" >issues.json
jq '.rows |= map(select(.severity == "high" or .severity == "error"))' <issues.json >issues_high.json
jq '.rows |= map(select(.severity != "high" and .severity != "error"))' <issues.json >issues_low.json
annotate_pr github [...] --dashboard_issues_json issues_high.json --report_type comment
annotate_pr github [...] --dashboard_issues_json issues_low.json --report_type warning
Although jq is used here, you can use any tool or scripting language to create the filtered JSON files.