6.4.8. Generating review comments with annotate_pr

Note

Starting from 7.11.0, you can also annotate pull requests as part of the Continuous Integration by adding an AnnotatePR rule in PostCIActions. The documentation below assumes that you use annotate_pr as a script, but the options are in essence the same across the two ways of using it.

Note that, while the usage as a rule is usually easier and preferable (e.g., due to GUI configuration and integration into the existing axivion_ci run), there are some cases where the command-line version might be needed. For example, under some configurations (such as the warning and error report types under GitHub) annotate_pr needs to be run on that Git platform’s CI framework for the comments to be picked up correctly, while you might want to run Axivion’s general continuous integration on some other platform.

This script is intended to be used as part of a CI/CD pipeline. It generates reports based on the issues found by the Axivion Suite and posts them to one of the supported platforms (see below). Reports are either posted as comments to a pull request (if --report_as_one_comment is used) or as inline code review annotations. Note that parameters can also be passed via environment variables, but command line options always take precedence.

Use annotate_pr $PLATFORM --help to view all available options for each platform, where $PLATFORM is one of the platforms listed below.

Supported platforms:

  • Bitbucket Cloud (bitbucket_cloud)

  • Bitbucket Server (bitbucket_server)

  • Gerrit (gerrit)

  • Gitea (gitea)

  • GitHub Free / Pro / Team / Enterprise Cloud (github)

  • GitHub Enterprise Server (github with the --enterprise_server option)

  • GitLab (gitlab)

It is also possible to pass dry_run as a platform. Rather than posting the detected issues to any platform, this will simply output them directly.

The Azure DevOps platform is not supported by annotate_pr, but similar functionality is provided by the Azure Devops Integration.

6.4.8.1. Example

Note

In all examples, we assume that the issues are retrieved from the Dashboard, which is authenticated with environment variables that we do not specify here (see Retrieving the issues for details). Additionally, we use HelloWorld as the name of the Dashboard project.

For example, you may want to post issues to a Bitbucket Cloud pull request. Issues are by default from the most recent version of the Dashboard project compared to the previous one. For Bitbucket, issues are marked as inline report annotations. To do this for pull request 42 at Bitbucket project foo/bar, you would use:

annotate_pr bitbucket_cloud --dashboard_project_name HelloWorld --server https://api.bitbucket.org/2.0 --token my-access-token --repository foo/bar --pull_request 42

Additionally, you may want to include issues added in earlier revisions. The command would look like this (substituting [...] for the options from the first example):

annotate_pr bitbucket_cloud [...] --start_revision 0

Alternatively, you can also includes issues that are not part of the changes in the pull request. In that case, it can also make sense to post these as a single comment, rather than as individual annotations. Otherwise, the script may fail with an error, as it cannot leave annotations outside of the changed lines. Put together, the command would be:

annotate_pr [...] --keep_issues all --report_as_one_comment

As a final use case, you might want to report all issues that are inside the diff as annotations, and all issues outside the diff as comments (since annotations cannot be left outside the diff). To do this, you can run annotate_pr twice: Once only on the issues inside the diff, and once only on the issues outside the diff:

annotate_pr bitbucket_cloud [...] --keep_issues inside_diff --report_type ANNOTATIONS
annotate_pr bitbucket_cloud [...] --keep_issues outside_diff --report_type COMMENTS

For full details on which configuration options exist, refer to annotate_pr $PLATFORM --help.

6.4.8.2. Retrieving the issues

There are four possible ways to retrieve issues from the Axivion Suite: You can either use the Dashboard API, a database file (.db), an AR file (.ar), or a JSON file containing an issue table retrieved from the Dashboard API (.json).

Retrieving from the Dashboard

To access the Dashboard, one of the following needs to be set (in addition to --dashboard_url and --dashboard_project_name):[*]

  • --dashboard_api_token: Uses an authentication token for the Dashboard API.

  • --dashboard_db_token: Uses a database uid string as credentials to access the project associated with the uid. Can be obtained by executing cidbman metadata list path/to/project.db.

  • --dashboard_username and --dashboard_password: Uses a username/password combination for the Dashboard.

Retrieving from a database file

As an alternative to accessing the Dashboard, you may also pass in a database containing results directly. You can do this by passing the path to a .db file using the --analysis_results_database option.

Retrieving from an AR file

Similar to the above, you can pass in a binary AR file generated by axivion_analysis (with --output_binary) directly, using annotate_pr --analysis_results_binary results.ar, where results.ar is the AR file containing analysis findings. This option does not require a Dashboard to be part of the workflow at all.

Caveat If you load issues from a database file or from an AR file, you may see findings in the git platform that are semantically unrelated to the current pull request. This can happen if an older, but still existing, issue has the same source line as a current issue. The reason for this behavior is that annotate_pr checks for every issue loaded from the files whether its source location lies within the git changeset for the current PR. For example, let us assume that the following code snippet is in the current git main branch:

int function_name(int X, int Y) {
   return X + Y;
}

Let us assume that the configured rule set specifies a rule which checks that every function parameters starts with a lower case letter. In this case, the analysis for the current main branch will produce a finding for both parameters X and Y, as they are uppercase letters. If a developer now creates a pull request in which the first parameter is changed from X to x, the AR or database file will still contain a finding for the second parameter Y. As both parameters are in the same source line, the Y finding will be propagated to the git platform, although it was not introduced in this pull request.

Retrieving from a JSON file

There are some use cases where you might want to use data from the Dashboard, but either do not want annotate_pr to access it, or want to modify/filter it before passing it to annotate_pr (for example, see Filtering). For such cases, you can use the --dashboard_issues_json parameter, which takes a JSON file containing a single IssueTable retrieved from the Dashboard API. You can pass this parameter multiple times to include multiple IssueTables.

A minor limitation of this approach is that, as of 7.10.0, the returned IssueTable does not contain information on the kind (style violation, dead code, etc.) of the included issues by default. Hence, you need to include the additional kind column when querying the Dashboard. [†] To do this, use the columns query parameter (see Issue Lists API) to explicitly specify the kind column. Note that you will also need to specify every other column you wish to include—as soon as one column is explicitly specified, the set of default columns is no longer included (see Issue Table Columns for an overview of such columns). We will amend the Dashboard API in a future version to include the kind column by default.

As an example, to retrieve the style violations this way and pass them to annotate_pr:

curl --header "Authorization: AxToken $DASHBOARD_API_TOKEN" "$DASHBOARD_URL/api/projects/$DASHBOARD_PROJECT_NAME/issues?kind=SV&columns=id,state,errorNumber,message,entity,path,line,provider,severity,justification,kind" >issues.json
annotate_pr [...] --dashboard_issues_json issues.json

6.4.8.3. Filtering

In some circumstances, you might want to run annotate_pr with different parameters depending on the handled issues. For example, if you are using GitHub, issues with a high severity could be posted as the “comment” report type, while issues with a lower severity could just be reported as warnings. Another common use case is to handle issues inside the changed lines of a given pull request a certain way, and to handle issues outside those lines in a different way (for example, some platforms only allow comments to be left on lines inside the diff, making it necessary to report other issues differently).

The solution to these scenarios is to simply call annotate_pr multiple times, filtering the handled issues each time. That way, you can specify different parameters for each of the subsets of issues. We will give concrete explanations on how to implement the two common example scenarios mentioned above.

Filtering inside/outside changed lines

As mentioned before, some platforms do not allow certain report types to be used for issues that are outside the changed lines of the pull request. For this example, we will assume GitLab is being used.

GitLab only allows notes (comments) to be posted for locations inside the diff, so if we want to report issues outside the diff as well, we need to either use the code_quality report type for those, or use the --report_as_one_comment flag (which will collect those issues and summarize them in a single comment posted to the merge request). Here, we choose the latter option, which looks like this:

# Post issues inside the changed lines as notes...
annotate_pr gitlab [...] --keep_issues inside_diff --report_type notes
# ... and summarize issues outside the changed lines as a single comment.
annotate_pr gitlab [...] --keep_issues outside_diff --report_as_one_comment

Filtering by severity

In this example, we want annotate_pr (in a setting where we use GitHub) to:

  • report severities “error” and “high” as review comments (so that they are immediately visible and need to be addressed before the PR can be merged), and

  • report all other severities as CI warnings.

To do this, we can use the JSON import format introduced earlier. We will first query all issues (for simplicity’s sake, only style violations) from the Dashboard, then partition the issues into one set with only “error”/”high” severities, and one with all others. Finally, we will call annotate_pr once for each set:

curl --header "Authorization: AxToken $DASHBOARD_API_TOKEN" "$DASHBOARD_URL/api/projects/$DASHBOARD_PROJECT_NAME/issues?kind=SV&columns=id,state,errorNumber,message,entity,path,line,provider,severity,justification,kind" >issues.json
jq '.rows |= map(select(.severity == "high" or .severity == "error"))' <issues.json >issues_high.json
jq '.rows |= map(select(.severity != "high" and .severity != "error"))' <issues.json >issues_low.json
annotate_pr github [...] --dashboard_issues_json issues_high.json --report_type comment
annotate_pr github [...] --dashboard_issues_json issues_low.json --report_type warning

Note that, while we use jq here, you can of course use any tool or scripting language you like to create the filtered JSON files.