QML coverage

QML is a user interface markup language that is used in the Qt Framework. Coco has an add-on that supports code coverage for QML.

The add-on is delivered as a separate package. It consists of two parts, a program to instrument the QML code and a QML plugin that is needed at runtime.

The instrumentation program is called cocoqmlscanner and it inserts code into the QML sources of a program. The plugin implements a QML object that at runtime collects the coverage data and writes them to a file. The plugin is delivered in source form and must be compiled by the customer.

Setup

The CocoQML package is a ZIP archive. When unpacked, it generates a directory with two subdirectories:

• bin: The directory with the cocoqmlscanner executable and some libraries that are needed by it.
• trackerplugin: The source code of the tracker plugin.

Both directories are independent of each other and it is possible to copy them to different places in a test installation.

Compilation of the plugin

The plugin is a qmake project. It must be compiled for the same Qt version as the application for which the coverage should be measured. We recommend that you make it part of your project and include it in the repository, so that it is automatically compiled correctly.

Under UNIX^®, the compilation looks like this:

$ /Qt/5.11.3/gcc_64/bin/qmake ../trackerplugin
Info: creating stash file /home/user/trackerplugin-build/.qmake.stash
$ make
g++ -c -pipe -O2 -std=gnu++11 -D_REENTRANT -Wall -W -fPIC -DQT_NO_DEBUG -DQT_PLUGIN -DQT_GUI_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -I../trackerplugin -I. -I/Qt/5.11.3/gcc_64/include -I/Qt/5.11.3/gcc_64/include/QtGui -I/Qt/5.11.3/gcc_64/include/QtQml -I/Qt/5.11.3/gcc_64/include/QtNetwork -I/Qt/5.11.3/gcc_64/include/QtCore -I. -isystem /usr/include/libdrm -I/Qt/5.11.3/gcc_64/mkspecs/linux-g++ -o csexeapi.o ../trackerplugin/csexeapi.cpp
[...]
mv -f libcocoqmltracker.so QmlJsCoverage/libcocoqmltracker.so
cp -f /home/user/trackerplugin/qmldir QmlJsCoverage

When compiled this way, the build directory contains a subdirectory QmlJsCoverage which contains the plugin:

$ ls QmlJsCoverage/
libcocoqmltracker.so  qmldir

The Microsoft^® Windows build is similar. The plugin directory then contains a DLL file instead of a .so file.

Use of CocoQML

Instrumentation

To measure QML coverage in a software project, its QML files must be instrumented. This is done by the cocoqmlscanner command, in the form

$ cocoqmlscanner <project directory>

All QML files in subdirectories of <project directory> are then (reversibly) instrumented and a program database (or .csmes file) is written. By default, the program database file is cocoqmlscanner_result.csmes.

After the QML files are instrumented, it is recommended to recompile the project. This is especially important in two cases:

• The QML files are resources of the program (i.e. specified in a .qrc file).
• The QML compiler is used. (This is the default for release builds that are compiled with Qt Creator).

In both cases, the compilation makes the instrumented files part of the executable.

Measuring the coverage

After instrumentation, an application needs the tracker plugin to run. To find the plugin, it uses the environment variable QML2_IMPORT_PATH. This variable must be set and its value must be the parent directory of QmlJsCoverage/. (This means that the string QmlJsCoverage does not occur at the end of the path in QML2_IMPORT_PATH.)

If this variable is set correctly, the instrumented AUT can run.

Debugging

If something goes wrong, it is useful to set the variable COCOQML_VERBOSE to 1 which allows the plugin to log its activity.

Reference

cocoqmlscanner

Syntax:

cocoqmlscanner [<options>] <paths>...

Options:

• <paths>: are the space-separated paths to the files and directories that will be instrumented or restored.

  The action depends on whether the option -r is set.

  If -r is not set, the <paths> are scanned recursively and each file that ends with .qml or .js is read and instrumented. A backup copy of each file is kept; it has the suffix .qmlbak or .jsbak, respectively. If there are already backup copies, nothing is done and the program returns with an error.

  If -r is set, all the backup files are renamed back to their old names, replacing the instrumented files. If there are no backup copies, the program does nothing and returns with an error.

• -r | --restore: Enable the restore mode.
• -c | --csmes-name=<filename>: Save the resulting .csmes file <filename>. A suffix .csmes is appended if it does not exist already.

  Without this option, the resulting data is saved as cocoqmlscanner_result.csmes.

• -b | --blacklist=<filename>: Exclude certain paths from instrumentation. The <filename> is the path to a text file which contains in each line a regular expression. If one of the regular expressions matches a file name, the file is excluded from the instrumentation.

  Lines beginning with a # are ignored.

• -n | --no-tracker: Do not insert an import statement for the tracker plugin into the instrumented code.

  With this option, cocoqmlscanner behaves the same way as it did before the tracker plugin was introduced. The option was introduced only for compatibility with the previous version of cocoqmlscanner and is rarely, if ever needed. In particular, it will break the tracker plugin functionality.

• -l | --logfile=<logfile>: If this variable is set, cocoqmlscanner writes its log messages to the given logfile.
• --quiet: Disables all messages generated by cocoqmlscanner. This setting overrides the verbose mode.
• --verbose: Enables verbose mode. This setting can be overridden by the quiet mode.
• -v | --version: Print the version number of the program and finish.
• -h | --help: Print a help message and finish.

The tracker plugin

The tracker plugin is provided in source form because the Qt version with which it is compiled must be the same as the Qt version of the QML application.

The following variables control various debug output generated by the tracker. All log messages are prefixed with CocoQML:.

• COCOQML_OUTPUT: Set this variable to determine the directory and/or base name of the output .csexe file. The value of COCOQML_OUTPUT must be a relative or absolute path to an existing directory, a base name or a combination of both. If only a directory is given, the .csexe file is created in the directory with the default base name. If a base name is given, the output file will be <base name>.csexe. The default base name is usually <application name>_qml.

  For example, you can set COCOQML_OUTPUT=/home/user/my_run and the tracker plugin will write the output to /home/user/my_run.csexe.

  If this variable is unset or empty, a file <application name>_qml.csexe is created in the current working directory.

• COCOQML_QUIET: Set this variable to 1 to disable all messages generated by the plugin. Usually, all messages will be printed to standard error, but if you rely on parsing standard error in your application, you can use this to avoid interference.

  This setting overrides the verbose mode, see below.

• COCOQML_VERBOSE: Set this variable to 1 to enable verbose mode. If the verbose mode is active, the CocoQML extension will print additional messages that might be helpful for debugging purposes. Usually, you will not need to enable verbose mode.

  This setting can be overridden by quiet mode, see above.

• COCOQML_LOGFILE: Set this variable to write the CocoQML plugin log messages to a file. The value of COCOQML_LOGFILE must be an absolute or relative file path. A relative file path is interpreted relative to the working directory of the application when it starts. If the log file already exists, the new data is appended to it. If the path contains directories, they must already exist, otherwise no log file will be created.

  For example, under Unix, you can set COCOQML_LOGFILE=/tmp/cocoqml.log. Since the directory /tmp/ exists in any normal Unix installation and is writable, all log messages are then written to the file /tmp/cocoqml.log