Coco Test Engine Driver
cocotestengine program 'drives' a target program to generate test data and to run tests based on this data. If during the test execution or discovery, the target program crashes,
cocotestengine restarts it and continues.
cocotestengine works in three different modes:
- Discovery mode: This mode is enabled with the
In discovery mode,
cocotestenginetries to find new tests. If there exists a data file with test data, it uses them as a starting point.
If a newly found row of data is better (covers more code) than an existing one, and the existing one is not
validated, then the existing row could be thrown out to keep the data set minimal.
- Test mode: This mode is enabled when neither
In test mode,
cocotestenginereads the test data from the data file, runs them, and reports how many were successful and which failed.
In test mode, it is possible to run the tests that are implemented in a target program, or only a single test.
- Learn mode: This mode is enabled with the
In learn mode, the driver runs all the rows of a test, but instead of checking whether the outputs agree with the values in the data file, it takes the outputs of its test run and stores them in the data file.
For rows that were previously
validated, if the output value is different, then the
validatedflag is reset to false for that row.
Learn mode is possible for one test or all tests in a target program.
cocotestengine -e <target> [-d <directory>] [-t <test>] [--learn | --discover] [options] cocotestengine -e <target> --list cocotestengine -h
|64||Invalid command line arguments|
|65||Invalid data file|
|66||A test failed|
|67||A test crashed|
|68||A feature was not implemented|
|69||An error occured in the target|
In case of a crashed and a failed test together, the crash takes precedence.
At the end of each test, learn, or discovery run, the target program writes a
.csexe file. If the file already exists, the new data is appended to it.
If the discovery process takes too long, one can press
Ctrl-C. The driver then stops discovery and writes the data that it has found to the data file.
--help: Print a help message and exit.
-e<path>: Set the path to the target executable.
--cs-discoveris set, the driver runs all rows for all tests that are supported by the target program, then prints a summary of the results.
--data-dir=<path>: Set the path to the directory that contains the data and configuration files.
The default value is the working directory of the driver.
-t<string>: Set the name of the test that should be executed.
--cs-discoveris set, the driver runs all rows for this test and prints a summary of the results.
--list: Print a list of all the tests that are supported by the target program.
--verbose: Enables verbose mode.
--learn: Enable learn mode.
-tmust be set. The driver then runs all rows for this test and stores for each row the output values of the test in the data file.
--discover: Enable discovery mode.
-tmust be set. The driver then runs test data discovery for this test.
The following options are only meaningful in the discovery mode.
--max-tries=<int>: Maximal number of rows tried during test generation.
The default value is 10000.
--max-time=<int>: Maximal time for discovery.
A value of 0 means unlimited time. Every other value sets a timeout in seconds, after which discovery stops. The default value is 0.
--max-coverage=<float>: Coverage at which discovery stops.
If the value is positive and the coverage of the test that were already reached exceeds that value, discovery is stopped. A negative value disables this check. The default is -1.
--update=<int>: Delay in seconds between status updates.
If the value is positive, it specifies the time period at which the driver prints a one-line summary about the results of the discovery that were reached so far. A value of 0 disables the feature.
The summary contains the number of data rows that were found so far and the coverage they achieve together.
--save-interval=<int>: Interval in seconds between saving the discovered tests.
The default value is 60. A value of 0 disables saving.
The following options are rarely needed.
--timeout=<int>: Set a timeout for communication with the target.
The value is measured in milliseconds, and the default is 300000, which is 5 minutes. If the driver communicates with the target and waits for an answer for longer than this time, then the target finishes with a failure.
--batch-sizeSet the number of rows that the driver sends in one batch to the target.
The default is 100. The number has some effect on the efficiency of the data transfer between driver and target.
Coco v7.0.0 ©2023 The Qt Company Ltd.
Qt and respective logos are trademarks of The Qt Company Ltd. in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.