Testing

To set preferences for handling tests, go to Preferences > Testing > General.

{General tab in Testing preferences}

The following table summarizes the general testing preferences.

SettingValue
Scan threadsSet the amount of worker threads used when scanning for tests. By default, Qt Creator uses a quarter of the available logical CPUs.
Omit internal messagesOmit internal messages.
Omit run configuration warningsOmit run configuration warnings for deduced configurations.
Limit result outputBy default, test result output shows a maximum of 100,000 characters.
Limit result descriptionSet the maximum number of lines in the test result tooltip and description.
Open results when tests startShow test results automatically when testing starts.
Open results when tests finishShow test results automatically when testing ends.
Open results only for unsuccesfull test runsShow test results only if thy contain failed, fatal, or unexpectedly passed tests.
Automatically scroll resultsThe view automatically scrolls to show the latest results.
Group results by applicationGroup test results by the executable path that you use to run the tests. This is useful if you have multiple test executables and run them all at once.
Process argumentsPass arguments to the test executable that you specify in the run configuration. This is an experimental feature that might cause the execution of the test executable to fail.
Automatically runAutomatically run all or selected tests after successfully building the current project.
TimeoutSet a maximum time in seconds to execute a test case.
Reset Cached ChoicesSometimes, Qt Creator cannot deduce which executable or run configuration to use. If it repeatedly asks you to select the tests to run when trying to execute tests, you can let it cache your choices and use them where appropriate. Qt Creator clears the cache when you switch to another project, close the current one, or select this setting.
Active Test FrameworksSelect tests for Qt Creator to handle. To improve the performance of full scans for tests, disable test frameworks you do not use.
GroupGroup related test cases for an active test framework. By default, Qt Creator groups tests that are in the same directory.

To override some preferences for the current project, go to Projects > Project Settings > Testing.

Qt Tests

To set preference for Qt tests, go to Preferences > Testing > Qt Test.

{Qt Tests tab in Testing preferences}

The code inside a benchmark test is measured, and possibly also repeated several times in order to get an accurate measurement. This depends on the measurement back-end that you select in Benchmark Metrics:

  • Walltime
  • CPU tick counter
  • Event counter
  • Valgrind Callgrind
  • Linux Perf

For more information, see Creating a Benchmark.

The following table summarizes the other Qt test preferences.

SettingValue
Disable crash handler while debuggingAllow the debugger to interrupt Qt tests on assertions.
Use XML outputXML output avoids parsing issues, but plain text is more human readable.

Note: Plain text misses some information, such as duration.

Verbose benchmarksReceive verbose output when running benchmarks.
Log signals and slotsRecord information about signals and slots in the test log.
Limit warningsExplicitly limit the maximum number of warnings in the test log. Set the limit to 0 if you want no limit at all. The default limit is 2000.
Check for derived Qt Quick testsCheck for Qt Quick Tests that are derived from TestCase.

Note: This feature is rather expensive and increases the scan time significantly.

Find user-defined locationsParse messages for the following pattern and use it as location information: file://filepath:line

where ":line" is optional.

Note: If the patterns are used in code, the location information for debug messages and other messages might improve, at the risk of some incorrect locations and lower performance.

Google Tests

To specify settings for running Google tests, go to Preferences > Testing > Google Test.

{Gooble Test tab in Testing preferences}

The following table summarizes the Google test preferences.

SettingValue
Run disabled testsExecute disabled tests during a test run.
Throw on failureTurn assertion failures into C++ exceptions.
Break on failure while debuggingTurn failures into debugger breakpoints.
Repeat testsRun several iterations of the tests.
IterationsThe number of times to run the tests.
Shuffle testsMake sure that the tests are independent and repeatable by running them in a different order each time.
SeedThe seed for initializing the randomizer. The value 0 generates a seed based on the current timestamp.
Group modeGroup Google tests by using a GTest Filter.
Active filterThe GTest filter to use.

For more information about GTest filters, see Running a Subset of the Tests.

Boost Tests

To specify settings for running Boost tests, go to Preferences > Testing > Boost Test.

{Boost Test tab in Testing preferences}

The following table summarizes the Boost test preferences.

SettingValue
Log formatThe type of events to record in the test report.
Report levelThe verbosity level of the test result report. Select No if you do not want a report.
RandomizeExecute the tests in a random order.
SeedThe seed for initializing the randomizer. The value 0 means no randomization, the value 1 uses the current time, and any other value generates a random seed.
Catch system errorsCatch or ignore system errors.
Floating point exceptionsDetect floating point exception traps.
Detect memory leaksDetect memory leaks.

Catch2 Tests

To specify settings for running Catch2 tests, go to Preferences > Testing > Catch Test.

{Catch Test tab in Testing preferences}

The following table summarizes the Catch2 test preferences.

SettingValue
Show successShow succeeding expressions as well. By default Catch2 prints only failed expressions.
Break on failure while debuggingTurn failures into debugger breakpoints.
Skip throwing assertionsSkip assertions that test for thrown exceptions.
Visualize whitespaceTurn whitespace into escape sequences.
Abort afterAbort the test after the number of failures specified inside the box.
Benchmark samplesThe number of samples to collect while running benchmarks.
Benchmark resamplesThe number of resamples to use for statistical bootstrapping after benchmarking.
Benchmark confidence intervalThe confidence interval for statistical bootstrapping.
Benchmark warmup timeThe warmup time for each test before benchmarking starts.
Disable analysisDisable statistical analysis and bootstrapping.

CTest-Based Tests

To specify settings for running CTest-based tests, go to Preferences > Testing > CTest.

{CTest tab in Testing preferences}

The following table summarizes the CTest preferences.

SettingValue
Output on failureShow test specific output if a test fails. Contrary to the CTest default this setting is on by default.
Schedule randomExecute the tests in random order.
Stop on failureAutomatically stop the test execution on the first failing test.
Output mode,The verbosity level of the CTest output.

Note: This setting only influences the output on the text display.

Repeat testsRe-run tests, as determined by Repetition mode. Set the maximum number of times for repeating a test in Count.
Run in ParallelRun the tests in parallel using the specified number of Jobs. In Test load, limit the parallel execution. CTest will not start a new test if it would cause the CPU load to pass the threshold set in Threshold.

See also How To: Test and Test Results.

© 2024 The Qt Company Ltd. Documentation contributions included herein are the copyrights of their respective owners. The documentation provided herein is licensed under the terms of the GNU Free Documentation License version 1.3 as published by the Free Software Foundation. Qt and respective logos are trademarks of The Qt Company Ltd in Finland and/or other countries worldwide. All other trademarks are property of their respective owners.