It can be a bit tricky to get the Test Adapter for Catch2 running. So, if you are having trouble getting the test adapter to work you are not alone. For this Walkthrough I'm using the latest version of Microsoft Visual Studio Community 2017 (version 15.9.33 at the time of writing). This walkthrough makes use of the v1.6.0 release of the Test Adapter for Catch2. Note, that in Visual Studio 2019 the Test Explorer has undergone a facelift and things work slightly different. As such, there is an alternate walkthrough for use with Visual Studio 2019.
I will use the ReferenceTests created for testing the Test Adapter for Catch2. You can find a Visual Studio solution in the ReferenceTests folder of this GitHub repository. I will assume you know how to open the Test Explorer Window.
The following topics are discussed.
- Make sure the test adapter is installed
- Select a runsettings file
- Trigger test discovery
- Running tests
- Jump to TEST_CASE in source
- Examples of test case detail views
There are several ways to install the Test Adapter for Catch2, but the most typical way is using the Manage Extensions dialog in Visual Studio. Installing via this route typically schedules the extension for installation after you shutdown Visual Studio. It is possible you missed the VSIX installer dialog that then automatically pops up, effectively resulting in the extension not being installed. I have seen this happen more than once, so don't feel bad if it happens to you.
To make sure the test adapter is installed, open the Extensions and Updates dialog in Visual Studio and search for it in your installed extensions list.
For information on installing and managing extensions see Microsoft Docs: Find and use Visual Studio extensions
Out of the box the test adapter does not work. This is by design. Visual Studio provides a list of all the executables in your solution to the test adapter. As part of the discovery process these executables are called. A worst-case scenario would be that you have a project in your solution for an executable that when executed formats your C-drive. Note, the Test Adapter for Catch2 does not discover Catch2 tests inside dll-files.
After you have opened the ReferenceTests_VS2017.sln in Visual Studio 2017 and made sure the Test Explorer window is open, the first thing to do is to configure the run settings.
For this walkthrough we will use the Minimal.runsettings, ReferenceTests.runsettings and ReferenceTestsCombined.runsettings that you can find in the ReferenceTests walkthrough folder of this GitHub repository. Note that use is made of custom discovery in the ReferenceTests.runsettings and ReferenceTestsCombined.runsettings file.
You can load multiple .runsettings files and switch between them. The one with a checkmark next to it is the one that is being used. Note that clicking on a .runsettings file in the Configure Run Settings sub-menu that has a checkmark next to it, will deselect it.
For an overview of the Test Adapter for Catch2 specific settings that can be used in a .runsettings-file, see the Settings documentation.
When you have selected the Minimal.runsettings and then build the solution you will get output similar to the following.
[07/03/2021 07:57:42 Informational] ------ Discover test started ------
[07/03/2021 07:57:43 Informational] Started Catch2Adapter test discovery...
[07/03/2021 07:57:45 Informational] Discover log:
Error: Unable to reconstruct long test name
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe
Name: NotDefaultDiscoverable. a123456789b123456789c123456789d123456789e123456789&{???}f123456789b123456789g123456789d123456789h123456789&{???}i123456789b123456789j123456789k123456789l123456789
File: d:\github\testadapter_catch2\referencetests\src\tests\catch_discover\ut_notdefaultdiscoverable.cpp
Line: 29
Error: Unable to reconstruct long test name
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe
Name: NotDefaultDiscoverable. a123456789b123456789c123456789d123456789e123456789&{???}f123456789b123456789g123456789d123456789h123456789&{???}i123456789b123456789j123456789k123456789l123456789
File: d:\github\testadapter_catch2\referencetests\src\tests\catch_discover\ut_notdefaultdiscoverable.cpp
Line: 34
Error Occurred (exit code 255):
error: TEST_CASE( "SameTestNames. Duplicate" ) already defined.
First seen at d:\github\testadapter_catch2\referencetests\src\tests\catch_duplicates\ut_sametestnames.cpp(28)
Redefined at d:\github\testadapter_catch2\referencetests\src\tests\catch_duplicates\ut_sametestnames.cpp(42)
[07/03/2021 15:57:45 Informational] Finished Catch2Adapter test discovery.
[07/03/2021 15:57:45 Informational] ========== Discover test finished: 129 found (0:00:02,849301) ==========
Note that the Catch_Duplicates.exe contains tests with duplicate names, which is why those tests do not appear in the Test Explorer. The errors resulting from this are also logged. In addition some errors are logged for Catch_Discover.exe, which contains some testcases with names that cannot be handled correctly using the Minimal.runsettings, but can be handled correctly when ReferenceTests.runsettings is used. These represent some edge cases that you typically do not need to worry about. In case there are no errors the normal logging output would have looked something like the following.
[07/03/2021 16:07:09 Informational] ------ Discover test started ------
[07/03/2021 16:07:10 Informational] Started Catch2Adapter test discovery...
[07/03/2021 16:07:10 Informational] Finished Catch2Adapter test discovery.
[07/03/2021 16:07:10 Informational] ========== Discover test finished: 131 found (0:00:01,1961667) =========
This is where it can get a bit annoying. Changing or setting a .runsettings file will not trigger test discovery. Due to the way the Test Explorer caches information, even building one of the test projects or reloading a solution/project will not always trigger discovery when test have already been previously discovered. If you added new tests and they don't show up after the build, then there seem to be two workarounds that work. You can first switch the solution configuration (e.g., from Debug to Release) and then trigger a build. Or, you can clean the test executable (its test cases should disappear from the Test Explorer) and then build it again. This is of course a major annoyance. Here having a refresh button in the Test Explorer to trigger discovery would be really helpful.
There are many ways discovery can go wrong or provide you with unexpected results. Above you have already seen one way for the test project that contained test cases with duplicate names. But there are more ways, here are a few of the common ones.
- In case you do not have a .runsettings file selected you may get output similar to the following.
[07/03/2021 16:38:26 Informational] ------ Discover test started ------
[07/03/2021 16:38:26 Error] Catch2 Test Adapter Settings not found, Catch2 test discovery is cancelled. Add Catch2Adapter settings to runsettings-file.
[07/03/2021 16:38:26 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Dummy.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_NoSEH.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset01.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset02.exe D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset03.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:38:27 Informational] ========== Discover test finished: 0 found (0:00:00,8439976) ==========
- In case the Test Adapter for Catch2 is disabled you may get output similar to the following.
[07/03/2021 16:42:44 Informational] ------ Discover test started ------
[07/03/2021 16:42:44 Informational] Catch2Adapter is disabled.
[07/03/2021 16:42:44 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:42:44 Informational] ========== Discover test finished: 0 found (0:00:00,750804) ==========
- In case you provided an invalid
<DiscoverCommandLine>option in the .runsettings file you may get output similar to the following. See explanation of this Setting on how to resolve this problem.
[07/03/2021 16:46:17 Informational] ------ Discover test started ------
[07/03/2021 16:46:17 Error] Catch2 Test Adapter Settings contain an invalid discovery commandline. Catch2 test discovery is cancelled. Add Valid DiscoverCommandLine to Catch2Adapter Settings in runsettings-file.
[07/03/2021 16:46:17 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:46:17 Informational] ========== Discover test finished: 0 found (0:00:00,6799988) ==========
Alternatively if you made a typo you may get output similar to the following.
[07/03/2021 16:48:01 Informational] ------ Discover test started ------
[07/03/2021 16:48:02 Informational] Started Catch2Adapter test discovery...
[07/03/2021 16:48:02 Informational] Discover log:
Error Occurred (exit code 255):
Error(s) in input:
Unrecognised token: --disc
Run with -? for usage
[07/03/2021 16:48:02 Informational] Finished Catch2Adapter test discovery.
[07/03/2021 16:48:02 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:48:02 Informational] ========== Discover test finished: 0 found (0:00:00,7759798) ==========
- In case you did not provide the
<FilenameFilter>option in the .runsettings file you may get output similar to the following. See explanation of this Setting on how to resolve this problem.
[07/03/2021 16:53:00 Informational] ------ Discover test started ------
[07/03/2021 16:53:01 Error] Catch2 Test Adapter Settings contains an empty filename filter, Catch2 test discovery is cancelled. Add a valid FilenameFilter to Catch2Adapter Settings in runsettings-file.
[07/03/2021 16:53:01 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:53:01 Informational] ========== Discover test finished: 0 found (0:00:00,7039919) ==========
- In some cases, it is possible that test case discovery fails every now and then causing previously discovered tests to disappear from the Test Explorer. This may occur if you set the
<DiscoverTimeout>option to a relatively short time. In this case increasing this timeout may solve your problems. See explanation of this Setting for more information. Of course, setting this timeout to a very small value may also result in test case discovery to always fail. To help with this problem output similar to the following is produced when a timeout occurs.
[07/03/2021 16:58:21 Informational] ------ Discover test started ------
[07/03/2021 16:58:22 Informational] Started Catch2Adapter test discovery...
[07/03/2021 16:58:22 Informational] Discover log:
Warning: Discovery timeout for D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe
[07/03/2021 16:58:22 Informational] Finished Catch2Adapter test discovery.
[07/03/2021 16:58:22 Warning] No test is available in D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[07/03/2021 16:58:22 Informational] ========== Discover test finished: 0 found (0:00:01,2404445) ==========
- In some cases, other test adapters may interfere with Catch2 test discovery. In those cases, look in the
Testsoutput for hints. Worst case you may have to disable the test adapter that is causing trouble. For instance, if the output includesCould not locate debug symbols, that is probably output from the Boost Test Adapter, which can be disabled from the [Tools]->[Extensions and Updates...] menu item. By the way, this is the reason a feature was added to the Test Adapter for Catch2 to disable it via the .runsettings file. So in case the reverse happens and the Test Adapter for Catch2 interferes with another test adapter, you have an easy way to disable the Test Adapter for Catch2 via the .runsettings file.
By default Logging level is set to Normal. However, if you need more information you can change this setting to get more information. See explanation of this Setting to see all available log levels. Below example output for discovery when Logging is set to Verbose
[07/03/2021 17:07:17 Informational] ------ Discover test started ------
[07/03/2021 17:07:18 Informational] Started Catch2Adapter test discovery...
[07/03/2021 17:07:19 Informational] Discover log:
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Discover.exe
Testcase count: 26
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Dummy.exe
Testcase count: 1
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Hidden.exe
Testcase count: 6
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_NoSEH.exe
Testcase count: 3
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset01.exe
Testcase count: 6
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset02.exe
Testcase count: 45
Source: D:\GitHub\TestAdapter_Catch2\ReferenceTests\walkthrough\_unittest64\VS2017\Release\Catch_Testset03.exe
Testcase count: 44
[07/03/2021 17:07:19 Informational] Finished Catch2Adapter test discovery.
[07/03/2021 17:07:19 Informational] ========== Discover test finished: 131 found (0:00:01,4122137) ==========
You can of course run all tests using the Run All Tests In View-button in the Test Explorer. However, this may trigger a full solution build, which is not always something you may want to occur. My preference is to select the tests I want to run and use the Run-button to run the selected tests.
You have several option to start a debug session for a test. You can just to it the standard way, and set the project for the test executable as the default project. Configure the appropriate commandline and then start a debug session. But you can also start a debug session directly from the Test Explorer. Select the tests you want to debug (though typically you would select only one), and use the right-click context menu. When you start a debug session via the Test Explorer you can set your own break points and/or enable the DebugBreak feature to help with debugging your code.
The ReferenceTests.sln contains a test ('Catch_Dummy') that will run forever. If you run this test using the Minimal.runsettings test settings the test will run until you press the Cancel button in the Test Explorer.
If you run this test using the ReferenceTests.runsettings test settings the test will be cancelled automatically after 20 seconds. The length of this timeout is set via the <TestCaseTimeout> setting in the .runsettings file.
Note, the Test Explorer also has the option to set a timeout, however that one is a timeout for the test session, not for individual tests. For more information on the test session timeout see Configure unit tests by using a .runsettings file on Visual Studio Docs.
By default each test case is run in a separate instantiation of the test executable, i.e. the default execution mode is Single. Starting in version 1.6.0 of the Test Adapter for Catch2 it is also possible to execute multiple test cases in a single instantiation of the test executable, i.e., the Combine execution mode was added. The advantage of this is that the test results can be obtained faster as the overhead of instantiating the test executable is minimized. The execution mode can be set via the <ExecutionMode> setting in the .runsettings file.
If you run the tests using the ReferenceTestsCombined.runsettings test settings you should be able to notice the difference compared to running the tests with the ReferenceTests.runsettings test settings.
The nature of some tests may require them to be preferably executed in the Single execution mode. E.g., a test may be slow, or potentially run forever. To enable this you can use the <ExecutionModeForceSingleTagRgx> setting in the .runsettings file to configure Catch2 Tag-names that signal a test should always be run in Single execution mode.
The different execution modes require different timeout logic. E.g., you may want tests that are run together in the Combine execution mode to timeout after 10s (i.e., all tests should be finished within 10s). However, tests that are run in Single execution mode may only timeout after 30s, because they are tests that are known to take a long time for example. You can set the different timeout settings using the <CombinedTimeout> and <TestCaseTimeout> setting in the .runsettings file.
To finish this walkthrough some screenshots of the detailed view for various test cases. Note the assertion statistics are displayed as part of the message only when the test was run in the Single execution mode. This is done for both Failed and Successful tests.
The details view will typically provide a link to the source of the test case in the detailed view of the test case. Clicking the link will bring focus to the source file and place the cursor on the line of the TEST_CASE. This link is available regardless of whether the test was run or not, as such it can be used to quickly navigate to a specific test case.
Note the 2 StackTrace items at the bottom. They provide information about the failure, and also act as links to the source of the failure. Note that only the assertion statistics are displayed as part of the message. Additional information can be reached via the Output link above the StackTrace section. If you prefer to have the additional information be part of the message adjust the StackTraceFormat option in the .runsettings file.
Note, only the additional information provides information about sections and info messages that may help diagnose the reason for the failure. In this example the root section is named "CHECK REQUIRE" and the nested section with the failed test assertion is named "CHECK_FALSE REQUIRE_FALSE".
Note, in case test failures occur any additional information will be prepended to the Standard Output section.
Note, some additional information, with regard to the number of warnings generated, is added to the message. See the additional information for the actual warnings.
