[Bug] Clicking Run Test To Execute Filtered Tests Executes All Tests Instead

Mar 11, 2025 by ADMIN 79 views

Brief Issue Summary

When filtering and executing tests in the VS Code test explorer using the CMake Tools extension, which discovers GTest tests, the extension (or VS Code?) executes all discovered tests instead of just the ones that are displayed after applying the filter. This issue is not specific to the CMake Tools extension, but rather a potential problem with VS Code itself.

Reproduction Steps

To reproduce this issue, follow these steps:

Create a new VS Code profile by going to "File" > "Preferences" > "Profiles" and clicking "New Profile..."
Install the Pre-Release of CMake Tools
Clone the utility repository from GitHub using the command git clone https://github.com/ungive/utility
Checkout the branch vscode-cmake-tools-bug using the command git checkout vscode-cmake-tools-bug
Update the submodules using the command git submodule update --init --recursive
Make sure you have a C++ compiler installed (e.g., MSVC with Visual Studio 17 2022 on Windows)
Configure and build the project in the CMake screen
Go to the test explorer and click "Refresh Tests"
Enter "Atomic." (with the period) in the filter text bar
Observe that tests are filtered
Click on the play button ("Run Test") under the "utility" group where you find 32 tests under
The loading icon shows it executes 45 tests (all tests)
The Output window contains test results for e.g. "counter_guard.DoesNotDecrementWhenIncrementFailed" which is clearly filtered out

Additional Information

The output window shows that the command executed is C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe with the option -R ^counter_guard\.ResultReturnsTheIncrementReturnValue$, which suggests that the issue is with the CMake Tools extension.

CMake Tools Diagnostics

The CMake Tools diagnostics provide information about the CMake configuration, including the version, generator, and compiler. The diagnostics are as follows:

{
  "os": "win32",
  "vscodeVersion": "1.99.0-insider",
  "cmtVersion": "1.21.4",
  "configurations": [
    {
      "folder": "w:\\dev\\tmp\\utility",
      "cmakeVersion": "3.28.0",
      "configured": true,
      "generator": "Visual Studio 17 2022",
      "usesPresets": false,
      "compilers": {}
    }
  ],
  "cpptoolsIntegration": {
    "isReady": false,
    "hasCodeModel": false,
    "activeBuildType": "",
    "buildTypesSeen": [],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 0,
    "executablesCount": 0,
    "librariesCount": 0,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": true
    }
  ]
}

Debug Log

The debug log provides a detailed record of the commands executed by the CMake Tools extension, including the ctest.exe command. The log is as follows:

[main] Building folder: w:/dev/tmp/utility/build 
[build] Starting build
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" --build w:/dev/tmp/utility/build --config Debug --target ALL_BUILD -j 16 --
[build] MSBuild-Version 17.9.8+b34f75857 für .NET Framework
[build] 
[build]   gmock_main.vcxproj -> W:\dev\tmp\utility\build\lib\Debug\gmock_main.lib
[build]   benchmark.vcxproj -> W:\dev\tmp\utility\build\third_party\benchmark\src\Debug\benchmark.lib
[build]   ungive_utility_test.vcxproj -> W:\dev\tmp\utility\build\Debug\ungive_utility_test.exe
[driver] Build completed: 00:00:00.600
[build] Build finished with exit code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.Example$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 1: Atomic.Example
[ctest] 1/1 Test #1: Atomic.Example ...................   Passed    0.09 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.10 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueDefaultConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 2: Atomic.ConstructWithValueDefaultConstructor
[ctest] 1/1 Test #2: Atomic.ConstructWithValueDefaultConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.01 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueMoveConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 3: Atomic.ConstructWithValueMoveConstructor
[ctest] 1/1 Test #3: Atomic.ConstructWithValueMoveConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.02 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueCopyConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 4: Atomic.ConstructWithValueCopyConstructor
[ctest] 1/1 Test #4: Atomic.ConstructWithValueCopyConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.02 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueInitConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 5: Atomic.ConstructWithValueInitConstructor
<br/>
**Q&A: [Bug] Clicking "Run Test" to execute filtered tests executes all tests instead**

**Q: What is the issue with the CMake Tools extension?**

A: The issue is that when filtering and executing tests in the VS Code test explorer using the CMake Tools extension, which discovers GTest tests, the extension (or VS Code?) executes all discovered tests instead of just the ones that are displayed after applying the filter.

**Q: Is this issue specific to the CMake Tools extension?**

A: No, this issue is not specific to the CMake Tools extension. It is a potential problem with VS Code itself.

**Q: How can I reproduce this issue?**

A: To reproduce this issue, follow these steps:

1. Create a new VS Code profile by going to "File" > "Preferences" > "Profiles" and clicking "New Profile..."
2. Install the Pre-Release of CMake Tools
3. Clone the utility repository from GitHub using the command `git clone https://github.com/ungive/utility`
4. Checkout the branch `vscode-cmake-tools-bug` using the command `git checkout vscode-cmake-tools-bug`
5. Update the submodules using the command `git submodule update --init --recursive`
6. Make sure you have a C++ compiler installed (e.g., MSVC with Visual Studio 17 2022 on Windows)
7. Configure and build the project in the CMake screen
8. Go to the test explorer and click "Refresh Tests"
9. Enter "Atomic." (with the period) in the filter text bar
10. Observe that tests are filtered
11. Click on the play button ("Run Test") under the "utility" group where you find 32 tests under
12. The loading icon shows it executes 45 tests (all tests)
13. The Output window contains test results for e.g. "counter_guard.DoesNotDecrementWhenIncrementFailed" which is clearly filtered out

**Q: What is the output of the CMake Tools diagnostics?**

A: The CMake Tools diagnostics provide information about the CMake configuration, including the version, generator, and compiler. The diagnostics are as follows:

```shell
{
  "os": "win32",
  "vscodeVersion": "1.99.0-insider",
  "cmtVersion": "1.21.4",
  "configurations": [
    {
      "folder": "w:\\dev\\tmp\\utility",
      "cmakeVersion": "3.28.0",
      "configured": true,
      "generator": "Visual Studio 17 2022",
      "usesPresets": false,
      "compilers": {}
    }
  ],
  "cpptoolsIntegration": {
    "isReady": false,
    "hasCodeModel": false,
    "activeBuildType": "",
    "buildTypesSeen": [],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 0,
    "executablesCount": 0,
    "librariesCount": 0,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": true
    }
  ]
}

Q: What is the output of the debug log?

A: The debug log provides a detailed record of the commands executed by the CMake Tools extension, including the ctest.exe command. The log is as follows:

[main] Building folder: w:/dev/tmp/utility/build 
[build] Starting build
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" --build w:/dev/tmp/utility/build --config Debug --target ALL_BUILD -j 16 --
[build] MSBuild-Version 17.9.8+b34f75857 für .NET Framework
[build] 
[build]   gmock_main.vcxproj -> W:\dev\tmp\utility\build\lib\Debug\gmock_main.lib
[build]   benchmark.vcxproj -> W:\dev\tmp\utility\build\third_party\benchmark\src\Debug\benchmark.lib
[build]   ungive_utility_test.vcxproj -> W:\dev\tmp\utility\build\Debug\ungive_utility_test.exe
[driver] Build completed: 00:00:00.600
[build] Build finished with exit code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.Example$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 1: Atomic.Example
[ctest] 1/1 Test #1: Atomic.Example ...................   Passed    0.09 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.10 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueDefaultConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 2: Atomic.ConstructWithValueDefaultConstructor
[ctest] 1/1 Test #2: Atomic.ConstructWithValueDefaultConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.01 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueMoveConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 3: Atomic.ConstructWithValueMoveConstructor
[ctest] 1/1 Test #3: Atomic.ConstructWithValueMoveConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.02 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueCopyConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 4: Atomic.ConstructWithValueCopyConstructor
[ctest] 1/1 Test #4: Atomic.ConstructWithValueCopyConstructor ...   Passed    0.01 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.02 sec
[ctest] CTest finished with return code 0
[proc] Executing command: "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\ctest.exe" -j16 -C Debug -T test --output-on-failure -R ^Atomic\.ConstructWithValueInitConstructor$
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project W:/dev/tmp/utility/build
[ctest] Cannot find file: W:/dev/tmp/utility/build/DartConfiguration.tcl
[ctest]     Start 5: Atomic.ConstructWithValueInitConstructor

**Q: What is the solution to this