Migration from specflow: Azure pipeline discovered test but incorrect number run

[cgiovenco]

Hello all, I recently migrated one of our specflow frameworks to reqnroll 2.2.1 using dotnet6. I got everything working as expected but when running the reqnroll framework via our existing azure pipeline I'm having issues and I don't know where to start troubleshooting this. When searching the web I mostly find issues related to not discovering test but I'm not having that problem. When executing everything is working as expected with building and executing but the issue is that the pipeline discovered number of test is the expected amount of test but the actual number of test being executed is about half of the discovered. For example, 525 test discovered and only 238 are now being executed. The pipeline was running all 525 test prior to converting to reqnroll so I'm unsure what would be causing the pipeline to be running less test than what was discovered if the only change was the framework migration. Any insight would be great.

[cgiovenco]

I reverted my code back to specflow and ran the same pipeline and all 525 test were picked up as expected. So still unsure what inside the rnr framework would be causing test from being discovered vs executed.

[cgiovenco]

I don't understand what you mean by running locally. Locally via the tag I'm targeting in the pipeline visual studio test explorer identifies all 525 test. There's no .net version identified in the pipeline it's using the same yaml specflow/reqnroll. No change to the pipeline was made between just swapping from the framework upgraded from specflow to reqnroll.

[melchiork]

I assume the pipeline runs a dotnet test command, when you run the same command locally is the result identical to the pipeline?

[cgiovenco]

had some time to circle around to my reqnroll branch. Tried with both dotnet test and vstest commands both identfied all 525 test with the same TestCategory that only picks up 238 in the pipeline. Still not sure where to go from here.

[melchiork]

Can you see what exact command is run in the pipeline? If you can and then run the same one locally you could see if the problem is in the pipeline command.

[cgiovenco]

I've tried running the pipelines in debug and I don't really see any commands that can be run locally for the yml vstest task that would be different than what I possibly tried locally. Spent most the day trying different things such as removing the TestCategory filter completely from pipeline and while the number of test did go up it still wasn't a 1:1 ratio of discovered to executed. I'm stumped.

[MacostaHitachi]

I had similar issue, and it was due to examples being translated into attributes, the rest of the tests are probably embedded within the 238 tests.
It also doesn’t work properly when Associating the test case.
You can switch to the old behavior (1 function per example)
By adding this to your reqnroll.json

"generator": {
      "allowRowTests": false
}

[cgiovenco]

Tried adding this but still had the same result via pipeline.

[tulavalle]

@cgiovenco

I'm having trouble running nunit via pipeline. I use Selenium Grid and we run the tests in an Azure pipeline in a Linux environment. It identifies all the tests, but when the first failure occurs, the execution is interrupted and the result is always success because it uses the number of tests that passed before the first failure as the total number of tests, always resulting in 100% success. I've already configured the pipeline task to continue even with an error, and I've included a .runtime file to continue if a test fails. In short, it works perfectly on Windows, but running the tests in nunit from a Linux machine seems to have some problem or require some additional configuration that I haven't identified yet.

For now, I've stopped using Nunit and returned to Xunit, which works both locally and via pipeline (Windows/Linux).

Has anyone else experienced this using NUnit in parallelism via Fixture or Children?

latest version of my csproj to test execution with nunit:

hooks.cs:

[tazambuja]

@cgiovenco , your tests run in nunit or xunit? If it is in nunit, could it not have some relation to the problem I reported? Does it use the same dependencies? In my case, it finds all the tests. The problem is that it stops the execution at the first failure and generates a result totaling 100% because it only considers the success scenarios before the first failure. I thought about the possibility of it being a bug in Reqnroll.NUnit...

[cgiovenco]

My test run in mstest

[MacostaHitachi]

can you check what tests are missing.
maybe try find common attributes/pattern for the ones missing like are they using Data attributes, do they have special characters like dashes.

try running this
vstest.console.exe bin\x64\Release\DummyProjectTests.dll /ListFullyQualifiedTests /ListTestsTargetPath:tests.txt
and see what comes up. is it the count you expect or is it less.. I still suspect your issue is with FullyQualifiedName of your tests

[gasparnagy]

@cgiovenco But could you see the impact of the config setting? I mean in the "Tests" tab of the pipeline, do you still see the scenario outlines in hierarchy?

[cgiovenco]

@gasparnagy no there wasn't any change after modifying the reqnroll.json in the pipeline test results. The additional examples are still nesting under 1 'test result'.

[MacostaHitachi]

is it fixed locally?, like when you get the fullyqualifiablenames are they coming back as they were in specflow.

also I had a typo on the JSON above at first (i edited after). Is possible that you copied a broken JSON

[cgiovenco]

@MacostaHitachi I have your most recent changes. the fullyquantifiablenames doesn't seem to show my scenario outlines in reqnroll but works fine in specflow.

specflow returns 633 line items
reqnroll returns 331 line items

[gasparnagy]

@cgiovenco you somehow use the reqnroll.json config file wrong. It has to be in the project root folder. You need to figure out first why that config file is not used in your setup.