Call for action: Share usage patterns of Before/After Feature hook and FeatureContext

[gasparnagy]

Through the last year we had many discussions about the concept of FeatureContext and the way how before/after feature hooks work. With the broader introduction of scenario (method) level parallel execution, the question became even more important.

The behavior of the feature context and the related hooks have been inherited from SpecFlow, but it has turned out that the behavior is not obvious and leads to many misunderstandings. Also this concept is considered to be a problematic by many BDD practitioners - in other Cucumber implementations there is no such concept.

Also make sense to note that keeping the concept of FeatureContext increases the complexity of Reqnroll quite a lot and causing ongoing efforts.

Because of all these I thought it would make sense to kick-off a discussion for the future of this concept.

However through the preliminary discussions I have realized that many of us, including myself do not use this concept anyway so we are discussing based on "imagined usage scenarios" that is very dangerous. (I have checked my codes back to 10 years and haven't found any usage of the feature context or the related hooks.)

So here is a call for action: 📢 Please briefly describe your usage of the FeatureContext and the before/after feature hooks here as a comment. Please make sure you describe only real usage scenarios, not wishes or imagined situations (there will be a round when we collect those, but not now).

We are interested in why you use these features, what actions you perform in before/after hooks and what do you store in the feature context. Every usage should be a new top-level comment here and we might ask back for clarifications in the sub-comment of these.

[ajeckmans]

This codebase does not exist anymore, I think. But we used it once to enable or disable messages within an event sourced system by adding tags to the feature and reading those in the hooks.

Mainly as a switch to speed up execution of scenarios that did not depend on the messages.

Though technically we could implement that on the scenario level hooks as well, it was easier to deal with at the feature level.

So there was a use case, but if the feature was to be removed or would not prohibit implementation.

[melchiork]

Would obsoleting FeatureContext mean ScenarioContext would expose something like FeatureInfo?

[gasparnagy]

Surely yes.
But just to be clear: I did not wanted to say that we obsolete FeatureContext, but rather rethink based on the usage.

[DrEsteban]

We use it to share certain cloud resources across the lifetime of a Feature. That way we don't have to pay the cost of creating and deleting them for every scenario of a feature.

For example, we register a "resource tracker" to the DI container in the [BeforeFeature], then we create the resources (once) using Givens in the "Background:", and only clean them up in the [AfterFeature] after all scenarios are done.

[gasparnagy]

Is the cached resource tracker is thread-safe in your case (so if multiple parallel-running scenarios would use the same resource tracker, would that cause any problem)? Note that this is NOT how reqnroll behaves currently, just a hypothetical question.

[DrEsteban]

In the areas we use this convention, we make sure we're running with class-level parallelism to avoid that problem. (Scenarios of a Feature running sequentially.) In areas where no resources are shared, we run method-level parallelism.

We run our tests in distributed test runners and use @tags to split them up and schedule appropriately.

[Insire]

We are currently still using Specflow, but i have started a branch with the migration to reqnroll already, which is currently dormant, since i couldnt find a solution on how to generate a livingdoc as html with reqnroll yet.

We use the FeatureContext to access its corresponding FeatureInfo. We are using the following properties

• folderpath (participates in generating a unique database name)
• title (participates in generating a unique database name)
• tags (control what services are needed during test execution)

We also use the FeatureContext to register services via the FeatureContainer that should only be created once per feature via the following methods

• IsRegistered<T>
• RegisterInstanceAs<T>

We do the same again for TestThreadContext and TestThreadContainer. And everything is accessed via IContextManager injected into:

• BeforeScenario
• BeforeScenario(tags)
• AfterScenario

[gasparnagy]

Could you please describe with a few sentences what kind of services you register to the feature container and what is the benefit of registering it there versus the scenario or test thread container?

[Insire]

The project consists of multiple applications (web api, windows service, console app, wpf apps). The services represent one for each and control their lifetime and configuration.

E.g. we may want to run one of the wpf applications, but for that to be possible the web api needs to be running, so that wpf app can talk to it via http requests.

For the web api to have all the data, we may need to run the windows service to do some processing. So that may need to be started.

And everything needs a database to be setup and seeded with data. so that needs to be in there somehow aswell.

The services in question ensure things are started in the correct order and at the right time. Also when those services are being disposed everything (database, filesystem) will need to be reset for the next test/scenario.

The primary usecase for TestThreadContainer and FeatureContainer are to ensure that there is only for each of my services. E.g. exactly one web api and one windows service per feature.

The only exception is the wpf apps. as there can only ever be one per process, since everything runs inside the same process. Tests that dont need a wpf app, run in parallel. Tests using a wpf app, run in sequence.

[shack05]

We have a test suite that uses Appium to test a native mobile app. The features run in parallel.
We use Before/After feature hooks to reserve and return objects representing a 'device slot' from an object pool. A device slot is simply the device identifier (udid/avd name) for a virtual device (emulator/simulator) + port number for an Appium server. The device slot resources are already running prior to starting the test suite.

We could move the logic to Before/After scenario, but it made sense to use feature hooks because we run features in parallel.

[shack05]

We inherited a large API test suite where data setup is expensive. To reduce execution time we introduced a cache to reuse data for non-destructive tests (such as tests that verify 400, 401, 404, ...). The cache is created in a BeforeFeature hook. If the cache is empty the test will perform the set up and store the result. If the cache is not empty, the test will skip the set up and reuse the data in the cache. Data is cleaned up and removed when a test fails and at the end of the feature (in an after feature hook).

That caching approach might be a bit of an anti-pattern but it was a relatively easy thing to implement for a quick win.

I think the AfterFeature hook is necessary to support this pattern because we need to clean up the data after the last scenario runs in the feature.

[shack05]

We use BeforeFeature hooks to register global resources (things we only want one instance of per test run) into the container in the TestThreadContext.
I don't remember exactly why we do this. I think its because BeforeTestRun and AfterTestRun hooks only run once, but we run tests in parallel at a feature level. We create the resources once in a BeforeTestRun hook, store the result in a field, and then use a BeforeFeatureHook to register the resources in the TestThreadContext.
Registering the resources is cheap, so I wouldn't expect any issues if we moved the registration to a BeforeScenario hook instead of BeforeFeature.

Examples of global resources:

• metadata such as version information for the system under test
• http clients
• classes we use to capture data for reporting purposes