Reqnroll Configuration subsystem (especially for plug-in authors)

[clrudolphi]

As I progress in development of the Cucumber Messages subsystem, I have attempted to solve the challenges of how to provide configurability for the subsystem. The existing Reqnroll configuration system (reqnroll.json) is not sufficiently extensible. The json schema doesn't explicitly provide for extensions.

I have in mind a configuration subsystem that provides the following features:

• An extensible json schema that allows other subsystems (and plug-ins) to register their (sub)-schema/format.
• A configuration file loader that can handle the dynamic nature of such a format.
• A pluggable set of mechanisms by which configuration information can be provided to Reqnroll (the default json file being the first among them). We might support composing configuration information from user-supplied configuration providers (such as database systems, docker/kubernetes controllers, etc)
• Configuration settings provide for overrides based upon Environment (one set of settings might be active when run on a developers workstation, but another used when run on the build server)
• A hierarchical system of computing overrides of configuration information. For example, the Cucumber Messages subsystem provides a 'default' configuration (basically turns it off). This is overridden by what the json config file indicates. This then is overridden by values provided by Environment Variables. Finally, this 'computed' configuration can be overridden at run-time by settings provided by a run-time API. This API could be surfaced as an injectable service and/or a set of tags.

What configuration challenges have been faced when developing plug-ins in the past? How were they overcome?

What challenges do you as Users of Reqnroll face when integrating Reqnroll into your build and test environments?

[gasparnagy]

Thank you for sharing this. This is almost exactly what I was thinking on.

I don't have a fixed idea how the extension point in reqnoll.json would look like, but to start thinking on it, let me throw in a sample config file (the concrete names/settings are just examples):

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",

  // standard Reqnroll settings
  "bindingAssemblies": [{"assembly": "MyExternalSteps"}],
  "language": {
    "feature": "hu-HU"
  },
  "runtime": {
    "stopAtFirstError": true
  }

  // main extensibility point
  "testSettings": {
    // settings for cucumber reports
    "cucumberMessagesProtocolReport": {
      "outputFile": "../myfile.ndjson"
      "someSetting": true
    },
    "browser": {  // extensibility for some UI automation "framework"
      "defaultBrowser": "chrome",  // example of a simple value setting
      "settings": {  // example of a complex config setting value
        "defaultTimeout": 60,
        "pollingInterval": 5
      },
      "allowedProtocols": [ "http", "https" ] // example of an array-like config setting value
    }
  }
}

A few additional aspects that I would like to throw in:

1. The possibility of overriding config settings via environment variables is super important in my opinion, because that allows a very easy flexibility option for CI servers and local runs. This is what I have missed the most.
   I would like to have the environment variable override system flexible and generic (so as a plugin author I should not need to do anything extra to let my params override). E.g. we could have a naming convention and for example to override the defaultBrowser setting of my example above, you would need to set the environment variable REQNOLL_TESTSETTINGS_BROWSER_DEFAULTBROWSER.
   E.g. some NCrunch as a test runner allows defining different run configurations for the tests and allows specifying environment variables for each. But also locally or on CI, you could run your specific test via:

REQNOLL_TESTSETTINGS_BROWSER_DEFAULTBROWSER=Edge & dotnet test

2. We have ported the Reqnroll.Actions.Configuration project that targets similar needs and I would like the new configuration system to replace this plugin. It does NOT have to be fully compatible with it, for example I would like to use the reqnroll.json as the primary source of config, not a separate config file.
   I think the capabilities you described are basically fulfilling the needs of Reqnroll.Actions.Configuration the only one that should be considered is the concept of "targets". See next point.

3. The Reqnroll.Actions.Configuration had a concept of "targets", which is a typical usage of the Reqnroll tests: you have a set of core tests and you would like to run it for different environments. The simplest example would be Web UI tests that you would like to run for different browsers. You can see how it was used to select browsers at the Reqnroll.Actions.Selenium plugin.
   I don't know if the concrete approach (having a separate config file for each target) is really a way to go (I would personally prefer having all the configs of the different browsers in the same file) and I'm not sure that our config system should know about "targets", but the use-case is still valid, so we can use it to "test" the new concept. E.g. the "targets" can be also implemented like this (in this case it will be the responsibility of the plugin to handle & merge the different browser settings):

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",

  "testSettings": {
    "browser": { 
      "selectedBrowser": "chrome", 
      "default": { // default browser settings
        "defaultTimeout": 60,
        "pollingInterval": 5
      },
      "chrome": {
        "defaultTimeout": 120 // specific for Chrome
      },
      "edge": {
        "defaultTimeout": 240 // specific for Edge
      }
    }
  }
}

4. Some test execution providers, like MsTest allows specifying test run parameters. These can be specified for a particular test run and accessible via TestContext.Properties in case of MsTest. We could use a similar convention as for environment variables, so for example setting the REQNOLL_TESTSETTINGS_BROWSER_DEFAULTBROWSER test property would override the defaultBrowser setting of my example above. (I am not sure what the precedence between env vars and test run parameters, but we can figure that out.)

5. One note on config overrides. There are a few settings (e.g. the "language" and "bindingAssemblies" sections) that has to be processed by the IDE extensions (VS extension). Handling overrides on these makes the IDE extension code super complicated, so I would suggest that for these settings we should disallow config overrides (so they can only be set via reqnroll.json). I think this is an acceptable restriction and will help a lot with IDE extensions.

6. It would be cool if the JSON schema of the config file could be extended by plugins, but I am not aware of any technical possibility to do that. So most probably we should just stick to have the schema validation and the IDE completion option only for the "known" config settings. But this is acceptable IMHO.

7. Related to the previous point, I'm not sure if the plugins would need to declare upfront what config settings they want to read (so specify the schema) or just provide them an API where they can read whatever they want from the config (or at least from the extensibility part of the config). I have another project where I used such thing and I processed the "extensibility" settings as IDictionary<string,object> structure and that was enough. So with my example, we could have an API, like ConfigProvider.GetTestSettings("cucumberMessagesProtocolReport") that would return a dictionary with outputFile as ../myfile.ndjson and someSetting as true. But maybe we can have a better option, e.g. ConfigProvider.GetTestSettings<CucumberMessagesProtocolReportSettings>("cucumberMessagesProtocolReport") that would serialize the related section to the specified type.

[Code-Grump]

No, I mean expose an IConfiguration and IConfigurationRoot via DI. It's the type from where all the Microsoft library begins, where all existing class libraries will want to integrate.

Unless there's a compelling reason to not provide this interface directly, I think adding an abstraction will only make it harder to use.

[clrudolphi]

I would expect to expose the IConfiguration interface to plug-ins; they're free to fetch whatever they like without any further involvement from us, using all the established conventions of this configuration system.

OK, let's suppose that we expose the IConfiguration interface after having created a ConfigurationManager, pointing it to the usual reqnroll.json file, establishing a "REQNROLL__" environmental prefix for overrides and then Build()-ing it.

Do we provide an event (during plugin initialization) to allow plugins to register their own customizations to the configuration builder (before we call .Build()? For instance, they may want to add the command-line configuration provider or add their own private configuration file.

[Code-Grump]

I would propose that we go without any customisable extension points initially: the value of this is providing a standard way to get configuration values into the Reqnroll runtime, and a standard way to read those values from any components. Once these conventions are understood, it's an easy process for end-users to provide and understand configuration values from the sources we have proposed.

Also, the formats we're proposing are open to values being included that we have no knowledge of and can still be consumed through an IConfiguration instance; extensions should be able to take full advantage of this common mechanism.

Adding an extension point now feels premature: these sources could cover every need and incorporation additional sources is a non-trivial feat, as ordering matters strongly. I'd like to see if there's a strong need for such a thing before trying to design a suitable extension-point.

[clrudolphi]

Makes sense.
Do you have a preference of whether to expose anything beyond IConfiguration? such as IConfigurationRoot and/or IConfigurationManager? If so, why? (I'm not opposed, just curious what the use-cases would be)

[Code-Grump]

I think starting with the singleton IConfigurationRoot and IConfiguration (which is just an alias to IConfigurationRoot) should be enough to meet every need. It's not hard to add further interfaces should a need arrive, but much harder to remove them if we add something prematurely.

[clrudolphi]

I've drafted an experimental branch for this and have the basics working for the test execution run time (not generation).

This wasn't as difficult as feared, as I was able to refactor this into the existing object graph structure. We may want to think about a complete overhaul instead.

[clrudolphi]

@gasparnagy when examining the existing code and how configuration is loaded, what is the purpose of the ReqnrollConfigurationHolder class? It would seem that there are paths through the code in which the configuration file is loaded as text by things other than the IConfigurationLoader. Why is that necessary? Can it be eliminated?

[gasparnagy]

It is still necessary and unfortunately cannot be eliminated. I detailed it in my comment here: #306 (comment)