Run Modulo Tests

[skiz]

🤔 What's the problem you're trying to solve?

I want to run an indeterminate number of tests in parallel across multiple test runners without tagging individual tests.

✨ What's your proposed solution?

Add a modulo and target option that deterministically selects every Nth scenario to run.

This should also work the same across all instances using the same seed.

I would pass this in as an environment variable to the configuration options.

r1$ SEED=1234 MOD=3 INT=1 make test.... (scenarios 1,4,7,10)
r2$ SEED=1234 MOD=3 INT=2 make test... (scenarios 2,5,8,11)
r3$ SEED=1234 MOD=3 INT=3 make test... (scenarios 3,6,9,12)

⛏ Have you considered any alternatives or workarounds?

I can do this with tags, but forgetting or typo on a tag will ignore the test completely.

📚 Any additional context?

No response

[skiz]

I have made a PR for this feature: #678

[vearutop]

Hi, thank you for this contribution!

I'm not sure godog API extension is needed for such a behavior. The problem seems a bit too specialized and uncommon to me.

I think needed behavior can be achieved with better flexibility using existing "before scenario" hooks and godog.ErrSkip.

For example (based on https://github.com/cucumber/godog/blob/main/_examples/godogs/godogs_test.go):

var (
	scenarioIdx = 0
	mod         = 3
	target      = 2
)

func InitializeScenario(ctx *godog.ScenarioContext) {
	ctx.Before(func(ctx context.Context, sc *godog.Scenario) (context.Context, error) {
		scenarioIdx++

		if scenarioIdx%mod != target {
			return nil, godog.ErrSkip
		}

		return ctx, nil
	})

...

Would such approach work in you case?