Add simplified integration framework

[rdner]

That can be used for running Beats binaries in integration tests.

This framework has a simplified API comparing to the existing one and uses a more efficient way to search for logs in the output of the command.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Motivation

Recently I needed to write a few integration tests that run the Filebeat binary and I found the existing integration framework very cumbersome. It requires a lot of boilerplate code and the purpose of any framework is to illuminate that.

Since we're going to eventually migrate all the integration tests written in Python, we need to have a very simple tool for writing integration tests that should cover most of the test-cases.

Both frameworks (old and new) can co-exist for now.

Pros

This design has a few advantages over the previous one:

• It does not use log files for inspecting/matching the expected logs. Instead it connects directly to stdout/stderr and matches all the output expectations in memory line by line as they arrive. Which makes it extremely efficient at expecting thousands of log lines (e.g. confirming each line of a file gets ingested).
• The test suite kills the process immediately once the defined expectations are met, no more polling with intervals.
• It runs the binary that we ship to our customers instead of a custom binary (debatable, I know but I think we should test what we ship)
• It has a call-chain interface which is more compact
• It supports testing cases when a Beat crashes with errors (not sure if the old framework supported that)
• It has very detailed output for debugging a test failure
• It's generic and in theory can be used with any Beat not just Filebeat (yet to be tested)
• Can be extended and specialized for each Beat, see the example with Filebeat in this PR.

For example, test similar to the one written in the older framework:

https://github.com/elastic/beats/blob/main/filebeat/tests/integration/filebeat_test.go#L37-L94

Can be replaced with this:

func TestFilebeat(t *testing.T) {
	messagePrefix := "sample text message"
	fileCount := 5
	lineCount := 128
	configTemplate := `
filebeat.inputs:
  - type: filestream
    id: "test-filestream"
    paths:
      - %s
# we want to check that all messages are ingested
# without using an external service, this is an easy way
output.console:
  enabled: true
`
	// we can generate any amount of expectations
	// they are light-weight
	expectIngestedFiles := func(test Test, files []string) {
		// ensuring we ingest every line from every file
		for _, filename := range files {
			for i := 1; i <= lineCount; i++ {
				line := fmt.Sprintf("%s %s:%d", messagePrefix, filepath.Base(filename), i)
				test.ExpectOutput(line)
			}
		}
	}

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	generator := NewJSONGenerator(messagePrefix)
	path, files := GenerateLogFiles(t, fileCount, lineCount, generator)
	config := fmt.Sprintf(configTemplate, path)
	test := NewTest(t, TestOptions{
		Config: config,
	})

	expectIngestedFiles(test, files)

	test.
		ExpectEOF(files...).
		ExpectStart().
		Start(ctx).
		Wait()
}

Additionally, it also includes validation that:

• every file has been read until EOF
• every line from each file was ingested

Another example, this time we expect Beat to crash:

func TestFilebeat(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	// path items are required, this config is invalid
	config := `
filebeat.inputs:
  - type: filestream
    id: "test-filestream"
output.console:
  enabled: true
`
	test := NewBeatTest(t, BeatTestOptions{
		Beatname: "filebeat",
		Config:   config,
	})

	test.
		ExpectStart().
		ExpectOutput("Exiting: Failed to start crawler: starting input failed: error while initializing input: no path is configured").
		ExpectStop(1).
		Start(ctx).
		Wait()
}

Cons

• This framework does not have all the functionality of the previous one. Only essential functions that cover most of the test-cases. It can be easily extended, every defined interface is an extension point.
• It heavily relies on the terminal output of a Beat, we can configure both console logging and file logging if necessary but it's outside of the scope of this framework.

Current functionality

Basic Assertions

• Assert an output line that contains a defined string
• Assert a list of output lines in a defined order that contain a given list of strings
• Assert an output line that matches a regular expression
• Assert a list of output lines in a defined order that match a given list of regular expressions
• Assert that the process started
• Assert that the process exited by itself with a certain exit code

Filebeat-specific Assertions

• Assert all files on the list have been read to EOF

Reporting

• Print out all defined expectations of the test
• Print last N lines of the output

Config

• Add additional arguments to the command to run the binary
• Pass a config file (e.g. filebeat.yml)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @rdner? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

Overall it's a very nice initiative. The only thing I'm missing is a readme/documentation about which framework to sue and the features this one has.

You wrote a very detailed PR description with examples, etc. That could become the readme. What do you think?

[belimawr]

[Suggestion]
Either rename NewTest to something like NewFilebeatTest or add Beatname as argument, so it's clear whether this function is specific to a Filebeat test or generic.

[rdner]

Since the full import is already filebeat/testing/integration it does not seem necessary to repeat that again in the function name.

[belimawr]

It's not a big deal, but each call to t.TempDir returns a different directory. Why not use a single directory?

[belimawr]

On a different topic, t.TempDir is very convenient, however when debugging failing (or flaky) tests often I find myself needing to look at the files the test is using, that's why our other integration tests framework keeps the test folder on failure.

What do you think about implementing a similar logic here?

[rdner]

I can't come up with a test case when you would need to look at the configuration file you generated and at the registry file. Logging is going to stderr instead.

If the test fails it prints the last N lines from the output, it's configurable.

Everything should be observed through the output of the Beat. That's my idea of integration tests, we should test everything like our customers run it. For example, if you want to see the stored offset in the state, it should be tested through ingested lines, not inspecting the registry. The registry implementation should be covered by its unit tests instead.

Good catch with the temporary directory though, I was not aware it returned a different directory every time.

[belimawr]

I can't come up with a test case when you would need to look at the configuration file you generated and at the registry file.

It's not a test case, it's us (human engineers) debugging/writing a test, where something is not working as expected and we need to debug/investigate manually rather than just running the tests.

At least for me, this is a very common workflow.

Anyways, it does not need to be implemented or discussed in depth on this PR. We can leave it as it is and later discuss/iterate on it.

Everything should be observed through the output of the Beat. That's my idea of integration tests, we should test everything like our customers run it.

I agree!

For example, if you want to see the stored offset in the state, it should be tested through ingested lines, not inspecting the registry. The registry implementation should be covered by its unit tests instead.

That I disagree because having every possible state we need to test in the logs can be very verbose and heavy. Anyways, that's not a discussion for now.

[rdner]

@belimawr I've added a README and addressed your feedback.

[rdner]

@AndersonQ suggested a good idea to always rebuild the binary when we run EnsureCompiled. I do it with the hash check now, this commit bb39651