[jUnit formatter] Skipped steps are considered as errors

[mdelapenya]

We noticed that when a step fails in an scenario, subsequent steps are marked as skipped, which is OK imho.

When processing a test case (a scenario), all the steps that are skipped are included as errors (although having the skipped type). And because they are marked as error XML elements, we are not able to identify the root cause of why the scenario failed.

But why? Because the CI (Jenkins) provides a higher ranking to errors than to failures, causing the UI to display the fist step error instead of the first failure, which is misleading. I.e.:

<testsuites name="main" tests="29" skipped="0" failures="29" errors="0" time="377.75344176">
    <testsuite name="Integrations" tests="27" skipped="0" failures="27" errors="0" time="322.571063757">
        <testcase name="activemq-5.13.0 sends metrics to Elasticsearch without errors" status="failed" time="5.936670176">
            <failure message="Step metricbeat is installed and configured for "activemq" module: stat /var/lib/jenkins/workspace/Beats_beats_PR-23854/src/github.com/elastic/beats/x-pack/metricbeat/build/distributions/metricbeat-8.0.0-SNAPSHOT-linux-amd64.docker.tar.gz: no such file or directory"/>
            <error message="Step there are "activemq" events in the index" type="skipped"/>
            <error message="Step there are no errors in the index" type="skipped"/>
        </testcase>
    </testsuite>
</testsuites>

Thankfully we store the XML file and then we were able to read the real error, hid by the first error.

With the above XML file, I'd expect the UI to display the failure step. In fact, removing all errors simply keeping failures makes the UI to display the proper error message.

Note: we are aware that, potentially, it could be an issue in the Jenkins side where errors are overriding the real failure, but wanted to share here our expectations first.

[lonnblad]

Hi @mdelapenya, thanks for the issue!

@mpkorstanje, do you have any insight on how we do this for other Cucumber implementations.

[mpkorstanje]

This is a rather complex issue:

1. Cucumber-JVM only reports the most severe result for a scenario rather then the results of all steps.
2. Depending on which XSD you pick, you get a different answer to the question if multiple test results are allowed at all:

• https://llg.cubic.org/docs/junit/
• https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/site/resources/xsd/legacy-surefire-test-report.xsd
• https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/site/resources/xsd/surefire-test-report-3.0.xsd

3. Should multiple be results be allowed in practice, then if I'm not mistaken error is worse then failure. The latter implies the test failed, for example on an assertion. The former suggests the tests stopped in a completely unexpected fashion. So it may be that Jenkins picks the first, most severe result when given a choice. But that is speculation on my end.
4. I believe both failure.type and error.type refer to the type of the error. Using skipped here would not be appropriate.
5. failure.type is mandatory according to the xsd but currently missing.
6. If multiple results are allowed, using the skipped element would probably be more appropriate then error with type="skipped".

[mdelapenya]

I tried to use <Skipped/> elements instead, and jenkins populated the UI with two messages: one for the failure and one for the first skipped step