Show all tests

[gsnedders]

It'd be nice to show all tests that appear in the manifests for the revisions we've run, regardless of whether we have test results for them. (This, in part, means manual tests will show up.)

Why? It makes it clear that there are more tests there aren't results for, and that what is on the dashboard isn't necessarily a complete view of interop.

[foolip]

That'd be nice, yeah.

[foolip]

@lukebjerring WDYT? This would require that we have the manifest for all runs that we show results for, but we could generate it for old runs.

[foolip]

This came up when discussing manual tests with jgraham on IRC. Missing results would make sense to list as an anomaly too.

[foolip]

@jgraham

[lukebjerring]

IMHO, this is probably a runner (or, more specifically, a WPTReport) behaviour.
If so, the JSON for results should start by building a map that includes empty entries for all tests which are in the manifest. Then, populate it with (actually run) results.

Passing along a separate file to the endpoint, or extra metadata, would be cumbersome.

[gsnedders]

That relies on everyone implementing runners correctly. I think there's definitely an argument that it's the collection side that should handle it?

[jugglinmike]

The term "collection" may have been a bad choice on my part. One sub-system receives test results from a browser and "collects" them into a report. This repository describes such a collector. Another sub-system receives those reports and "collects" them into a database of historic test results. That's what @Hexcles is working on in his recent design document.

Either "collector" could ensure each dataset specifies missing results with null data. Since the latter is expected to validate the data generally, it will have to be aware of all the tests expected to be run for any revision of WPT regardless.

Still, I think it makes sense for the result reporter to null out missing data. That way, the sub-system in charge of receiving and archiving the results doesn't have to muck around with externally-produced data.

[foolip]

Moved to web-platform-tests/wpt.fyi#56.