Rethinking Results

[dcramer]

Let's rethink how we're exposing information in the UI.

At a high level you view results of test runs by one of two categories:

• by Change (either commit or change request)
• by Build

While we primarily only care about Change-focused builds, its common practice to have other kinds of CI runs which are done by trigger or periodically. These are often not associated with a change, and coupling them to that is confusing to a user.

Let's focus on Change, as individual Build is known. A change is often a commit, but sometimes will be a patch build (something like a Phabricator code review). These are coupled to Source in Zeus. At the end of the day Change consists of two types:

• Commit
• Change Request (PR, Patch)

In both of these we want to aggregate the latest build per provider together and share the results of that. That means every function, whether its jobs, tests, or code coverage, needs to be able to aggregate result sets upwards and dedupe. We already do this in many cases given Job's can report duplicate results.

On the backend we have a bunch of revision_X endpoints. These are fine for commit focused changes, but require more effort for Change Request and/or Patch focused changes (these vary based on what code review process you use). We need to provide a better solution here that doesnt require having N endpoints per resource type.

One option is to do resource dedicate endpoints. For example, '/tests' (or test_index), could have a set of query params passed to it to filter which tests you want. This would work like build_tests does today, but you might pass:

/tests?source=XYZ
/tests?revision=XYZ
/tests?build=XYZ

The choice comes on output: should they be aggregated together or return individual results? In almost every case we want them aggregated, and while you might think we need control over that, I'd challenge that its just extra work.

So for example, if you query /tests?job=XYZ, you'll only ever get one "run" per test result. If you query /tests?build=XYZ, you'll get one test result per unique identifier, and possibly multiple "run" values. With this style of change we'd need to make it so you can either do:

1. /tests?build=XYZ&build=ZYX
2. /tests?source=XYZ

The challenge here is that the "source" is going to multiple builds (as its the latest build per provider), and we can fetch those build IDs and then query it (meaning the endpoint ALWAYS aggregates), or we can say "the endpoint only lets you query against one top level identifier at a time". It seems easier to always aggregate, as it means the output format is always identical, but it also means the top level format also needs to be build-agnostic.

With that said, I think I'd prefer just passing source=XYZ, but letting you pass either. In all of these endpoints we'd then just say "find all the builds queried against, and aggregate their results". It ends up always being the same logic, it will just be a chunk of work to make all endpoints behave like build_tests does today (with the AggregateTestCaseSchema).

[dcramer]

@jan-auer threw up initial thoughts on how we might approach this with an example

[dcramer]

This is making me pivot back to my original concept of Changes (from Dropbox days), where my intention was to highlight an actual Changeset -- either one that originated via a PR (and is collapsed into a PR) or one that was straight into a branch.

You can imagine that you load up zeus and the core view is simply your "Changes" (which happen to be builds). Clicking in one would show you to aggregated build results (per this issue's core description), as well as describe the originated Pull Request or Commit. We'd need some more metadata to lock this in, but we're already seeing the metadata we have today (from purely looking at Travis build data) isn't enough.

I BELIEVE in this world, we could treat Source often as the source of truth for "Change", with the caveat that every commit has a Source, and a Change Request might be N commits.

[dcramer]

Additionally focusing endpoints at the root level (e.g. /builds vs /repos/XXX/builds) will make it easier to restructure the navigation to show all results vs repo-specific results (as a default).