Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: add "test vector" tests to this repo. #64

Open
znewman01 opened this issue Mar 8, 2023 · 2 comments
Open

Proposal: add "test vector" tests to this repo. #64

znewman01 opened this issue Mar 8, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@znewman01
Copy link

Previously, we've discussed the merits of two sorts of conformance testing:

  • "flow-based" end-to-end tests, like what's currently in this repo. These tests are good for checking interoperability, common cases (both failure and success), compare good for checking interoperability, common cases (both failure and success), compatibility with the infrastructure (including tests that can run against staging/dev environments). The cost is that they're heavierweight and involve network calls.
  • "test-vector style" tests: e.g., for verification flows. They may miss some coverage and it's harder to exercise network calls, but these are much faster and support fuzzing better.

I, for one, left that conversation convinced that it's a good idea to do both.

We're starting to pick up on test vector work (e.g., sigstore/protobuf-specs#66 ; CC @asraa). At the Sigstore clients meeting the other day (CC @bdehamer, @codysoyland, @kommendorkapten) we discussed where these (and related harness code etc.) should live. While previously, we had planned to put them in sigstore/protobuf-specs, it seems like sigstore/conformance-tests is a quite natural fit.

If you all agree, we can add those in here, update the README to articulate the overall test strategy, and use this as the home for test vector testing. If you think they'd be better off elsewhere, we can discuss where too.

@znewman01 znewman01 added the enhancement New feature or request label Mar 8, 2023
@woodruffw
Copy link
Member

That works for me! I agree that we want both kinds of conformance testing, and a singular point of reference for them makes sense to me.

@tetsuo-cpp
Copy link
Collaborator

Sounds good. I also agree this seems like a good spot for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants