-
Notifications
You must be signed in to change notification settings - Fork 466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
parallel-workload: 0dt: thread 'coordinator' panicked at src/storage-controller/src/lib.rs:703:17: dependency since has advanced past dependent (u417) upper #28885
Comments
Should help with diagnosing MaterializeInc#28885
We can use CI Failures to see when this occurs again with more context. So far nothing. |
Aha! we have one: https://buildkite.com/materialize/nightly/builds/9115#01915669-0a3f-438b-be63-2bc0b7b1b8bd The relevant log snippets from that one are:
edit: See diagnosis below! So looks like that shard/collection has been dropped and is in the process of finalization. But then we also try and bootstrap it in the storage controller. In the controller, we can see from the read holds that the I'll try and get to the bottom of how this can happen, but I still think it's not an issue for now. |
I believe I have this diagnosed! What happens is this:
I updated the log snippets above to basically lay out precisely what happens:
Thoughts:
|
Is this something we need to solve up front, or are you proposing that we downgrade the assert and then see if anything like this happens before trying to address it? |
(Because I'm very in favor of this.) |
That's my preferred approach, yes! |
Fixes MaterializeInc#28885 With this change, we might now get panics in other places where we make assumptions about sinces/uppers that don't hold in read-only mode or when there are concurrent coordinator-like processes.
Fixes MaterializeInc#28885 With this change, we might now get panics in other places where we make assumptions about sinces/uppers that don't hold in read-only mode or when there are concurrent coordinator-like processes.
Fixes MaterializeInc#28885 With this change, we might now get panics in other places where we make assumptions about sinces/uppers that don't hold in read-only mode or when there are concurrent coordinator-like processes.
What version of Materialize are you using?
736303f (Pull Request #28873)
What is the issue?
Serious version of #28634 in Parallel Workload (0dt deploy)
ci-regexp: dependency since has advanced past dependent
The text was updated successfully, but these errors were encountered: