storage: track latest reported status and report on reconnect #30576
+51
−12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aljoscha
force-pushed
the
storage-report-latest-status-on-reconnect
branch
2 times, most recently
from
3947a06 to
c2840f6
Compare
aljoscha
force-pushed
the
storage-report-latest-status-on-reconnect
branch
from
c2840f6 to
de6c0ab
Compare
5 tasks
Before, we would purely pump upgrades from running dataflows over to the controller. Which would make it so that when a controller re-connects it doesn't get status messages for the latest state. Now, we track the latest status update, per object, and also what has been reported, per object. And on re-connect we clear out what has been reported, which makes it so that we report the latest status when a reconnect happens.
aljoscha
force-pushed
the
storage-report-latest-status-on-reconnect
branch
from
de6c0ab to
6ae2711
Compare
teskje
reviewed
Feb 6, 2025
src/storage/src/storage_state.rs
Outdated
| @@ -1288,6 +1340,8 @@ impl StorageState { | |||
| self.ingestions.remove(&id); | |||
| self.exports.remove(&id); | |||
|
|
|||
| let _ = self.reported_status_updates.remove(&id); | |||
There was a problem hiding this comment.
Don't we need to remove the latest_status_updates entry too?
| } | ||
|
|
||
| if to_report.len() > 0 { | ||
| self.send_storage_response(response_tx, StorageResponse::StatusUpdates(to_report)); |
There was a problem hiding this comment.
A place that's going to conflict with #31261 (fyi @petrosagg)
src/storage/src/storage_state.rs
Outdated
Comment on lines
323
to
326
| /// The latest status update that has been _reported_ back to the | ||
| /// controller. This will be reset when a new client connects, so that we | ||
| /// can determine what updates we have to report again. | ||
| pub reported_status_updates: BTreeMap<GlobalId, StatusUpdate>, |
There was a problem hiding this comment.
The existence of this map surprises me! Naively I'd have thought that we can just unconditionally send the contents of latest_status_updates on reconnect. Or, if that's somehow hard to set up, wouldn't an initial_status_reported flag suffice?
aljoscha
added
the
release-blocker
|
@teskje I updated with your suggestions, ptal. 🙏 |
|
Thanks for the speedy review/iteration on this one! 🙇♂️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before, we would purely pump upgrades from running dataflows over to the
controller. Which would make it so that when a controller re-connects it
doesn't get status messages for the latest state.
Now, we track the latest status update, per object, and also what has
been reported, per object. And on re-connect we clear out what has been
reported, which makes it so that we report the latest status when a
reconnect happens.
Motivation
Tips for reviewer
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.