[wip] Allow reading back data in structured order #29666
Draft
+130
−39
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Part of #24830 - allows reading back data in the structured-data order.
Before this change, we could configure Persist to compact data in the new structured-data order, but when we were consolidating at read time we'd reorder it back to the codec order. This is inefficient, and we'd like to allow callers to rely on the sorted order of data in the future for things like
PARTITION BY
.Tips for reviewer
This uses the same ordering flag as the compaction side, which is a little weird: when we first flip the flag, we'll start using the new order for the fast-path peeks before there's any meaningful percentage of batch parts in the new order. It's possible we'll want to add fancier knobs for the true rollout, but this is enough to toggle things in CI / staging for now.
Checklist
$T ⇔ Proto$T
mapping (possibly in a backwards-incompatible way), then it is tagged with aT-proto
label.