Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[persist] Thread chunks of data from consolidation to the batch builder #29577

Merged
merged 9 commits into from
Sep 19, 2024
Merged

[persist] Thread chunks of data from consolidation to the batch builder #29577

merged 9 commits into from
Sep 19, 2024

Commits on Sep 19, 2024

  1. Minor fixups

    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    372cd05 View commit details
    Browse the repository at this point in the history

  2. Pass around BlobTraceUpdates in the batch builder

    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5c082fb View commit details
    Browse the repository at this point in the history

  3. Remove the per-item schemas in favour of batch schemas 

    This is a vestige of back when we had two sets of schemas in compaction,
which was weird and is long gone. Enforce that all the data in a batch
has the same schema.
    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    b525f43 View commit details
    Browse the repository at this point in the history

  4. Allow pushing entire sets of updates into the builder

    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    b58eb98 View commit details
    Browse the repository at this point in the history

  5. Estimate the output size via ArrayOrd

    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    b345465 View commit details
    Browse the repository at this point in the history

  6. BlobTraceUpdates::concat 

    Following the similar method on columnar records.
    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    095fdf5 View commit details
    Browse the repository at this point in the history

  7. Work harder to hit the target size

    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    95e3fe6 View commit details
    Browse the repository at this point in the history

  8. Count the number of bytes we've had to copy in concat 

    If this is more than a small faction of bytes, it's worth tuning
consolidation to return larger chunks of output.
    @bkirwi
    bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    6aa51b5 View commit details
    Browse the repository at this point in the history

  9. cargo test: Rework test_builtin_connection_alterations_are_preserved_… 

    …across_restarts

To prevent stack overflows
    @def- @bkirwi
    def- authored and bkirwi committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    463af6f View commit details
    Browse the repository at this point in the history