storage: concurrent housekeeping
#25134
Open
+151
−67
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WillemKauf
force-pushed
the
concurrent_housekeeping
branch
from
926f409 to
b60242c
Compare
Retry command for Build#62113please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#62113
test results on build#62145
|
WillemKauf
force-pushed
the
concurrent_housekeeping
branch
from
b60242c to
e1d4d8b
Compare
This lock controls concurrency between `gc()` and `housekeeping()`, which are two functions that in the past have not been concurrent. We are locking to invoke these concurrently in two separate fibres from the `log_manager`. It is expected that `gc()` is a fast process, while `housekeeping()` (which performs compaction), is not.
We are going to separate garbage collection from housekeeping (garbage collection then compaction) into seperate loops within the `log_manager`. Add `housekeeping_job_t` to specify which of the two jobs should be run by a generic worker function in future commits.
To be specific about the future uses of the semaphore and jitter variables, rename them to `_housekeeping_jitter`. Also, add a new `_gc_sem`.
Using the `housekeeping_job_t`, we now kick off two background fibres to handle urgent garbage collection and housekeeping. The underlying functions called are `log->gc()` and `log->housekeeping()`, which as mentioned, have their concurrency managed by `disk_log_impl::housekeeping_lock`.
To improve observability for long running compaction processes which may be starving out urgent garbage collection.
WillemKauf
force-pushed
the
concurrent_housekeeping
branch
from
e1d4d8b to
4d4c4a6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP dont look at this before #24991
Backports Required
Release Notes
Improvements
disk_log_impl