Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Subscriptions performance improvements #88

Merged
merged 36 commits into from
Nov 16, 2023
Merged

Subscriptions performance improvements #88

merged 36 commits into from
Nov 16, 2023

Conversation

jeromegn
Copy link
Member

@jeromegn jeromegn commented Nov 9, 2023

Subscriptions were affecting the main database because there was only 1 subscriptions database and it was being ATTACHed. Any writes to either would lock both.

These changes greatly improve subscriptions performances by:

  • Using a dedicated SQLite database per subscription
  • Never writing to the main database or to the subscription database when attached (therefore preventing any write locks from occurring)

jeromegn added 20 commits November 7, 2023 08:22
@jeromegn
remove many transactions from the subscription code
f93fbbe
@jeromegn
massive refactor of subscriptions to use an on-disk queue so as to no…
caa87d7 
…t lock the state DB
@jeromegn
remove subs migrations and subs table references
2379a4e
@jeromegn
don't break backup process if certain tables never existed
3761ce2
@jeromegn
log path where we're initializing the subscription
01fd7ad
@jeromegn
better default subscriptions path
00d39cf
@jeromegn
get rid of the queue, just use tables and make sure we're never writi…
5456613 
…ng to the state db from subscriptions
@jeromegn
use a unique index on expressions to make sure we accomodate null pri…
72eb8c4 
…mary keys
@jeromegn
only actually upsert if there were changes
609cd3f
@jeromegn
add an assertion so that we don't regress this new fix
b3253f0
@jeromegn
use a condvar to prevent fetching metadata from a subscription so as …
18224c6 
…to not get incomplete / corrupted queries
@jeromegn
cleanup incomplete subscriptions on boot
c17efb0
@jeromegn
don't use a queue, jsut use crsql_changes as a queue
5ed8c7f
@jeromegn
simplify some code and added tests to make sure the correct change id…
03e0d40 
… is sent
@jeromegn
prevent blocking on full partial apply channel
450f1f9
@jeromegn
already have the only priority conn, don't try to get it again from w…
8b1d0c1 
…ithin
@jeromegn
much faster bookkeeping compaction by doing 1 actor at a time, using …
91f2b98 
…indexes and sleeping between clears to let the DB breath
@jeromegn
don't log as much, create indexes if they don't exist on boot (remove…
18bd6b3 
… that logic later...)
@jeromegn
async-ify cleared versions processing so it won't block anymore
ebf1e70
@jeromegn
simplify pubsub logic by adding a SubsManager that handles all matchers
c7c029f
@jeromegn jeromegn merged commit 1c5096b into main Nov 16, 2023
2 of 3 checks passed
@jeromegn jeromegn deleted the subs-perf branch November 16, 2023 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jeromegn