Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Backup command #4

Open
sabledb-io opened this issue Mar 31, 2024 · 4 comments
Open

Add support for Backup command #4

sabledb-io opened this issue Mar 31, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@sabledb-io
Copy link
Owner

SableDb should backup its data every N minutes (configurable)
A nice to have: built-in support for S3 bucket upload + the ability to start from an S3 backup

@sabledb-io sabledb-io added the enhancement New feature or request label Mar 31, 2024
@crrow
Copy link

crrow commented Aug 3, 2024

Hi @sabledb-io I'd like to take this one.

@sabledb-io
Copy link
Owner Author

Hi @crrow feel free to submit a PR.

Few pointers:
SableDb supports replication (both fullsync and WAL tailing). In the fullsync case, SableDb creates a snapshot of the database (which is basically a folder with files), tar the folder and sends it over to the replication node.

A usage can be found here:

To create a snapshot, use this method and to load it, use this method

As opposed to the initial issue statement (backup every N minutes), I would rather have this as an extended commands. This will basically separates the control path form the data path.

As a suggestion we could use these commands (these commands do not exist in the REDIS command set):

checkpoint create <file:///path/to/checkpoint>
checkpoint restore <file:///path/to/checkpoint> [delete-all-before-restore]

we could later support different URI formats, for example:

checkpoint create s3://aws-account-id/mybucket/backup.tar
checkpoint restore s3://aws-account-id/mybucket/backup.tar

WDYT?

@crrow
Copy link

crrow commented Aug 4, 2024

@sabledb-io Agree with an external command and the URL design, but maybe we can call the command as backup rather than checkpoint ? And Rocksdb also provides the ability of backup.
So in my opinion, it may look like this:

# For backup
backup s3://$BUCKET/$BACKUP_FOLDER

# For restore
restore s3://$BUCKET/$BACKUP_FOLDER

But I don't know how to implement increment backup in this case. A backup operation may take sometime.
I think we can refer to this repo https://github.com/facebookarchive/rocks-strata

@sabledb-io
Copy link
Owner Author

  1. I am ok with changing the command to backup
  2. There is no need for incremental backup as it is already supported by SableDb (in the form of replication) - you can start a secondary instance (or more replicas) of SableDb which will be receiving incremental updates from the primary node
  3. The whole purpose of this was the ability to backup the data in case of disaster recovery and for faster startup time.

You can read more about how SableDb replicates its data to another instance here:
https://sabledb-io.github.io/sabledb/design/replication/

@sabledb-io sabledb-io changed the title Add support for Backup Add support for Backup command Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants