Taking Snapshots Periodically

[tguner-arteriaai]

Hi,

I was just wondering if there is any support in the configuration file or somewhere that we can set periodicity of taking snapshots of the entire storage, like at every 5secs.

This can be manually achieved by hitting the snapshot endpoint or through defining periodic crontab kind a job to hit these endpoints but I want to use configuration directly if such a config exists first.

This I need as part of the backup strategy. Thank you

[timvisee]

There is not. In fact, creating snapshots this quickly will become very expensive.

For this I'd recommend to snapshot the file system instead. Note that I mean a file system level snapshot such as a K8s PVC volume snapshot, not a Qdrant snapshot. Those will guarantee proper data consistency as well.

In our cloud you can set up a backup interval, or create one manually with the press of a button.

Even then, creating such backup/snapshot every 5 seconds is considered very often. How about running two separate Qdrant instances and sending operations to both at the same time. That may be sufficient?

[tguner-arteriaai]

Is this already part of the distributed deployment? I mean if I have 2 nodes, they are going to be communicating as part of the consensus algorithm right?

[timvisee]

Correct.

If you have multiple nodes you can allow node failure if set up properly.

Note that the formula for number of nodes is like this: (allowed_number_of_failures * 2) + 1. So if you'd like up to one node to fail you'll need at least 3 machines. Of course you'll also set up the replication factor accordingly.

Generally speaking: the more nodes and the higher replication factor, the more resilient and robust your node is. But, that's a tradeoff against cost.