Negentropy + DHT Sync #19
Conversation
|
@illuzen I see what you mean with the deserialize event method and making graviton conform to the storage interface for negentropy, realistically we can add some additional functions for taking the event into the database as raw bytes but verifying that the event is correct is important so we are probably going to end up deserializing it and verifying them all anyway. If more data is required to be stored for the negentropy sync then it's probably best we cache it as I would think this additional data is always generated from the event data itself anyway. Let me know if I have misunderstood anything. |
|
the go.mod has this line
it just looks kind weird, the other imports don't have long hex strings after the semver |
|
Ah yeah I wouldn't worry about that at the moment, probably a way to specify the actual version so it doesn't just v0.0.0 with the time stamp and potentially the commit id? It's been on my list of things to look into. |
|
ok @f7f376a1fcd0d0e11a10ed1b6577c9 , made some serious progress, the main thing left to do is upload and download raw events that are identified as being missing. There are two TODOs in the code for this. Can you advise on how to do this using existing machinery if possible? Probably should use the same mechanism clients use to upload events... also, I was thinking, should we make git commits a special kind of event? create a NIP for it? maybe you've already discussed this |
|
Ok @f7f376a1fcd0d0e11a10ed1b6577c9 I think this is done, the test passes. |
|
I'm adding dht code + tests, fyi |
|
waiting for review from @f7f376a1fcd0d0e11a10ed1b6577c9 |
This PR creates a new lib subdirectory "sync" which handles the negentropy syncing between hosts and methods for putting and getting records about relays from the mainline DHT. There are new tests for both of these functions as well.
Mutable inputs "targets" to the DHT are required to be sha1(pubkey + salt), where the salt is optional and arbitrary. For our purposes we leave it empty.
We use the DHT to broadcast user's relay lists to other relays. This is triggered by kind11011 events and we periodically (every 2 hours) re-upload because that's about how long the dht will store data for us. This mostly happens in kind11011handler.go and relay-store.go. We also store the uploadables, the relays and the authors of kind11011 events in a sql database called sync_store.db.
When we are missing events in a kind1 reply, if there is a dht_key tag, then we use that to look up the relay list and try to sync with those relays on the missing-author to find the missing event. We also add those discovered relays and author to the periodic-sync list in the RelayStore object.
The negentropy code closely mirrors that of str-fry, but not exactly, since we add a couple methods for uploading and downloading missing events. Basically it takes a libp2p host and initiates a protocol like this one: https://github.com/hoytech/strfry/blob/master/docs/negentropy.md
There are several unit tests in two files, test/event_test.go which tests the negentropy syncing on a random collection of events and test/dht_test.go which tests putting and getting objects from the mainline dht. All tests should pass, although we test against the real mainline DHT so it takes a couple minutes and may fail if bit-torrent is blocked. Using a VPN seems to work.