Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

host chain artifacts like archives, reindexer dbs, and genesis files #4903

Open
conorsch opened this issue Oct 23, 2024 · 1 comment
Open
Assignees
Labels
A-tooling Area: developer tooling for building Penumbra itself

Comments

@conorsch
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Reindexing workflows depend on access to hard-to-find data, like the original genesis file for a pre-upgrade chain. Also, newly joining nodes must bootstrap from node state backup.

Describe the solution you'd like
There should be a URL like https://artifacts.plinfra.net/ that hosts a directory tree of useful data, separated by chain.

Describe alternatives you've considered
Right now we just encourage folks to find the data on their own, like in the pindexer docs:

The <ORIGINAL_GENESIS_FILE> must be the genesis.json file for the chain, before any upgrades. Note, in particular, that after an upgrade, there will be a new genesis file containing only a checkpoint. pindexer specifically needs the original genesis file, because it needs to read information about the start of the chain, such as initial allocations, to track changes over time accurately.

Additional context
Here's a list of artifact types for consideration:

  • pre-upgrade node state, sanitized (i.e. node key material removed)
  • post-upgrade node state, sanitized (i.e. node key material removed)
  • reindexer_archive.bin for storing reindexer sqlite3 dbs
  • genesis.json files; especially important for chains that have upgraded
  • db dumps of event data? ideally not, since the above is sufficient to regenerate the rest, but might be useful as a reference

We don't need to go crazy about completeness, just having a directory to point to would greatly improve a lot of first-run experience.

@conorsch conorsch added the A-tooling Area: developer tooling for building Penumbra itself label Oct 23, 2024
@conorsch conorsch self-assigned this Oct 23, 2024
@github-actions github-actions bot added the needs-refinement unclear, incomplete, or stub issue that needs work label Oct 23, 2024
@conorsch
Copy link
Contributor Author

conorsch commented Nov 6, 2024

Started collecting this data here: https://artifacts.plinfra.net/ It's got subdirs for chains like penumbra-1 and penumbra-testnet-phobos-2. Not 100% complete, but it's a great start. Will continue to add more assets.

One thing we've lost track on is the ability for a new node to bootstrap onto a post-upgrade chain easily. We should revisit the --archive-url logic in pd to make that a bit easier.

@conorsch conorsch removed the needs-refinement unclear, incomplete, or stub issue that needs work label Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-tooling Area: developer tooling for building Penumbra itself
Projects
None yet
Development

No branches or pull requests

1 participant