Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve pindexer speed #4908

Open
cronokirby opened this issue Oct 29, 2024 · 0 comments
Open

Improve pindexer speed #4908

cronokirby opened this issue Oct 29, 2024 · 0 comments
Labels
C-enhancement Category: an enhancement to the codebase needs-refinement unclear, incomplete, or stub issue that needs work

Comments

@cronokirby
Copy link
Contributor

It takes too long to sync up pindexer on a raw database: on the order of hours. The processing logic we do is not so complicated as to make this take this long, and I suspect that the architecture of processing one event at a time and touching the database for every event makes this slow.

I think we should move to have a "per batch of blocks" processing logic, only reading the database before the batch, and writing after the batch.

This also has the advantage of making certain app views simpler, because right now they would like to process all the events in a block at once, e.g. to know the timestamp for each event, but have to resort to tricks like using an in database queue instead.

@cronokirby cronokirby added the C-enhancement Category: an enhancement to the codebase label Oct 29, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Penumbra Oct 29, 2024
@github-actions github-actions bot added the needs-refinement unclear, incomplete, or stub issue that needs work label Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category: an enhancement to the codebase needs-refinement unclear, incomplete, or stub issue that needs work
Projects
Status: Backlog
Development

No branches or pull requests

1 participant