Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ops cli changes in a separate pr #242

Closed
wants to merge 74 commits into from
Closed

Add ops cli changes in a separate pr #242

wants to merge 74 commits into from

Conversation

Nagaprasadvr
Copy link
Collaborator

  1. Add bubblegum changes
  2. add ops-cli changes

fanatid and others added 30 commits April 16, 2024 11:47
@fanatid @kespinola
fmt
ea5379e
@fanatid @kespinola
clippy
8b12316
@fanatid @kespinola
ci: add lock, fmt, clippy checks
974c90f
@fanatid @kespinola
include migration to workspace
788973e
@fanatid @kespinola
include das_api to workspace
7a870c2
@fanatid @kespinola
add global clippy lints
7e3f9ef
@fanatid @kespinola
use workspace
aa916a4
@fanatid @kespinola
add crate program_transformers
033873d
@fanatid @kespinola
nft_ingester: use program_transformers crate
1008f86
@fanatid @kespinola
remove not used deps
83b171e
@fanatid @kespinola
remove AccountInfo
9d8031e
@fanatid @kespinola
remove plerkle from program_transformers
096616d
@fanatid @kespinola
nft_ingester2: grpc2redis
ef0b0e9
@fanatid @kespinola
add redis streaming for ingester
076bffa
@fanatid @kespinola
create pg pool
d8673cf
@fanatid @kespinola
parse incoming message from redis
c93adff
@fanatid @kespinola
add force shutdown with signals
53fa481
@fanatid @kespinola
insert download metadata tasks
11b4151
@fanatid @kespinola
download-metadata subtask
8fca2c2
@kespinola @Nagaprasadvr
refactor: rename nft_ingest2
3d5eb18
@kespinola @Nagaprasadvr
fix: applying account and transction filters to grpc subscription req…
e50e9ba 
…uest
@kespinola @Nagaprasadvr
refactor: take out multiple connections to dragonmouth and then have …
973a599 
…the main thread keep a chache of seen events so that it doesnt push to ingest if seen by another connections
@kespinola @Nagaprasadvr
refactor: switch to accepting multiple dragonmouth endoints to mimic …
dd564e5 
…dual validators writing to ingest stream
@kespinola @Nagaprasadvr
fix: clippy errors
5a39d3f
@kespinola @Nagaprasadvr
chor: refactor grpc command to use topograph to clean up the control …
139ed01 
…flow. also pull in backfill to be used in ingest
@kespinola @Nagaprasadvr
chore: switch to run_v2 for ingest which handles ingesting accounts a…
0883efc 
…nd transactions
@kespinola @Nagaprasadvr
chore: handle metadata json within the ingest command
07b0936
@kespinola @Nagaprasadvr
test: metadata json stream processing
dff868b
@kespinola @Nagaprasadvr
feat: set num threads for topograph. report on postgres and redis in …
8f28a69 
…ingester using dedicate exec
@kespinola @Nagaprasadvr
chore: switch to topograph for thread management. create separate ing…
2cf9e57 
…est stream which handle reading messages and handling messages.
kespinola and others added 29 commits February 10, 2025 16:04
@kespinola @Nagaprasadvr
Remove topograph from ingest command (#167)
aea798a
@kespinola @Nagaprasadvr
Configure database connection idle timeout and max lifetime (#169)
fd58811
@kespinola @Nagaprasadvr
Bubblegum verify (#150)
1bc38e5 
* feat: move backfill to bubblegum crate and verfiy to it

* feat: grpc ingest monitor
@Nagaprasadvr @kespinola
Separate global stream to multiple Program separated streams and hand…
794fc32 
…le each stream (#168)

* feat : separate global stream to multiple streams depending on programs

* WIP : add metrics for each program subscription

* refactor: add structs with builder for SubscriptionTask and add better shutdown handling

* refactor : separate subscription builder to work on each subscription

* refactor: change pipeline to handle single stream instead of multiple streams

* Update: readme for grpc-ingest, index-readme, ops (#172)

* Update: readme for grpc-ingest, index-readme, ops

* Update : README

* fix: prometheus ports

* chore: resolve comments

* chore: add comments to config files in grpc-ingest (#173)

* change tag name of subscriptions

* Remove topograph from grpc2redis

---------

Co-authored-by: Kyle Espinola <[email protected]>
@kespinola @Nagaprasadvr
Report proofs as they come (#178)
3b9e68a
@kespinola @Nagaprasadvr
Buffer redis messages (#179)
18a6913
@kespinola @Nagaprasadvr
Fix tracking ack tasks
e53f7eb
@kespinola @Nagaprasadvr
fix: add back ack tasks total
39fb1b7
@kespinola @Nagaprasadvr
recreate consumer but not groups (#181)
ac053f4
@kespinola @Nagaprasadvr
Time Ingest jobs (#182)
51c5ae6 
* time the program transformer tasks

* add consumer labels to metrics
@kespinola @Nagaprasadvr
Use Example Configs (#180)
e317912 
Co-authored-by: Ahzam Akhtar <[email protected]>
@kespinola @Nagaprasadvr
Respect commitment level set in grpc config (#191)
7e302f1
@kespinola @Nagaprasadvr
Use tokio join set ingest stream task tracking
2e7efd1
@Nagaprasadvr @kespinola
Add metrics to track metadata_json_download success and failure (#207)
62f99f7 
Metadata download status metrics

---------

Co-authored-by: Kyle Espinola <[email protected]>
@Nagaprasadvr
add retry config to download_metadata_json task (#208)
d086bfc
@kespinola @Nagaprasadvr
Resolve clippy
741032a
@kespinola @Nagaprasadvr
Use multiple pipelines per subscription (#211)
df915d0
@Nagaprasadvr @kespinola
Das 106 add metadata json backfiller to das (#210)
86f223d 
* Update to rust 1.79.0

* feat : add metadata_json backfiller

* cleanup

* revert metaplex/plerkle-validator image to from ..79 to ..75

---------

Co-authored-by: Kyle Espinola <[email protected]>
@kespinola @Nagaprasadvr
Shutdown program on stream error or close (#213)
e2deabe
@kespinola @Nagaprasadvr
Global shutdown in the None arm
a46726d
@fernandodeluret @kespinola @Nagaprasadvr
Migrate to channel and worker pool for download notifier (#217)
9988bb2 
* `wip` concurrent download metadata notifier with channels

* `chore` Clean debugging changes

* `chore` remove duplicated config files and debug trace

* Refactor to download metadata publisher

* Time metric for metadata josn publish

---------

Co-authored-by: Kyle Espinola <[email protected]>
@Nagaprasadvr
add redis pipes for processing metadata download (#222)
03f5d88 
* add redis pipes for processing metadata download

* resolve comments
@Nagaprasadvr
chore : cleanup
8c4720d
@Nagaprasadvr
pull main
ac28e5e
@Nagaprasadvr
chore: pull main
8a9b835
@Nagaprasadvr
chore: remove bubblegum
d7dd0d0
@Nagaprasadvr
chore: remove clippy changes
1ad25e5
@Nagaprasadvr
chore: remove clippy changes
f29c5fe
@Nagaprasadvr
add bubblegum and ops-cli changes
c33e473
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Nagaprasadvr @fanatid @kespinola @fernandodeluret