Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: various improvements in pusher, pushsync, salud, reacher #4958

Merged
merged 13 commits into from
Feb 3, 2025

Conversation

istae
Copy link
Member

@istae istae commented Jan 23, 2025

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

  1. Minor clean up in reacher.go and increased go worker count
  2. Light nodes error out if a reserve capacity doubling is set in the config
  3. in the pusher, as an optimization, instead of a full stamp validation which is expense, we instead only check if the batch is still alive. Also increased max parallel push count from 100 to 128.
  4. in the pushsync protocol, forward nodes no long perform receipt validation. Also, for the shallow receipts check, we apply some tolerance to the proximity of the store node and chunk based on the maximum allowed reserve doubling. Tag reporting was also being made for direct uploads, this is has been changed to only happen for deferred uploads by checking the chunk's tag ID.
  5. In the salud service, we sort the peer list in desc order by protocol response duration so that the faster peers are given priority if a bin's population drops below the salud threshold.
  6. increased max allowed cache worker count from 16 to 128.
  7. as an optimization, the kademlia closest peer terminates early if the iterator has already a closest peer and the current bin count is greater than the chunk's proximity with the node's address.

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

@istae istae marked this pull request as ready for review January 27, 2025 22:43
@istae istae requested a review from acha-bill as a code owner January 27, 2025 22:43
@istae istae changed the title fix: various improvements in pusher, pushsync, salud, reacher perf: various improvements in pusher, pushsync, salud, reacher Jan 31, 2025
@istae istae requested a review from gacevicljubisa February 3, 2025 11:55
Comment on lines -288 to -289
case errors.Is(err, ErrShallowReceipt):
fallthrough
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now it will go to default?
If this is ok, should we have a unit test for this case? both err==nil and default case are not covered by unit tests...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forwarder nodes no longer perform receipt checks, this is why the shallow receipt error check is removed.
see https://github.com/ethersphere/bee/pull/4958/files#diff-522faf538629254cbcb2fed372ba3f9c216497f868f45f6bd82cd6d5d6143c69R471

if err != nil {
logger.Warning(
"stamp with is no longer valid, skipping direct upload for chunk",
ok, err := s.batchExist.Exists(op.Chunk.Stamp().BatchID())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If batch not found it will return nil? Is this also ok?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, for direct uploads, the chunk resync is retried if this error is not nil. because batch is no longer valid at this point, we should stop syncing, hence the nil return

@istae istae merged commit 92057ff into master Feb 3, 2025
15 checks passed
@istae istae deleted the push-fixes branch February 3, 2025 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants