Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[1.19][FLINK-37100][tests] Fix test_netty_shuffle_memory_control.sh with Netty4 RPC #25956

Merged
merged 2 commits into from
Jan 23, 2025
Merged

[1.19][FLINK-37100][tests] Fix test_netty_shuffle_memory_control.sh with Netty4 RPC #25956

merged 2 commits into from
Jan 23, 2025

Conversation

ferenc-csaky
Copy link
Contributor

@ferenc-csaky ferenc-csaky commented Jan 10, 2025 •
edited
Loading

What is the purpose of the change

Fixes the test executed by test_netty_shuffle_memory_control.sh that can possibly fail the CI in case Netty4 cannot reserve enough memory, hence Pekko is not able to start up.

Brief change log

  • Reverted the commit backported from the 2.0 branch, which I believe invalidates the purpose of this test case.
  • Increased the off-heap memory from 7BM to 12MB, which according to local testing has to be enough to stabilize the CI execution.

Verifying this change

Existing test should succeed consistently in CI.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? handled in different PR

@ferenc-csaky ferenc-csaky mentioned this pull request Jan 10, 2025
Closed
@flinkbot
Copy link
Collaborator

flinkbot commented Jan 10, 2025 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@ferenc-csaky
Copy link
Contributor Author

@flinkbot run azure

He-Pin
He-Pin reviewed Jan 11, 2025
View reviewed changes
Copy link
Member

@He-Pin He-Pin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

reswqa
reswqa approved these changes Jan 16, 2025
View reviewed changes
Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.

@ferenc-csaky
Copy link
Contributor Author

There is still a discussion going on about how exactly should we proceed, will merge this when we have agreement about every aspect.

@ferenc-csaky
Copy link
Contributor Author

@flinkbot run azure

@ferenc-csaky
Revert "[FLINK-36348][test] Fix the Netty shuffle direct memory test …
06d42a7 
…by increase the direct memory of TM"

This reverts commit 6d4004e.
@ferenc-csaky
[FLINK-37100][tests] Fix test_netty_shuffle_memory_control.sh with …
9cc2ca5 
…Netty4 RPC

With Pekko updated and using Netty4, the default memory buffer allocation
is different compared to Netty3, thus to stabilize this test we increased
the given memory a bit.
@ferenc-csaky ferenc-csaky requested a review from afedulov January 23, 2025 16:19
@ferenc-csaky ferenc-csaky changed the title [1.19][FLINK-37100][tests] Fix test_netty_shuffle_memory_control.sh in CI for JDK11+ [1.19][FLINK-37100][tests] Fix test_netty_shuffle_memory_control.sh with Netty4 RPC Jan 23, 2025
@ferenc-csaky ferenc-csaky merged commit 5ec21ad into apache:release-1.19 Jan 23, 2025
@ferenc-csaky ferenc-csaky deleted the FLINK-37100-1.19 branch January 23, 2025 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@He-Pin He-Pin He-Pin left review comments

@reswqa reswqa reswqa approved these changes

@afedulov afedulov afedulov approved these changes

Assignees
No one assigned
Labels
component=<none>
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ferenc-csaky @flinkbot @He-Pin @afedulov @reswqa