Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs get lost under high load #100

Open
slinkydeveloper opened this issue May 15, 2023 · 4 comments
Open

Logs get lost under high load #100

slinkydeveloper opened this issue May 15, 2023 · 4 comments
Milestone

Comments

@slinkydeveloper
Copy link
Collaborator

@slinkydeveloper slinkydeveloper added this to the 1B milestone May 15, 2023
@slinkydeveloper
Copy link
Collaborator Author

One idea could be to collect logs just at the end of the tests, but this apparently doesn't always work as some logs might have been lost in the meantime: https://github.com/restatedev/e2e/tree/trying_to_change_logging_tools

@slinkydeveloper
Copy link
Collaborator Author

slinkydeveloper commented May 15, 2023

Another completely different solution for logs would be to just pick them up from journald (which keeps them even after the container is removed). Podman by default sends logs to journald, and docker supports that as well https://docs.docker.com/config/containers/logging/journald/ (although i don't know what is the default there). But we need to make sure this solution works on MacOS (and how complicated is to setup wrt privileges)

@tillrohrmann
Copy link
Contributor

Can this be closed @slinkydeveloper? I remember faintly that you've solved it by switching to some other log daemon or so.

@slinkydeveloper
Copy link
Collaborator Author

On my machine yes, but i'm not sure this affects CI as well. Let me keep it open, and revisit once I enable CI in other repos as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants