dt: disable core dumps in crash tracker tests #25117
Conversation
I chose to set the core_pattern as it is the most reliable and simplest option for our current setup. Note that in our current CDT setup, core dumps are not written to a file directly by the kernel but instead use a piping mechanism where they pipe the coredump to `apport`. Because of this, the RLIMIT_CORE of the process (set through ulimit) is not respected, meaning ulimit alone cannot disable core dumps. For reference, see the "Piping core dumps to a program" section of https://www.man7.org/linux/man-pages/man5/core.5.html
pgellert
requested review from
rpdevmp,
oleiman,
a team and
michael-redpanda
and removed request for
a team and
oleiman
CI test resultstest results on build#62030
|
| @@ -57,6 +57,9 @@ def __init__(self, test_context): | |||
| ) | |||
| self.broker = self.redpanda.nodes[0] | |||
|
|
|||
| # Disable core dumps as they take a long time (>1min). Core dumps are uninteresting for this test, since this test intentionally trigger crashes. | |||
| self.broker.account.ssh("sysctl -w kernel.core_pattern='|/dev/null'") | |||
There was a problem hiding this comment.
Doesn't this have to be reverted at the end of the test?
Note that apport is a giant piece of garbage so there is a point for disabling/removing it in our ansible altogether (at which point ulimit should work again).
There was a problem hiding this comment.
Doesn't this have to be reverted at the end of the test?
yeh i am concerned about running this on my local machine
There was a problem hiding this comment.
Yeah, that's fair. This PR is meant to be a way of disabling apport, but yeah, good point about this affecting local machines and other tests. Let me reach out to devprod to ask them to disable apport or move to a non-piping core_pattern in CDT.
There was a problem hiding this comment.
cool. yeh i'm not opposed to this, but just as a drive-by review it seems like there might be some other ways to approach this like a one-off test outside the normal test harnesses. but not immediately sure the best thing to do
|
Putting this to draft mode while I discuss with devprod an alternative approach of making a CDT-only change to core_pattern / disabling apport. |
Currently, the
CrashLoopChecksTest.test_crash_report_with_signalducktape tests consistently fail in CDT. This is because when a crash signal is sent to the redpanda process during the test, in CDT a core dump gets generated, which takes a long time (>1min) and causes the test to time out waiting for redpanda to stop (in 10s).To fix this, this PR disables core dumps for
CrashLoopChecksTestducktape tests.See the commit message for implementation details.
Fixes https://redpandadata.atlassian.net/browse/CORE-9044
Tested this on PR #25100
Backports Required
Release Notes