Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libpf: panic: runtime error: slice bounds out of range [456:0] #33

Open
gipi opened this issue Jan 16, 2025 · 1 comment
Open

libpf: panic: runtime error: slice bounds out of range [456:0] #33

gipi opened this issue Jan 16, 2025 · 1 comment

Comments

@gipi
Copy link

gipi commented Jan 16, 2025

In our environment parca-agent is crashing with the following stacktrace

Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: time="2025-01-16T13:49:38Z" level=info msg="eBPF tracer loaded"
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: panic: runtime error: slice bounds out of range [456:0]
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: goroutine 102 [running]:
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/readatbuf.(*Reader).ReadAt(0xc004da1770, {0xc0054f0600, 0x1f0, 0x1f0}, 0xc004eb7220?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/readatbuf/readatbuf.go:119 +0x277
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).ReadAt(0xc00552ebc8, {0xc0054f0600, 0x1f0, 0x1f0}, 0x0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:640 +0x85
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).Data(0xc00552ebc8, 0xc006bce980?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:681 +0xa8
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*Prog).DataReader(0x201e720?, 0xc000013008?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:687 +0x17
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.newFile({0x2902840, 0xc004da1770}, {0x2900560, 0xc00007eed8}, 0x0, 0x0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:275 +0x70d
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.Open({0xc005512180?, 0x10?})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:171 +0x71
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/process.(*systemProcess).OpenELF(0xc005596870, {0xc005277a40, 0x6c})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/process/process.go:253 +0x17c
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/libpf/pfelf.(*File).OpenDebugLink(0x1d?, {0xc004e80500, 0x32}, {0x2902880, 0xc004dab5c0})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/libpf/pfelf/file.go:607 +0x17b
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.(*elfExtractor).extractDebugDeltas(0xc004eb7640)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaextraction.go:97 +0x35
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.ExtractELF(0xc004dab5c0, 0xc000012fa8)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaextraction.go:152 +0x247
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/nativeunwind/elfunwindinfo.(*ELFStackDeltaProvider).GetIntervalStructuresForFile(0xc00406a010, 0xc003f5fe60?, 0xc004dab5c0, 0x0?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/nativeunwind/elfunwindinfo/stackdeltaprovider.go:40 +0x27
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager/execinfomanager.(*ExecutableInfoManager).AddOrIncRef(0xc000714690, 0x4f1b206233f1e03b, 0xc004dab5c0)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/execinfomanager/manager.go:188 +0x152
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).handleNewMapping(0xc00406c840, {0x29321c8, 0xc005596870}, 0xc004eb7998, 0xc004dab5c0?)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:219 +0x59
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).processNewExecMapping(0xc00406c840, {0x29321c8, 0xc005596870}, 0xc003c23fc8)
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:344 +0x456
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).synchronizeMappings(0xc00406c840, {0x29321c8, 0xc005596870}, {0xc003c21008, 0xf6, 0x13f})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:462 +0x579
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/processmanager.(*ProcessManager).SynchronizeProcess(0xc00406c840, {0x29321c8, 0xc005596870})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/processmanager/processinfo.go:590 +0x2a7
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: github.com/elastic/otel-profiling-agent/tracer.(*Tracer).processPIDEvents(0xc003f4cdd0, {0x2924aa8, 0xc000909a00})
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/tracer/events.go:53 +0x1c7
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: created by github.com/elastic/otel-profiling-agent/tracer.(*Tracer).StartPIDEventProcessor in goroutine 1
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]:         github.com/elastic/[email protected]/tracer/events.go:42 +0x7a
Jan 16 13:49:38 perf-30-proxy1.openstacklocal parcaagent[63568]: time="2025-01-16T13:49:38Z" level=error msg="======================= unexpected error ======================="

Adding logging I have found out that the problem is that len(data) is equal to zero. Probably is a problem affecting also upstream.

Let me know if you have any hint regarding debugging this, for now I added locally a break if len(data) == 0 to be able to profile.
@gipi
Copy link
Author

gipi commented Jan 17, 2025

After some investigation seems like that the file is truncated: is trying to read the DYNAMIC segment

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000001640 0x0000000000001640  R      0x1000
  LOAD           0x0000000000002000 0x0000000000002000 0x0000000000002000
                 0x0000000000000139 0x0000000000000139  R E    0x1000
  LOAD           0x0000000000003000 0x0000000000003000 0x0000000000003000
                 0x00000000000066bc 0x00000000000066bc  R      0x1000
  LOAD           0x000000000000a670 0x000000000000b670 0x000000000000b670
                 0x0000000000000eb0 0x0000000000000eb8  RW     0x1000
  DYNAMIC        0x000000000000adc8 0x000000000000bdc8 0x000000000000bdc8
                 0x00000000000001f0 0x00000000000001f0  RW     0x8
  NOTE           0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000020 0x0000000000000020  R      0x8
  NOTE           0x00000000000002c8 0x00000000000002c8 0x00000000000002c8
                 0x0000000000000024 0x0000000000000024  R      0x4
  GNU_PROPERTY   0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                 0x0000000000000020 0x0000000000000020  R      0x8
  GNU_EH_FRAME   0x000000000000960c 0x000000000000960c 0x000000000000960c
                 0x000000000000002c 0x000000000000002c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x000000000000a670 0x000000000000b670 0x000000000000b670
                 0x0000000000000990 0x0000000000000990  R      0x1

at offset 0xadc8 but if I check the actual file size is 0x55f0. Probably a check regarding eof reached with zero data read should print a warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gipi