Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tetra: add a new debug maps command and move dump under debug #2959

Merged
merged 2 commits into from
Sep 30, 2024

Conversation

mtardy
Copy link
Member

@mtardy mtardy commented Sep 27, 2024

Here is the help for this new command:

Retrieve information about maps on the host related to tetragon.

- AllMaps: all the BPF maps loaded on the host
- PinnedProgsMaps: all the BPF maps used in programs pinned under /sys/fs/bpf/tetragon
- PinnedMaps: all the BPF maps pinned under /sys/fs/bpf/tetragon

The first table is the total of memlock bytes for the maps, those values are
accurate mostly from v6.4[^1].

The second table is in number of maps, inter, exter and union are the respective
math operations on the two first set, diff is the number of maps that were
detected as pinned maps but not used in pinned progs, ideally this number should
be 0. Note that the union should be the set of all the tetragon related maps on
the host if all the programs are properly pinned under the bpffs and all the
unused maps are also pinned under the bpffs.

The third table is the details of the maps that are in the diff, which means
maps that are pinned but unreferenced from pinned programs. Ideally, this table
should be empty.

The fourth table aggregate map by name and add the count column indicating how
many maps were aggregated for the given name. It ranks map from the one
consuming the most memory to the one consuming the less. Use the -n flag to
adjust the number of item in the table.

[^1]: https://lore.kernel.org/all/[email protected]/

Here is a typical output for tetragon loading with a TCP tracing policy:

AllMaps    PinnedProgsMaps   PinnedMaps
28981232   28972720          27344136

PinnedProgsMaps   PinnedMaps   Inter   Exter   Union   Diff
132               83           83      49      132     0

ID   Name   Type   KeySize   ValueSize   MaxEntries   Memlock

Name              Type          KeySize   ValueSize   MaxEntries   Count   TotalMemlock   PercentOfTotal
execve_map        Hash          4         632         32768        1       23073936       79.6%
socktrack_map     LRUHash       8         16          32000        1       2829376        9.8%
data_heap         PerCPUArray   4         32768       1            4       787744         2.7%
tg_execve_joine   PerCPUArray   4         8           2            2       722416         2.5%
tg_mbset_map      Hash          256       8           1024         1       338880         1.2%
string_maps_ro_   PerCPUArray   4         16384       1            3       295896         1.0%
string_maps_hea   PerCPUArray   4         16384       1            3       295896         1.0%
process_call_he   PerCPUArray   4         24312       1            1       146200         0.5%
retprobe_map      Hash          16        24          1024         1       108160         0.4%
buffer_heap_map   PerCPUArray   4         4352        1            4       105760         0.4%

@mtardy mtardy added the release-note/misc This PR makes changes that have no direct user impact. label Sep 27, 2024
@mtardy mtardy marked this pull request as ready for review September 27, 2024 15:57
@mtardy mtardy requested a review from a team as a code owner September 27, 2024 15:57
Copy link
Contributor

@kkourt kkourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

This looks really, useful!

Some questions; Is this part of bugtool? Should we make it?

Can we have JSON output? JSON output helps if we want to write scripts to parse the data which will be useful if we run this via sysdmp/bugtool across the whole cluster.

cmd/tetra/debug/maps.go Outdated Show resolved Hide resolved
Copy link
Contributor

@olsajiri olsajiri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, left few comments, thanks

cmd/tetra/debug/maps.go Outdated Show resolved Hide resolved
pkg/bugtool/maps.go Show resolved Hide resolved
cmd/tetra/debug/maps.go Outdated Show resolved Hide resolved
@mtardy
Copy link
Member Author

mtardy commented Sep 30, 2024

Thanks for the reviews

Some questions; Is this part of bugtool? Should we make it?

It's not currently as most commands parts of dump and I think it should.

Can we have JSON output? JSON output helps if we want to write scripts to parse the data which will be useful if we run this via sysdmp/bugtool across the whole cluster.

agree

This new command will allow us to debug the map situation and especially
regarding memory usage of those maps on host running tetragon.

Signed-off-by: Mahe Tardy <[email protected]>
Also creates a hidden alias to not break compatibility with dump being a
top command. But now we have the debug command to host those kind of sub
commands.

Signed-off-by: Mahe Tardy <[email protected]>
@mtardy
Copy link
Member Author

mtardy commented Sep 30, 2024

Thanks for the reviews

Some questions; Is this part of bugtool? Should we make it?

It's not currently as most commands parts of dump and I think it should.

Can we have JSON output? JSON output helps if we want to write scripts to parse the data which will be useful if we run this via sysdmp/bugtool across the whole cluster.

agree

I will merge that and add that in a follow-up PR.

@mtardy mtardy merged commit 22dde65 into main Sep 30, 2024
39 of 40 checks passed
@mtardy mtardy deleted the pr/mtardy/tetra-maps-details branch September 30, 2024 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/misc This PR makes changes that have no direct user impact.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants