how to read global variables directly in a user-mode program

[wubo0067]

In libbpf project，file cachestat.bpf.c global variables are defined：

__s64 total = 0;	/* total cache accesses without counting dirties */
__s64 misses = 0;	/* total of add to lru because of read misses */
__u64 mbd = 0;  	/* total of mark_buffer_dirty events */

In user-mode code, you can directly read variable values through atomic operations without passing them through defining maps.

	while (1) {
		sleep(env.interval);

		/* total = total cache accesses without counting dirties */
		total = __atomic_exchange_n(&obj->bss->total, 0, __ATOMIC_RELAXED);
		/* misses = total of add to lru because of read misses */
		misses = __atomic_exchange_n(&obj->bss->misses, 0, __ATOMIC_RELAXED);
		/* mbd = total of mark_buffer_dirty events */
		mbd = __atomic_exchange_n(&obj->bss->mbd, 0, __ATOMIC_RELAXED);

		if (total < 0)
			total = 0;
		if (misses < 0)
			misses = 0;
		hits = total - misses;

The c code generated by using bpftool gen skeleton can produce a bss structure.

	struct xm_cachestat_bpf__bss {
		volatile __s64 xm_cs_total;
		volatile __s64 xm_cs_misses;
		volatile __s64 xm_cs_mbd;
	} *bss;

Can I use cilium ebpf’s go package to implement logic similar to the above c code? I feel this would be more efficient

[wubo0067]

I traced the code and found that global variables are placed in the .bss section, and the type of .bss section is SHT_PROGBITS, whose reference=0. The following code will ignore creating DataSection, so can’t access global variables. What is the reason for doing this?

func (ec *elfCode) loadDataSections(maps map[string]*MapSpec) error {
	for _, sec := range ec.sections {
		if sec.kind != dataSection {
			continue
		}

		if sec.references == 0 {
			// Prune data sections which are not referenced by any
			// instructions.
			continue
		}

[ti-mo]

.bss is where zero-initialized global variables go. Variables initialized to any other value go into .data by default.

This skip is here due to compilers sometimes emitting empty data sections that don't have any relocations against them, meaning they won't be linked into bpf programs and will be immediately closed after being created. Writing any values to these is ineffectual, as they're never accessed by any bpf progs.

You're likely barking up the wrong tree. If you have a zero-initialized global var that gets accessed by bpf code, your Collection(Spec) will contain a .bss map that can be modified at will.

[ti-mo]

What you're describing is an example of mmapable maps. In the example you pasted obj->bss->total is a pointer into an mmap()ed piece of memory that is used to back key 0 of the .bss map.

If you want to implement something similar, from what I understand from the libbpf code, you should be able to write a pointer to an mmaped memory region to a map key. I think only non-per-cpu arrays are supported currently. You can use Go's atomic package to safely modify the contents of the mmaped region.

I'm not sure why exactly this is useful; using atomics on the bpf side should be sufficient for safely incrementing the counters from multiple concurrent progs. From userspace, a normal map get should be safe. As far as I understand, accessing bss from userspace needs to be done using atomics since libbpf defaults to using mmap for 'internal' maps, like bss and kconfig, although the latter is made read-only after loading.

[wubo0067]

@ti-mo Sorry for the interruption! If there are accessible variables in .data, then i can be modified at runtime in userspace, thereby changing the kernelspace execution strategy. Of course, now it is possible to modify them through mapping, but the code logic is not as straightforward."

[mscastanho]

Just adding another data point to the discussion: I also find the ability to modify global variables from userspace very useful and is a feature I miss from cilium/ebpf.

For my use case, I define a bunch of global variables that act as knobs to modify the behaviour of an XDP program running in the kernel. So the user space code is able to modify the value of these variables at any time by altering the bss map directly.

The downside of not having the mmaped memory region to interact with is that when issuing an update via BPF syscall, we are forced to write to all variables at once, even though we are only modifying one. If for some reason (not my case) these global variables should be modified by both userspace and kernel, then updating the entire struct like this could overwrite previous updates done by the kernel.

Another side issue is that struct definitions for the bss section are unfortunately filtered out during bpf2go code generation, so even if we want to go the "update via syscall" route, it's harder without the automatic struct def generation =(

[lmb]

I agree that it's nice to use, but I'm also not sure how to make it safe in the Go language. The libbpf stuff handwaves away a lot of atomicity problems (are you always doing READ_ONCE / WRITE_ONCE or friends?) and libbpf also doesn't have to deal with the garbage collector and the rules around unsafe.Pointer. So it's pretty unlikely that we'll can build a mechanism where you can do *vars.foo = 123.

Good point that currently there is no way to update only a single variable. That's also something holding back bpf2go where you might only want to override some constants, not all of them. It might be possible to wrap this via an explicit API func UpdateVariable(name string, val Int) error which does mmap + sync.Atomic behind the back. But would this live on Map or Program? Drop by our Slack channel if you are motivated to work on this!

[ti-mo]

This was implemented in #1572 and will be shipped in 0.17 to be released later this year or early 2025.