Bug: Data stored it in ".fin7" or ".fin9 is ignored when calculating how much flash is used

[EmperorArthur]

Background

We are all familiar with F strings and the PROGMEM attribute. However, on devices with over 64k of flash, another problem emerges. The pointers are only 16 bits, so the flash memory is paged.

The normal fix is to do:

unisgned char * some_data PROGMEM = {<Large Amount of Data Here>};
#define some_data_real pgm_get_far_address(some_data)

With only some_data_real ever being used to obtain a 32bit pointer.

However, injecting large amounts of data can result in an F string jumping page boundaries, or other strange issues. Technically, this is a problem with the Ardunio core itself, and requires fixes. However, there is a work-around.

#define PROGMEM_END __attribute__((section(".fini7")))
unisgned char * some_data PROGMEM_END = {<Large Amount of Data Here>};
#define some_data_real pgm_get_far_address(some_data)

This puts the embedded data at the end of the flash section, and prevents those issues.

The Problem

Unlike the PROGMEM macro, __attribute__((section(".fini7"))) does not show as using flash. Given that this code is mostly used on things consuming multiple Kilobytes of flash that is a real problem.

If I am not careful, I could easily run out of flash, with the build system showing plenty of free space remaining!

References

https://sudonull.com/post/11901-When-in-gcc-16-bit-addresses-and-memory-suddenly-256k
https://forum.arduino.cc/index.php?topic=134649.0

[maxgerhardt]

Linking will fail when the stored data goes beyond the boundaries, but it's indeed not shown as data usage.

The problem is that the regex

platform-atmelavr/builder/main.py

Lines 104 to 105 in 8d58569

	SIZEPROGREGEXP=r"^(?:\.text|\.data|\.bootloader)\s+(\d+).*",
	SIZEDATAREGEXP=r"^(?:\.data|\.bss|\.noinit)\s+(\d+).*",

Only captures some sections in the ELF file towards the flash (/program) and RAM (/data) usage. That could be updated.

[EmperorArthur]

Linking will fail when the stored data goes beyond the boundaries

Not sure what you mean by this, unless those sections have explicit boundaries. I am storing well over 64k of data using this method. Admittedly in 32k-1 (max ptr) chunks, and have not run into a problem yet.

The problem I was seeing when using PROGMEM was garbage at runtime. Sometimes it was in the data being stored*, and other times it was an F string issue. The F strings are "fun" since they're zero terminated, so random data gets read until the next 0x00 in flash.

I am on the fence of if the underlying cause is an Arduino or a compiler bug. My guess, having not looked at the elf/hex in too much detail, is that all PROGMEM / F strings are stored together, and that enough code alone will probably trigger the bug.

This was a work problem though, and the owners aren't the type to let me take the time to really dig into the issue. I don't have an AVR with enough memory to ever see this problem at home, so this will probably be the end of it for now.

* Before I started using 32 bit pointers.