Skip to content
Tech News
← Back to articles

Debugging Information for Inlined Functions

read original more articles
Why This Matters

Enhancing debugging for inlined functions in the Linux kernel is crucial for more accurate tracing and performance analysis. By adding inlining information to BPF type format (BTF), developers can better trace kernel functions, even when they are inlined, improving debugging precision and system reliability. This advancement benefits both kernel developers and users by enabling more effective troubleshooting and optimization of Linux systems.

Key Takeaways

We're bad at marketing We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.

BPF programs use BPF type format (BTF) debugging information in order to determine how to interact with functions in the kernel. Specifically, tracing a kernel function involves finding its address in the kernel's BTF section — but that doesn't work for functions that have been inlined, and therefore don't have a single, specific address. Alan Maguire wants to add information about inlined functions to BTF in order to allow them to be traced, and led a session on that topic at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit.

There are more than 100,000 inlined functions in the kernel, Maguire said, spread across five times as many locations. Worse, some of them are partially inlined: called normally in some places and inlined in others. That can lead to cases today where it appears that a function was traced successfully, but some invocations were not seen.

The good news is that the rest of the infrastructure for tracing inlined functions is already in place to enable kprobes, which can be attached to arbitrary locations. It is just a matter of getting the data about where functions have been inlined into a usable format, Maguire stated. " The story is actually pretty complete. "

So, what is needed to store this information in BTF? The DWARF debugging format already has a way to indicate inlining information, but DWARF is also difficult to work with, and doesn't have a simple way to represent the common cases. A solution for BTF should be compact, and permit deduplication, to keep the memory overhead low, Maguire said. Ideally, inlining information could be stored in a separate section of the kernel binary, or even be distributed as a separate kernel module, so that it is not loaded until it is needed.

Concretely, Maguire proposed three new pieces of information be added to BTF. The first was inline-site-specific information about which function was inlined at each call site and how it would have been called if it had not been inlined, called the "location section". That data can't be easily deduplicated because it's specific to a given call site, so as much as possible the information should consist of pointers to data that can be deduplicated.

The second and third pieces of information he wants to add would be the pointed-to data: a "location prototype" and "location parameter". The location prototype specifies how the inlined function's arguments are represented at the call site, as a list of pointers to location parameters, which each store how to access a single function argument. In theory, the compiler could store a given function parameter at a different location for every inlined call site (of which there are 538,090); in practice, there are a limited number of ways that the compiler will transform parameters, and many functions have compatible signatures that result in the compiler making the same choices. In the current kernel, deduplicating location prototypes results in just 57,141 distinct entries referencing only 17,535 location parameter entries in total.

This means that the location sections take up most of the added data. Overall, the additions to BTF that Maguire proposes would come to approximately 11MB of additional data, or about 21 bytes per inlined call site. When pulled out into a separate kernel module and compressed, the total amount of data goes down to 3.5MB.

Maguire then went through an example of how information about a specific function would be stored. Consider this function:

int foo(int a, void *b, bool c);

... continue reading