Combining Trace(r)s: Kernel ftrace & LTTng UST

<p>Tracing complex systems often requires insights from both the kernel and userspace. While tools like Linux's ftrace excel at kernel-level observability and LTTng provides low-overhead userspace tracing, unifying these disparate data sources for a holistic view remains a challenge: using LTTng for kernel tracing requires an out-of-tree kernel module, which can be a barrier for many users.</p> <p>This talk introduces bt2-ftrace-to-ctf - a new open-source project designed to bridge this gap. Our solution processes a trace.dat file from ftrace (kernel part) and a LTTng UST for userspace, then aligns and rewrites the trace in the Common Trace Format (CTF), as used by LTTng. The resulting output is directly consumable by tools like Trace Compass, enabling comprehensive, synchronized analysis of system behavior across all layers without the need for custom kernel modules.</p> <p>The project consists of two key components:</p> <ul> <li>A Babeltrace2 plugin: This plugin allows babeltrace2 to directly read trace-cmd's trace.dat files, providing a standardized interface for ftrace data. It includes source and sink components for flexible data handling and metadata emission.</li> <li>A trace.dat to CTF converter: This utility utilizes the Babeltrace2 plugin to transform ftrace data into an LTTng-alike kernel trace in CTF format. Crucially, it can also combine this kernel trace with an existing LTTng userspace trace, producing a single, unified trace directory.</li> </ul> <p>In the talk, we will give an overview on the tool and discuss challenges during its implementation.</p> <p>Project: https://github.com/siemens/bt2-ftrace-to-ctf (MIT, LGPL-2.1-or-later)</p>