Rich Packet Metadata - The Saga Continues

<p>So here's the deal: if you want to tag a packet (<code>sk_buff</code>) with some info that actually sticks around as it travels through the Linux network stack, your only real option right now is a measly 32-bit <code>SO_MARK</code> field. That's not a lot of room, and everyone's fighting over those bits. In this talk, we'll share Cloudflare's quest to get 128+ bytes of metadata attached to packets—enough space to do cool stuff like keeping config consistent across network layers, classifying packets in XDP, and figuring out exactly why a packet got dropped.</p> <p>We've been at this for a while, and we'll walk you through the three main chapters of our journey—the dead ends, the duct tape solutions, and what we think might actually work:</p> <p><strong>Part One: SKB Traits.</strong> Remember that per-packet key-value store idea called "SKB Traits"? We'll tell you why we got excited about it and why we eventually tossed it out in favor of not having a second metadata area.</p> <p><strong>Part Two: XDP Metadata.</strong> Next up, we tried piggy-backing on <code>skb-&gt;data_meta</code> and brought in <code>bpf_dynptr</code> to wrangle the metadata. Spoiler: it gets messy. L2 tunnels and pulling packet headers can leave your metadata corrupted or just plain gone. We'll dig into our attempts to untangle metadata tracking from MAC header offsets and the backward compatibility headaches that come with it.</p> <p><strong>Part Three: SKB Extension.</strong> Finally, we'll show you our latest idea: using <code>sk_buff</code> extensions (<code>skb_ext</code>) backed by BPF local storage. The nice thing here is that metadata lives separately from packet headers. But we're not out of the woods yet—memory allocation on the hot path is still an open problem.</p>