If You Skip the Kernel, You Have to be the Kernel
XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modifications.
This post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.
At Loophole Labs, we live migrate everything - containers, VMs, and even network connections.
During a migration every single packet for a workload needs to be intercepted, modified, encapsulated, encrypted, and rerouted to its new destination - all without the application noticing. Our scale requires us to be able to move workloads across clouds at hundreds of gigabits per second - and with that sort of performance requirement, every single CPU cycle matters.
All of this is to say, we need to be able to process packets at line-rate (however much the underlying network can support, whether that's 20Gbps or 200Gbps), and there's really only one approach that lets us do that:
Linux Packet Processing Performance Comparison
In Linux, the gold standard for high-performance packet processing is XDP (eXpress Data Path). By intercepting packets as soon as they arrive at the network driver (before reaching the kernel) XDP is able to achieve line-rate speeds in most environments.
Our own benchmarks above show how easily we were able to reach line-rate with XDP, not to mention the fact that major companies like Meta, Cloudflare, and GCore have already been using it for more than 5 years now to handle 10s of millions of packets per second.
Unfortunately XDP has one fundamental flaw that everyone accepts as fact: it only works for ingress (incoming) traffic. This isn't a bug or an oversight - it's the entire identity of XDP, one of the main characteristics that define it. XDP only processes packets on ingress. Period.
... continue reading