GoKawiilA while back we rolled a routine Envoy upgrade to one of our customers' dedicated proxies. Same config, same traffic, and the CPU graphs stepped up by roughly a fifth. This is the kind of chart that earns you a calendar invite with no agenda, so we decided to get ahead of it and started digging in.
Bisecting versions pointed at Envoy v1.34, which is where Envoy switched its default HTTP/2 codec from nghttp2 to Google's oghttp2. We weren't the first to notice: users had been reporting 15–25% latency regressions since that release, and v1.37.0 eventually flipped the default back, with a comment in the source promising to try again "once performance aligns with nghttp2." That fixed our customer. It did not fix our curiosity. Why doesn't it align? What does an HTTP/2 codec even spend its time on?
This post is the investigation. There's a sequel coming where we make the fast codec faster still, but first things first.
The best of codecs, the worst of codecs
Envoy has two implementations of HTTP/2 behind a runtime flag, though nobody could say which was which without a flamegraph:
nghttp2 - Tatsuhiro Tsujikawa's C library, the HTTP/2 workhorse since 2013 and Envoy's original codec. Lean C structs, caller-provided buffers.
- Tatsuhiro Tsujikawa's C library, the HTTP/2 workhorse since 2013 and Envoy's original codec. Lean C structs, caller-provided buffers. oghttp2 - Google's C++ codec from the QUICHE family, which shares code with the HTTP/3 stack. It became Envoy's default in v1.34, and that's when the regression reports started.
We benchmarked both on four microarchitectures (Intel Sapphire Rapids, AMD Zen 4, AWS Graviton4, Google Axion). Setup, briefly: same-host loopback (h2load → Envoy → Go h2c backend, or Envoy serving direct_response ), every process pinned to disjoint physical cores, Envoy at --concurrency 1 , so RPS/core is a pure CPU-efficiency number.
The user reports reproduce almost exactly: nghttp2 beats oghttp2 by 15–25% RPS/core on header-heavy proxied traffic, on every host, and by 7–19% under heavy connection churn. So the regression is real, portable, and lives somewhere in the codec. Time to open the profiles.
PSA If you build Envoy yourself, check your optimization flags. Envoy's .bazelrc does not default to -c opt - a plain bazel build //source/exe:envoy-static produces a fastbuild (debug) binary that looks completely functional and benchmarks 15–32× slower per core than the same source at -c opt . Every number in this post is from -c opt builds. We found this out the embarrassing way.
... continue reading