GoKawiilIn the beginning, there was mmap. It was convenient: it let us lazily read huge numbers of Arrow IPC files from disk without managing memory ourselves. It fit our file format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you exactly that.
Then we deployed to production, ran real concurrent query loads, and mmap became a real problem.
Our Workload
At Conviva, we analyze trillions of events a day to pinpoint and diagnose end user experience. At the core of our architecture is an event and pattern analysis engine built on DataFusion, Arrow, Rust, Rayon, and Tokio. Raw events get transformed, encoded in a proprietary mostly-numeric format, and stored in the cloud. We copy them to local NVMe and read large (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and speed — its memory and disk layouts are identical, so decode cost is minimal, and mmap gives us zero-copy reads natively supported by arrow-rust. A typical query touches 6 columns across 8 batch files (one batch per file), ~1.6 GB per batch, ~13 GB total per day of data.
The Test Setup
Hardware: 192-core box, ~750 GB RAM. Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling). Kernel 5.15 during investigation, 6.x in production.
The Production Symptom
At lighter loads, mmap worked well — fast, serving queries from raw events in seconds. The trouble started under heavier concurrency. Some latency increase under load is expected — more queries competing for the same CPU. But we saw p95s and p99s spike well beyond what linear scaling would predict, with rows scanned per core dropping sharply even after accounting for concurrency:
OS page cache shrank — each pod consumed more memory as private allocations, less as shared cache
A huge number of page faults
... continue reading