How to build a distributed queue in a single JSON file on object storage February 12, 2026 • Dan Harrison (Engineer)
We recently replaced our internal indexing job queue, which notifies indexing nodes to build and update search indexes after data is written to the WAL. The queue is not part of the write path; it's purely a notification system used to schedule asynchronous indexing work. The prior version sharded queues across indexing nodes, so a slow node would block all jobs assigned to it even if other nodes were idle. The new version uses a single queue file on object storage with a stateless broker for FIFO execution, at-least-once guarantees, and 10x lower tail latency versus our prior implementation, so indexing jobs spend less time in the queue.
Loading chart data...
Why are we so obsessed with building on object storage? Because it's simple, predictable, easy to be on-call for, and extremely scalable. We know how it behaves, and as long as we design within those boundaries, we know it will perform.
Rather than present the final design of our new queue from the top down, let's build it from the bottom up, starting with the simplest thing that works and adding complexity as needed.
Step 1: queue.json
The total size of the data in a turbopuffer job queue is small, well less than 1 GiB. This easily fits in memory, so the simplest functional design is a single file (e.g., queue.json ) repeatedly overwritten with the full contents of the queue.
A queue pusher reads the contents of the queue, appends a new job to the end, and writes it using compare-and-set (CAS).
A queue worker similarly uses CAS to mark the first unclaimed job as in progress (○ → ◐), and then gets to work.
We'll call pushers and workers clients, and push and claim operations requests.
... continue reading