GoKawiilJob queues are deceptively tricky
One of the fun things about being a programmer is that as I look more into systems that I didn’t know much about, what superficially appears to be a simple system actually reveals interesting facets of underlying complexity. In other words, reality has a surprising amount of detail.
In this post, I want to talk about job queues, which I’ve been thinking about for the past few days (and while drafting this post in my head, I realized I’d thought about them for longer at a previous job, but not nearly with as much clarity.)
What do I mean by “job queue”? I mean a system where there is some notion of submitting batch jobs, scheduling them, and running them. Generally, the system is expected to be FIFO or FIFO-like, but that’s not required. Usually, the queue bundles together a native way to schedule jobs periodically – this lets you specify submissions using configuration files (e.g. using JSON or YAML).
Job queues arise naturally in all sorts of situations where throughput requirements are high, but latency requirements are not that high. Continuous integration is a common example. Another example is summaries for the purpose of data analysis.
Over the past year or two, a handful of lenses that I’ve found useful when thinking about system design are:
Being wary of queues: Queues tend to typically be close to full or close to empty, requiring some thought around appropriate sizing. I’ve also learnt a bunch of counter-intuitive behavior of queues when it comes to latency from Marc Brooker’s blog. In blog posts I’ve read about queues, the focus is generally on latency, not on throughput, which is I think partly why I never really mentally linked “queues” and “job queues” earlier.
Limits: This is based on the Tiger Style guide written by the folks at TigerBeetle, which talks about explicit limits on various things. On generalization, this leads to the notion of tolerances and having budgets for modular reasoning. Funnily enough, near the start of my career at Apple, I remember being mildly bemused at certain discussions where teams would discuss how much memory budget they would have to bargain for, especially for low-level code. I have grown more respect for that approach over time.
... continue reading