GoKawiilEvery distributed system has different kinds of state
The mistake we made, early on, was asking the wrong question.
We kept asking: "Which consistency model should the cluster use?"
The useful question turned out to be: "Which consistency guarantee does this specific piece of state actually need?"
Those sound similar. They aren't. The first one assumes a single answer exists for the whole system. The second one assumes it doesn't — and it's the second one that this whole piece converges on, in the form of one table that, by the end, does most of the explaining for us.
We discovered the difference the hard way while building a clustered message broker. The lesson underneath has very little to do with MQTT. Any distributed system holding more than one kind of state runs into this question eventually. Get it right, and the rest of the design gets simpler almost automatically.
The incident
Two pods, five minutes apart, killed by the kernel's OOM handler. Memory limit: 512Mi. Nothing exotic — a normal container limit for a normal stateless service.
The first hypothesis was the obvious one: load-balancer imbalance, probably compounding into a reconnect cascade. Plausible. Wrong.
Active connections, three pods, same window:
... continue reading