GoKawiilMost observability stories start with a production incident. This one starts with a bad shot of espresso.
If you've never chased good espresso, here's the problem in one paragraph. Starting out, it feels simple: grind the beans, tamp them flat, run hot water through at pressure. But the moment coffee stops being purely functional and becomes something you want to perfect, you discover the entire universe is working against you. Humidity changes how your beans grind overnight. Grind size changes how fast water moves through the puck. Pressure shapes which flavors you extract, and extraction time decides whether you land on sweetness or on something bitter and burnt. The beans themselves are a moving target, aging week to week. Every one of these variables affects the others, and your only feedback is a taste and a vague memory of yesterday's attempt.
What I'm trying to do here is what you'd do to any misbehaving system: measure as many of those variables as I can, store them somewhere I can query, and start controlling them one at a time.
I have a Gaggia espresso machine running GaggiMate, an open-source ESP32 controller that replaces the stock brain with a touchscreen, a PID loop, and pressure profiling. It's a genuinely impressive bit of kit. But when a shot came out sour or gushed through the puck, I had no idea why. Was the grind too coarse? Did the boiler temperature sag mid-pull? Did water channel through one side of the puck instead of extracting evenly?
These are exactly the questions observability is built to answer, for distributed systems anyway. So I treated the espresso machine like any other service I'd want to debug in production: I instrumented it with OpenTelemetry, shipped the telemetry to ClickHouse Cloud, and put ClickStack on top of it.
It turns out an espresso machine is a surprisingly honest distributed system, and the same stack that runs observability for real infrastructure handles a microcontroller pulling a 30-second shot without breaking a sweat. This is the nerdy version of that story: the protobuf, the FreeRTOS scheduling, the ring buffers, the custom collector, and the agent that ties it together.
Why ClickStack for an ESP32? #
ClickStack is ClickHouse's observability stack: an OpenTelemetry collector for ingest, ClickHouse for storage and queries, and HyperDX as the UI for logs, metrics, traces, and session replay. The whole point is that it's schema-agnostic and OTel-native. It doesn't care whether telemetry comes from a Kubernetes fleet or a coffee machine, as long as it speaks OTLP.
That last part is what made this project viable. I didn't have to invent a data model. A shot of espresso maps cleanly onto concepts ClickStack already understands:
Metrics: boiler temperature and target, boiler pressure (bar), pump flow rate, puck resistance, scale weight (grams), plus cumulative counters for total shots, brew time, water consumed, and connectivity events.
... continue reading