Tech News
← Back to articles

Debugging Humidity: Lessons from deploying software in the physical world

read original related products more articles

The first time I deployed code to an actual factory floor, I learned that "edge compute" doesn’t live in climate-controlled racks. It lives next to dust, grease, and forklifts. The idealized world of distributed systems I had studied-clean APIs, reliable networks, consistent power-was a comforting fiction.

The reality was far more brutal. Power flickers. Networks throttle to near-zero bandwidth because a welder turns on. Devices disappear mid-packet because a forklift driver unplugs them to charge their phone.

Every assumption from a decade in cloud engineering melts the moment you connect to something that moves.

The Cloud is a Lie (in the Physical World)

In the cloud, we build on abstractions designed to create infinite, reliable resources. On the factory floor, you have finite, hostile resources. The mismatch is violent.

Idempotency isn't about APIs, it's about Physics. A common pattern in distributed systems is the idempotent request. If a POST request fails, just send it again. The server is designed to handle the duplicate.

Now, imagine your request is actuator.rotate(90) . The network cuts out after you send it. Did the arm rotate? You have no 200 OK . Do you send it again? If the first one worked, you're about to crash a multi-thousand-dollar robotic arm by rotating it another 90 degrees. Suddenly, you're not debating RESTful principles; you're implementing a complex state reconciliation protocol over an unreliable network just to prevent physical damage.

"Offline-first" is not a UX pattern; it's survival. On a mobile app, "offline-first" means a user can like a photo without a connection, and it syncs later. That’s nice. On an assembly line, the PLC controlling a conveyor belt must function for its 8-hour shift even if the facility's internet goes down completely. The device cannot be a thin client for the cloud. It needs local autonomy. Computation, state, and decision-making have to live on the metal because the link to the mothership is a privilege, not a right.

Time is a suggestion, not a constant. You learn that logs are useless when your system clocks drift by three seconds between devices. We had two sensors that were supposed to fire in sequence. According to the aggregated logs on our server, sensor B was firing before sensor A. We spent a week tearing our hair out over a supposed race condition in our logic. The problem wasn't the code; it was cheap hardware with no real-time clock and an overwhelmed, drifting NTP client. The physical world is hostile territory for software ideas.

... continue reading