GoKawiilUber builds context-aware retry system to stop cascading service failures
Uber engineers describe a new mechanism built into shared infrastructure that makes retries 'context-aware,' allowing services to distinguish between errors originating locally versus those merely passed through from downstream dependencies. The system aims to replace uniform, blanket retry policies with conditional logic that adapts based on the severity of degradation across Uber's deep, fan-out-heavy microservice call chains.
GoKawiil's interpretation of the reporting above, not reported fact.
At massive scale, uncoordinated retries can turn a single failing service deep in a dependency chain into a stack-wide outage, since every upstream caller multiplies retry traffic against an already struggling downstream node. Existing fixes like retry budgets and manual tuning don't scale across Uber's constantly evolving call graphs, so automating context-awareness in shared infrastructure reduces the risk of retry storms degrading the broader platform and user experience.
- Uniform retry policies can backfire during moderate-to-severe outages by amplifying load on already-struggling services.
- Uber built a context-aware retry mechanism in shared infrastructure rather than relying on manual per-service tuning.
- The system addresses difficulty distinguishing locally generated errors from errors merely propagated through a service in complex fan-out call chains.
Source: uber.com, 2026-09-17
Published there as: “How Uber Protects Against Retry Storms”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.