Retries are one of distributed systems engineering’s most useful half-truths. A second attempt can hide a lost packet, a brief connection race, or one unhealthy replica. Put that same reflex at every layer of a deep service graph, however, and one failing dependency can receive far more work precisely when it has the least capacity to handle it.

That is a retry storm: failure creates extra traffic, extra traffic prolongs the failure, and the prolonged failure triggers still more attempts. Backoff, jitter, attempt limits, and retry budgets slow the feedback loop. They do not answer the harder question: which layer should spend the retry?

Uber’s answer is error ownership. The service where a failure originates may permit its direct caller to retry. Services that merely pass the error upward mark it as already handled, so more distant ancestors do not repeat the same work. The result is a retry boundary that follows the cause of the failure instead of the depth of the call stack.

This article builds that mechanism from first principles, including its production trade-offs and the safeguards a smaller system can adopt without reproducing Uber’s infrastructure.

Why ordinary retry limits still multiply

Consider a linear request path A → B → C → D. Every service makes one downstream call for each inbound request. Under normal operation, a rate of N requests at A produces approximately N calls at every hop.

Now D begins returning errors and each caller allows one retry. C makes two attempts to D. Because C still fails, B makes another attempt to C, and each of those attempts can produce two calls to D. A does the same to B. The failing service can see 2³ × N = 8N calls even though only one user request entered the chain.

The general shape is the product of the attempt counts at every retrying layer. Three retrying layers with four total attempts each can turn one logical operation into 64 database attempts. Google SRE uses exactly this kind of example when warning against retries at multiple levels.

A retry budget improves the bound. If retries may consume at most 10% of ordinary traffic, a failing node three levels deep sees roughly 1.1³ × N, or 1.33N, rather than 8N. That is a large improvement. But it still spends a little extra capacity at every layer and it still treats every propagated error as a fresh opportunity.

The missing information is causality. An HTTP 500 or RPC internal error tells a caller that its immediate callee failed. It does not reveal whether that callee created the failure or merely returned an error received from somewhere deeper.

Separate the cause from the symptom

Suppose service B calls C while serving A:

• If C succeeds and B returns a server error, B owns the error. The fault arose in B’s own work.
• If C fails on a hard dependency path and B fails because of it, B is a symptom. C, or a service below C, owns the error.
• If C is a soft dependency and B can still complete successfully, the failure should not escape through B at all.

Uber calls the first relationship fail-close: when the callee fails, the caller consistently fails. A fail-open dependency can fail without forcing the caller to fail. This distinction matters because merely noticing that an outbound and inbound error happened during the same request is not always enough. Two unrelated faults can coincide.

Uber’s companion work on automated dependency analysis observes each service’s inbound result together with the results of its outbound calls. It aggregates those observations per caller endpoint and callee endpoint. If caller and callee failures strongly correlate, the edge is classified fail-close; weak correlation indicates fail-open; ambiguous evidence stays unknown. The published thresholds classify a conditional failure probability of at least 0.8 as fail-close and at most 0.2 as fail-open.

That learned dependency memory is more useful than a static map. Service graphs evolve, endpoints behave differently, and a service can use the same dependency as mandatory on one route and optional on another. The question is not simply “does B call C?” but “when C fails on this edge, does B fail too?”

Carry ownership with the error

Once the system can relate an inbound request to its outbound failures, the propagation rule is small enough to fit in middleware.

1. A service that returns an error without a failed hard dependency claims ownership.
2. A direct caller receiving a claimed error may retry, subject to its usual policy and budget.
3. If those attempts fail and the caller propagates the error, it unclaims it.
4. An upstream caller receiving an unclaimed error does not retry it.

In Uber’s design the signal travels in an error-claim header. The exact header is less important than the invariant: a claim is permission for the immediately adjacent caller to consider a retry, not a permission slip that survives every hop.

This is deliberately conservative. When a service receives a claimed error, it gets a chance to recover from a transient failure. If recovery fails, distant services are prevented from replaying the whole subtree. The policy preserves the retry with the best information and the smallest blast radius.

The scheme also handles a mixed deployment. If the ownership header is missing because an old service or broken context path cannot participate, the first capable caller may retry and then propagate an unclaimed error. Protection becomes less precise near the context gap, but the disturbance remains bounded instead of expanding to the root.

Why retrying next to the owner is the best bet

Retries are valuable only when a second attempt has a reasonable chance of observing a different condition. That is often true for a connection race, packet loss, a freshly rotated endpoint, or a single bad replica. It is much less likely during sustained overload, a broken shard, or a dependency-wide outage.

The direct caller is best placed to make that judgment. It knows the callee, the operation, the status, the deadline remaining, and whether its own policy has already retried. A service five levels above sees only that a large composite operation failed. Replaying it can repeat successful work, widen fan-out, and revisit the same bottleneck through many branches.

This is where ownership complements familiar controls rather than replacing them:

• Deadlines stop attempts after the result is no longer useful.
• Exponential backoff and jitter spread retry timing so clients do not synchronize.
• Per-request attempt limits cap one operation’s persistence.
• Retry budgets cap retry traffic relative to ordinary traffic.
• Circuit breakers and load shedding protect a dependency that has crossed its capacity boundary.
• Idempotency prevents a repeated request from duplicating side effects.
• Error ownership prevents multiple layers from spending retries on the same underlying failure.

gRPC, for example, supports bounded exponential backoff, retryable status selection, server pushback, and token-based throttling. Envoy can limit concurrent retries as a percentage of active and pending requests. Those mechanisms constrain volume at an edge. Ownership supplies the cross-edge context that a local budget cannot infer.

Availability improves only while failures are recoverable

A 10% retry budget sounds small, yet it can mask sparse independent failures well. If a callee succeeds 99% of the time and failed attempts are independent, retrying the failed 1% once leaves roughly 0.01 × 0.01 = 0.01% failures, or 99.99% perceived availability.

The independence assumption is doing most of the work. During an overload event, a retry is likely to hit the same saturated fleet. If the base error rate reaches 20% while only half of failed requests fit inside a 10%-of-total retry budget, the caller cannot transform that service into four nines. Many errors receive no retry, and the retried requests may fail for the same reason.

This yields a practical rule: retries repair random failures; they often reinforce correlated failures. As error rates rise, systems should become less optimistic, not more aggressive. Retry throttles, ownership propagation, circuit breakers, and explicit “do not retry” responses are all ways to encode that change in posture.

The awkward cases are part of the design

Context-aware retry control introduces failure modes of its own. They deserve explicit treatment.

No service currently retries

If ownership suppression simply blocks every ancestor, a call chain that never configured a retry next to the owner can lose availability. Uber passes an additional “retry criteria satisfied” signal through its retry middleware. The ownership logic can preserve a claim until one eligible retry opportunity has existed. It does not create a new retry policy; it ensures that suppression does not remove the only configured chance.

Request context is lost

Async boundaries, detached work, or instrumentation gaps can break correlation between an inbound call and an outbound failure. The service at the gap may incorrectly claim the propagated error as its own. That permits an extra retry boundary, but the next cooperating service unclaims it again. The storm radius grows locally instead of reopening across the full graph.

Two failures happen at once

A caller can have its own internal error while a soft dependency also fails. A naive “any outbound failed” rule would blame the dependency and suppress a legitimate retry of the caller. Uber reports that such coincidences are rare in its observations, but its dependency memory further narrows unclaiming to learned fail-close edges. Unknown and fail-open relationships should not erase ownership casually.

Retried operations have side effects

Error ownership answers where a retry may occur, not whether replaying the operation is safe. A timeout can hide a successful write whose response was lost. Use idempotency keys, conditional writes, deduplication, or operation-specific semantics before allowing application-level retries. AWS’s guidance on idempotent APIs is the essential companion to any retry design.

Production results show the value of a smaller radius

Uber describes a November 2025 incident in a core entity service more than five levels down a critical call chain. Without ownership-aware suppression, its retry budgets were projected to add 46% to 135% traffic to the degraded service. The deployed mechanism stopped some immediate callers from making as many as 200,000 additional requests and, aggregated at root services, prevented an estimated 9.5 million spurious requests.

The more revealing metric is not the dramatic request count but the retry storm radius: the maximum call-path depth over which a failure can trigger retries. Uber reports reducing the maximum from 25 to 3 and the average from 20 to 2 across user-facing APIs.

That is a better reliability objective than “all services have a retry budget.” It measures how far a local failure can recruit unrelated capacity into its feedback loop.

A practical adoption path

Most teams do not need probabilistic dependency discovery on day one. The ideas can be introduced in layers:

1. Inventory where retries occur on a few critical request paths. Count total attempts, not just the retry setting in each client.
2. Make retry metadata observable: attempt number, remaining deadline, reason, and whether the response asked clients to stop.
3. Choose one retrying layer per dependency path. Prefer the layer immediately above the component that can plausibly recover.
4. Add a global or per-client retry budget so retries cannot dominate healthy traffic.
5. Propagate an explicit retryable/non-retryable cause through shared RPC middleware. Treat missing metadata conservatively.
6. Learn or declare hard versus soft dependency edges per endpoint, then validate them from production telemetry.
7. Test sustained overload, partial fleet failure, context loss, and coincidental errors. A happy-path fault injection is not enough.
8. Track retry amplification and storm radius alongside availability and latency.

The most important operational habit is to count work at the failing dependency. A dashboard showing “retries succeeded” can look reassuring while hiding the load imposed by failed attempts, duplicated subtree calls, and work completed after its deadline.

The broader lesson: failures need provenance

An error code describes a result. In a deep service graph, safe recovery also needs provenance: where the failure began, which layers already acted on it, and whether another attempt can change the outcome.

Error ownership turns that provenance into a small control-plane signal carried with the data-plane failure. It lets the closest informed caller try once while telling every more distant ancestor that replaying the whole path is unlikely to help. Backoff controls when, budgets control how much, and ownership controls where.

That final dimension is what keeps a local outage local.

Sources

• Uber Engineering, How Uber Protects Against Retry Storms
• Uber Engineering, Large-Scale Automated Dependency Analysis Across Uber’s Service Mesh
• Google SRE, Addressing Cascading Failures
• Google SRE, Handling Overload
• gRPC, Retry guide
• Envoy, Circuit breaking and retry budgets
• AWS Builders’ Library, Making retries safe with idempotent APIs