Why do outages get worse after they're fixed?

A service goes down for 10 seconds. Pick how its clients retry, and watch what happens after it comes back.

Retry storm simulator

Backlog cleared
Never
Requests failed
5,158
outagecapacity0s30s60s90s120s

Every failed request comes back 2 seconds later, together. The outage ended, but retries keep the service down.

Each tap runs a fresh simulation: 60 requests/sec into a service that can handle 100, then a 10 second outage. Rejecting a request still costs the server 30% of the work of serving one.

Real outages

Status pages don't publish request rates, so the simulator can't reproduce these incidents. They do publish how long each outage lasted. Replay one to watch an outage that long play out, one real minute per simulated second.

From the cloud providers' own incident reviews

Microsoft Azure, Jul 23, 2026about 5 hWest US datacenter cut off by a repair (West US)

A repair to one optical device was widened to every device leaving a datacenter, because the safety check judged each device on its own instead of their combined effect. Routes were withdrawn, cutting the datacenter off from the network for about five hours. Automated recovery retried several times, but every retry depended on the same broken connectivity.

A different lesson: retries only help when the thing being retried can recover without them.

Sources: Azure status history (ZJV6-SGG), Surfing Complexity

Microsoft Azure, Feb 3, 20265 h 55 minManaged Identity overwhelmed after a fix (East US, West US)

A storage policy change broke VM operations on February 2. The mitigation for that set off a second incident: a large spike in traffic overwhelmed Managed Identity, and several scale-up attempts could not keep up with the backlog and retry volume. Engineers removed traffic so the service could be repaired without load. It ran from 00:10 to 06:05 UTC.

The textbook retry storm: the fix arrived, and the queued and retried traffic kept the service down.

Sources: Azure status history (M5B-9RZ), Network World

AWS, Oct 20, 20252 h 52 minDynamoDB DNS fixed, EC2 stuck in congestive collapse (us-east-1)

A race condition in DynamoDB's DNS automation left the us-east-1 endpoint with an empty DNS record. DynamoDB was back in under three hours, but EC2's lease manager had fallen behind: its work kept timing out and being requeued, and AWS's summary says it "entered a state of congestive collapse." Engineers throttled incoming work and restarted hosts. Full EC2 recovery came about 11 hours after DynamoDB was fixed.

The outage took three hours. Draining the backlog took eleven more.

Sources: AWS post-event summary

Google Cloud, Jun 12, 20253 hService Control restart caused a herd effect (global, worst in us-central1)

A policy change with blank fields replicated globally and crashed Service Control, the layer that checks Google Cloud API requests. As it restarted in us-central1, it created a herd effect on the infrastructure it depends on, because, in Google's words, "Service Control did not have the appropriate randomized exponential backoff implemented." us-central1 took up to about 2 hours 40 minutes to fully recover.

The missing piece is the simulator's middle button: randomized exponential backoff.

Sources: Google Cloud incident report

Recently on public status pages

Live data refreshes daily from Google Cloud's and other public status pages. The first refresh hasn't run yet.

Found in the wild

I wrote a few scanner rules for retry anti-patterns and ran them on public code from companies at AfroTech. Then I reviewed every hit by hand.

Netflix/spectatorPatternSmart retry policy, one fast path

The default policy is idempotency-aware: writes only retry on connection failures, and 429 or 503 responses back off exponentially. But other 5xx responses on GET, and connection failures, retry immediately with no delay.

if (shouldRetry && (s == 429 || s == 503)) {
  // Request is getting throttled, exponentially back off
  // - 429 client sending too many requests
  // - 503 server unavailable
  try {
    long delay = initialRetryDelay << (attempt - 1);
    LOGGER.debug("request throttled, delaying for {}ms: {} {}", delay, method, uri);
    Thread.sleep(delay);
  } catch (InterruptedException e) {
    Thread.currentThread().interrupt();
    throw new IOException("request failed " + method + " " + uri, e);
  }
} else if (!shouldRetry) {
  return response;
}

It also never reads the Retry-After header, which the simulator above shows recovers fastest. Small change, big effect when a dependency is struggling.

View the exact lines on GitHub
Netflix/EVCacheDone wellBackpressure that fails fast

When a cache node's write queue is full, the client waits briefly a few times, then drops the write and records a metric instead of blocking forever.

while (true) {
    final int size = evcNode.getWriteQueueSize();
    final boolean canAddToOpQueue = size < maxWriteQueueSize;
    // (debug logging omitted)
    if (canAddToOpQueue) break;
    try {
        Thread.sleep(writeBlock.get());
    } catch (InterruptedException e) {
        throw new EVCacheException("Thread was Interrupted", e);
    }

    if(i++ > 3) {
        incrementFailure(EVCacheMetricsFactory.WRITE_QUEUE_FULL, call);

Bounded waiting plus load shedding is the right shape. A cache write is cheaper to lose than a stuck caller thread.

View the exact lines on GitHub
Netflix/conductorPatternRetry any exception, fixed 500ms

The task client's retry helper retries every exception, waits a flat 500ms, and swallows the interrupt, so it keeps retrying during shutdown.

while (index < count) {
    try {
        return operation.apply(input);
    } catch (Exception e) {
        index++;
        try {
            Thread.sleep(500L);
        } catch (InterruptedException ie) {
            LOGGER.error("Retry interrupted", ie);
        }
    }
}

Netflix stopped maintaining this repo in Dec 2023. It's still a clean example of the fixed-delay pattern in the simulator above.

View the exact lines on GitHub
conductor-oss/conductorPatternSmarter retry, one subtle gap

The community fork only retries connection errors, which is the right call. But the delay is 100ms × (count + 1), and count is the max retries. Every caller passes 3, so every attempt waits a flat 400ms.

if (t instanceof ConnectException
        || (t.getCause() != null && t.getCause() instanceof SocketException)) {
    index++;
    Uninterruptibles.sleepUninterruptibly(
            100L * (count + 1), TimeUnit.MILLISECONDS);
} else {
    break;
}

Looks like backoff, behaves like a fixed delay. Easy to miss in review.

View the exact lines on GitHub
aws/aws-sdk-java-v2Done wellExponential backoff, done right

The EC2 metadata client doubles its wait each attempt, gives up cleanly on the last try, and restores the interrupt flag.

int pause = (int) (Math.pow(2, DEFAULT_QUERY_ATTEMPTS - tries) * MINIMUM_RETRY_WAIT_TIME_MILLISECONDS);
try {
    Thread.sleep(pause < MINIMUM_RETRY_WAIT_TIME_MILLISECONDS ? MINIMUM_RETRY_WAIT_TIME_MILLISECONDS
                                                              : pause);
} catch (InterruptedException e1) {
    Thread.currentThread().interrupt();
}

The one addition I'd suggest is jitter, so many instances booting at once don't retry in lockstep.

View the exact lines on GitHub
aws/aws-sdk-java-v2NoiseFlagged by the scanner, not a retry

My rule flagged sleep-in-a-loop-with-a-catch. This is a background reaper that closes idle connections on a timer. Correct code.

while (!stopping) {
    try {
        Thread.sleep(sleepPeriod);
        // close idle connections...

Scanners are only useful if someone tunes out noise like this. The fix: exclude loops whose condition is a shutdown flag.

View the exact lines on GitHub