Resilience

Resilience is deciding, in advance and in writing, what your application does when something it depends on is slow or unavailable. Not whether that happens — it happens — but what the user sees when it does, and whether your application is still able to do the things that do not need the failing dependency.

Slow is the case worth thinking about first, because it does more damage than down. A dependency that refuses connections fails your calls in milliseconds and everyone finds out immediately. A dependency answering in forty seconds holds a thread, a connection and a request slot for each caller, and within a minute your pools are full. Requests to perfectly healthy dependencies then start queueing behind the slow one, and the application looks entirely broken while every dashboard says only one service has a problem.

builder.Services
    .AddHttpClient<IInvoicingApi, InvoicingApi>(client =>
    {
        client.BaseAddress = new Uri("https://invoicing.internal/");
    })
    .AddStandardResilienceHandler(options =>
    {
        // No single attempt may take longer than two seconds.
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);

        // Three retries: roughly 200ms, 400ms, 800ms, each nudged by a random amount.
        options.Retry.MaxRetryAttempts = 3;
        options.Retry.Delay = TimeSpan.FromMilliseconds(200);
        options.Retry.BackoffType = DelayBackoffType.Exponential;
        options.Retry.UseJitter = true;

        // Stop calling when a tenth of a 30-second window has failed, over at
        // least 20 calls. Fail instantly for 5 seconds, then test one request.
        options.CircuitBreaker.FailureRatio = 0.1;
        options.CircuitBreaker.MinimumThroughput = 20;
        options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
        options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(5);

        // Everything above, including all retries, inside eight seconds.
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(8);
    });

• Start with the timeout, because it is the cheapest change with the largest effect. HttpClient's default is 100 seconds, which for a web application is indistinguishable from forever: a handful of requests per second against a hung dependency exhausts the thread and connection pools, and then nothing works. A call with no timeout is a call that can hang until you restart the process.
• Two timeouts, and both are needed. The per-attempt timeout abandons one hung call. The total timeout stops three retries plus their delays from adding up to more time than any caller is prepared to wait.
• Exponential delay gives the dependency room. Retrying immediately means your recovery attempt arrives while the dependency is least able to serve it, which is how a two-second blip becomes a two-minute outage.
• Jitter matters more than it looks. Without it, a thousand callers that failed at the same instant all retry at the same instant, and the dependency is struck by synchronised waves. A small random offset spreads the same number of calls across the window.
• The circuit breaker protects both sides. It stops your retries piling onto something already failing, and it stops each of your own requests spending eight seconds discovering the same fact. While it is open, calls fail in microseconds, which is what makes a degraded response fast rather than slow.
• Every number here is a decision, and the caller's deadline is the thing to hold in view. Eight seconds of retrying is reasonable for a background job and poor for a screen someone is watching, where failing at two seconds with a retry button is usually kinder.

Two rules get skipped, and each one turns a resilience feature into a defect.

The first: retry only what is safe to repeat. A request that charges an invoice is not safe. Your first attempt may have reached the provider, been processed, and had its reply lost on the way back — from your side that is indistinguishable from never arriving. The retry takes the money a second time, no exception is thrown anywhere, and finance finds it. Retries look like a transport concern and they are a correctness concern. There are two honest ways out: give the operation an idempotency key the remote side deduplicates on, which is why the payment port in the adapter lesson carried a reference, or do not retry and record the uncertain outcome for a reconciliation job to settle.

The second: retries without backoff amplify an outage. A dependency at its limit starts failing. Every caller retries three times, so it now receives four times the traffic it could not handle in the first place, and it cannot recover while that lasts. This is a retry storm, and it routinely makes an outage last far longer than the fault that started it. Backoff, jitter and a circuit breaker are what stop your recovery attempts from being the reason recovery is impossible.

public async Task<InvoiceListView> BuildAsync(int customerId, CancellationToken token)
{
    // Essential: without these there is no page.
    IReadOnlyList<InvoiceListRow> invoices =
        await invoiceQuery.GetOutstandingAsync(customerId, token);

    // Useful, not essential: a credit badge beside the customer's name.
    CreditDecision? credit = null;
    try
    {
        credit = await creditAssessment.AssessAsync(new CreditEnquiry(customerId, 0m), token);
    }
    catch (Exception ex) when (ex is HttpRequestException
                                  or TimeoutRejectedException
                                  or BrokenCircuitException)
    {
        log.LogWarning(ex, "Credit assessment unavailable for customer {CustomerId}.", customerId);
    }

    return new InvoiceListView(
        Invoices: invoices,
        Credit: credit,
        CreditUnavailable: credit is null);
}

• The design decision happens before any of this code: which parts of this screen are essential. Invoices are the page. The credit badge is a nicety. Once that is written down, a credit outage costs a badge rather than a screen.
• The catch lists three transport failures rather than catching everything. Catching Exception would also swallow a defect in your own mapping code, and a permanently degraded page caused by your own bug is worse than an error, because nobody is alerted.
• BrokenCircuitException is what the breaker raises while it is open, and it arrives almost instantly. That is what keeps a degraded page fast: without the breaker, every request waits for the timeout before degrading.
• CreditUnavailable is in the view model deliberately. A missing figure rendered as a blank reads as "no credit issues", which is a worse outcome than an error — the user acts on an absence as though it were information. Say the value is unavailable.
• The timeout for this call must be shorter than the page's own budget. If the screen has to render in two seconds and the credit call is allowed five, the degradation never happens; the page fails slowly instead.
• One more thing to add before this is finished: a metric counting how often the degraded path runs. Otherwise the credit service can be failing a third of the time for a fortnight and every dashboard stays green.