Lesson 20 of 23Advanced14 min

Rate Limiting

By the end of this lesson

Protect endpoints from excessive use with appropriate limits.

A rate limit is a cap on how much one caller may ask of your service in a period of time. Requests above the cap are refused quickly and cheaply, before they consume a database connection or a worker thread.

The reason to have one is rarely malice. It is usually a client with a retry loop that has no delay in it, an import script that was pointed at the wrong environment, or a scheduled job that started running every minute instead of every hour. One caller behaving badly can exhaust a connection pool and make the service unavailable for everyone else.

A limit turns that outage into a 429 for the one caller causing it. That is the whole value: it contains the damage to its source.

ASP.NET Core ships four algorithms. They behave differently at the edges, and the differences are what make one right for a given endpoint:

Fixed window: A counter that resets on a clock boundary: 100 requests per minute, starting again at each new minute. Simple and cheap. Its weakness is the boundary — a caller can use the full allowance at the end of one window and again at the start of the next, briefly doubling the rate. Fine for general protection of ordinary endpoints.
Sliding window: The same idea with the window divided into segments that roll forward, so the boundary burst is smoothed out. Costs a little more to track. Use it where a sudden doubling would actually hurt, such as a login endpoint.
Token bucket: A bucket holds tokens, each request takes one, and tokens are replenished at a steady rate. A caller who has been idle can spend a saved-up burst, then settles into the refill rate. This is the closest fit for APIs where occasional bursts are legitimate and the sustained average is what you care about.
Concurrency limiter: Caps how many requests are in flight at once rather than how many arrive per period. Nothing to do with time. This is the right tool for an expensive operation — a report that runs for thirty seconds — where the real constraint is how many can run together.

Per-caller limits, plus a concurrency cap on an expensive endpoint

builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;

    // Partition by authenticated user, falling back to remote IP for
    // anonymous callers. Without a partition key, one global counter is
    // shared by everyone and one busy client throttles the whole world.
    options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(context =>
    {
        string partitionKey =
            context.User.FindFirst("sub")?.Value
            ?? context.Connection.RemoteIpAddress?.ToString()
            ?? "unknown";

        return RateLimitPartition.GetTokenBucketLimiter(partitionKey, _ =>
            new TokenBucketRateLimiterOptions
            {
                TokenLimit = 120,              // burst allowance
                TokensPerPeriod = 60,          // sustained rate
                ReplenishmentPeriod = TimeSpan.FromMinutes(1),
                QueueLimit = 0,                // refuse rather than queue
                AutoReplenishment = true,
            });
    });

    // One named policy for the expensive report endpoint.
    options.AddConcurrencyLimiter("headcount-report", limiter =>
    {
        limiter.PermitLimit = 2;
        limiter.QueueLimit = 4;
        limiter.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
    });

    options.OnRejected = async (context, cancellationToken) =>
    {
        if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out TimeSpan retryAfter))
        {
            context.HttpContext.Response.Headers.RetryAfter =
                ((int)retryAfter.TotalSeconds).ToString(NumberFormatInfo.InvariantInfo);
        }

        context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;

        await context.HttpContext.Response.WriteAsJsonAsync(new ProblemDetails
        {
            Status = StatusCodes.Status429TooManyRequests,
            Title = "Too many requests.",
            Detail = "Slow down and retry after the interval in the Retry-After header.",
        }, cancellationToken);
    };
});

WebApplication app = builder.Build();

app.UseRateLimiter();
app.MapControllers();

// ---

[HttpGet("headcount-report")]
[EnableRateLimiting("headcount-report")]
public async Task<IActionResult> HeadcountReport(CancellationToken cancellationToken)
{
    return Ok(await reports.BuildHeadcountAsync(cancellationToken));
}

PartitionedRateLimiter.Create takes a function that returns a partition key per request. The key is what makes the limit per-caller rather than global, and getting it wrong is the difference between protecting the service and throttling it.
Preferring the authenticated subject over the IP address matters because many callers can share one address. An office or a mobile network puts hundreds of users behind a single IP.
The token bucket here allows a burst of 120 and a sustained 60 per minute. QueueLimit of 0 means excess requests are refused immediately instead of waiting, which is usually what an API wants — a queued request still holds a connection.
The concurrency limiter on the report is a different kind of limit: at most two running, with four allowed to wait. Time does not appear in it, because the constraint is how many heavy queries the database can take at once.
OnRejected is where you shape the refusal. Retry-After is the part a well-behaved client reads, and it turns a blind retry loop into a co-operative one. Only the time-based algorithms can estimate it, which is why the metadata is checked rather than assumed.
UseRateLimiter is middleware, so it applies to every endpoint including ones that are not controllers. Register it early, so a rejected request costs as little as possible.

What a refused request looks like

HTTP

GET /api/employees?page=4 HTTP/1.1
Host: api.internal.example
Authorization: Bearer <token>

HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 37

{
  "title": "Too many requests.",
  "status": 429,
  "detail": "Slow down and retry after the interval in the Retry-After header."
}

429 is the status code for this and nothing else. Returning 503 instead tells the caller your service is broken, and they will escalate rather than back off.
Retry-After is in seconds here. A client that reads it waits 37 seconds; a client that ignores it keeps hammering and keeps getting 429, which is at least cheap for you.
The body follows the same problem-details shape as every other error in the API. Consistency means a client needs one error path, not one per status code.
Nothing in the response reveals the limit, the algorithm or other callers' usage. Publish your limits in documentation if you want clients to respect them; the response itself does not need to describe your configuration.

Common mistake

Limits that do not limit what you think

One global limiter with no partition key. Every caller shares a single counter, so a busy integration exhausts the allowance and every other client is refused. You have built the outage you were trying to prevent.

Partitioning by IP address alone when your API sits behind a load balancer or reverse proxy. Without forwarded headers configured, every request appears to come from the proxy, so all your callers land in one partition. The production configuration lesson covers the fix.

A generous QueueLimit. Queued requests are still connections held open and memory occupied. Queueing is right for a short, expensive operation and wrong as a way to soften a rate limit.

Leaving the expensive endpoints unlimited because they are rarely called. Rarely called and cheap to call are different things; a report endpoint with no limit is the one that takes the database down.

Limiting only at the application. If a flood never reaches your process you spend nothing on it, so a limit at the gateway or CDN is worth having as well. Keep the application limit too — it is the one that still applies when traffic arrives another way.

Trade-off

Every limit refuses somebody legitimate

A limit tight enough to stop a runaway client will sometimes refuse a real one. A customer running a legitimate bulk import hits the same wall as a broken retry loop, and from your service's point of view they look identical.

So pick limits from evidence rather than instinct. Look at what your busiest well-behaved caller actually does, set the limit some way above it, and log rejections with the partition key so you can see who you are refusing.

Give the well-behaved caller a route through: a documented limit, an accurate Retry-After, and a bulk endpoint for work that genuinely needs volume. A 429 with no path forward turns into a support conversation.

Start looser than feels right and tighten with data. A limit that is too tight produces incidents that look like your service failing, which is the opposite of the outcome you wanted.

Summary

A rate limit contains the damage from one runaway caller instead of letting it become an outage for everyone
Fixed window is cheap but allows a burst across the boundary; sliding window smooths it; token bucket allows a burst then a steady rate
A concurrency limiter caps simultaneous requests and is the right tool for expensive operations
The partition key is what makes a limit per-caller, and the authenticated subject is a better key than the IP address
Refuse with 429 and an accurate Retry-After. Every limit refuses somebody legitimate, so set it from observed traffic and log who you reject

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Think about it

Choose the algorithm

Pick a limiter for each: the sign-in endpoint; a public read endpoint used by a mobile app that syncs in bursts when it reconnects; an export that takes twenty seconds and holds a database connection throughout.

Say what would go wrong with the obvious alternative in each case.

Show solution

Sign-in: sliding window. The fixed-window boundary allows a short burst of double the intended rate, and sign-in is exactly where you do not want that. The cost is slightly more bookkeeping per request, which is trivial at sign-in volumes.

Mobile sync: token bucket. Bursts after reconnection are legitimate, and a bucket lets an idle client spend a saved allowance while still capping the sustained rate. A fixed window sized for the burst would allow far too much sustained traffic.

Export: concurrency limiter. The constraint is not requests per minute, it is how many twenty-second queries the database tolerates at once. A per-minute limit of, say, ten would happily allow ten simultaneous exports, which is the situation you were trying to avoid.

The general lesson: match the limiter to the resource under pressure. Time-based limiters protect against volume; the concurrency limiter protects against simultaneity, and they are not substitutes.

Try it yourself

Confirm the partition actually partitions

Configure a small per-caller limit. Send requests as one user until you get a 429, then immediately send a request as a different user.

The second user should succeed. If they do not, find out why before doing anything else.

Show solution

The second user succeeds when the partition key varies per caller. If they are refused, the key is constant — a common cause is reading a claim that is absent, so every request falls through to the same fallback value.

This is worth testing explicitly because a broken partition key is invisible under light load. With one test user, a global limiter and a per-user limiter behave identically, and the difference only appears when real traffic arrives.

Log the partition key alongside rejections while developing. It makes this whole class of problem obvious in seconds, and it stays useful later when you want to know which caller you are refusing.

Saved in this browser only.