Skip to main content
ANVISoftware Solutions
Lesson 22 of 23Advanced18 min

Performance

By the end of this lesson

Find and fix the bottlenecks that actually affect request latency.

Most guesses about performance are wrong. That is not a criticism of anyone's judgement; it is what happens when a request passes through routing, binding, validation, your code, an ORM, a database, a network and serialisation, and you are asked to name which part took the longest.

The consequence is a method rather than a list of tricks. Measure, find where the time actually goes, change that one thing, measure again. The profiler is the arbiter. Code that looks slow and code that is slow overlap far less than people expect.

Measure the right number as well. An average hides the problem: if ninety-five requests take 20ms and five take four seconds, the average looks acceptable and one caller in twenty is having a bad time. Look at the 95th and 99th percentiles, because those are the requests people complain about.

The order to work in. Skipping step one is how afternoons get lost:

  1. Establish a baseline

    Record the current latency of the endpoint under a realistic load, at the percentiles you care about. Without a number to compare against, you cannot tell whether a change helped.

  2. Reproduce it

    Find a request you can run repeatedly that is slow. An intermittent problem you cannot trigger is not yet ready to be optimised, and the work to make it reproducible is part of the fix.

  3. Measure where the time goes

    Use a trace or profiler to break the request down. Log the SQL your ORM produced. The aim is a sentence like "1.9 of the 2.1 seconds is in the database, across 340 queries".

  4. Form one hypothesis

    Name the cause and what you expect the fix to do. "Fixing the N+1 should take the query count from 340 to 1 and the endpoint under 200ms." A prediction you can be wrong about is the point.

  5. Change one thing

    Apply the single change. Changing three at once means you will not know which one helped, and one of them may have made things worse while another hid it.

  6. Measure again, and stop

    Compare against the baseline. If the endpoint is now fast enough, stop. Further optimisation has a cost in complexity and no remaining benefit.

When a request is slow, it is usually one of these. Not because they are the only possibilities, but because they are what actually turns up:

N+1 queries
One query to load a list, then one more per item to load something related. Fast with ten rows in development, ruinous with two thousand in production. The symptom is a query count that scales with the size of the result.
Synchronous I/O blocking threads
Calling an asynchronous method with .Result or .Wait(), or using a synchronous database or file API. The thread sits idle but occupied. Under load the thread pool runs dry and every request slows down, including ones that do nothing wrong.
No caching on a hot read path
Reference data loaded from the database on every single request. A department list that changes twice a year does not need querying forty times a second.
Oversized payloads
Returning entire entities with every column and every related collection when the client uses four fields. The cost lands in the database, the serialiser and the network at once, and a page size of "all of them" is the common cause.
Missing indexes
A filter or sort on an unindexed column makes the database read the whole table. This is invisible in a small development dataset and dominates everything at production volume. The database's own execution plan will tell you.
Two fixes that account for most real wins
C#
// ---- Before: 1 + N queries, every column, tracked entities ----
List<Employee> all = await db.Employees.ToListAsync(cancellationToken);

List<EmployeeListItem> slow = [];
foreach (Employee employee in all)
{
    // A separate query per employee to reach the department name.
    Department department = await db.Departments
        .FirstAsync(d => d.Id == employee.DepartmentId, cancellationToken);

    slow.Add(new EmployeeListItem(employee.Id, employee.FullName, department.Name));
}

// ---- After: one query, three columns, no tracking, and a page ----
List<EmployeeListItem> fast = await db.Employees
    .AsNoTracking()
    .Where(e => e.IsActive)
    .OrderBy(e => e.FullName)
    .Select(e => new EmployeeListItem(e.Id, e.FullName, e.Department.Name))
    .Skip((page - 1) * pageSize)
    .Take(pageSize)
    .ToListAsync(cancellationToken);

// ---- Before: blocks a thread pool thread for the whole call ----
EmployeeResponse blocked = payrollClient.GetSummaryAsync(id, cancellationToken).Result;

// ---- After: the thread is released while the call is in flight ----
EmployeeResponse released = await payrollClient.GetSummaryAsync(id, cancellationToken);
  • The loop is the N+1. With 2,000 active employees it issues 2,001 queries, and each one carries its own round trip. The fix is not a faster loop; it is not having a loop.
  • Select projects to exactly the fields the client needs, so the database returns three columns instead of every column of two tables. The projection also reaches the department name through the navigation property, which becomes a join in one query.
  • AsNoTracking tells the ORM not to keep change-tracking snapshots of what it loaded. For a read-only list that work is pure overhead, and it grows with the number of rows.
  • Skip and Take add paging. An endpoint with no upper bound on result size has a performance problem that is waiting for your data to grow rather than one you have yet.
  • The .Result call is the quieter of the two problems and often the more damaging. It holds a thread pool thread for the entire duration of a network call, so under load the pool starves and unrelated requests queue behind it. Await it instead, all the way up.

What to reach for, roughly in the order you would use them:

  • Your ORM's SQL logging, to see the query count and the actual statements
  • The database's own execution plan, to find a table scan or a missing index
  • dotnet-counters, for a live view of thread pool queue length, allocation rate and requests per second
  • dotnet-trace or a profiler, to break one slow request down by where the time went
  • OpenTelemetry traces, to follow a request across services and see which hop is slow
  • A load test, because latency under concurrency is a different measurement from latency of one request

Summary

  • Measure before changing anything, and compare percentiles rather than averages
  • The usual causes are N+1 queries, blocking I/O, uncached hot reads, oversized payloads and missing indexes
  • Projection with Select, AsNoTracking and paging remove most list-endpoint slowness by doing less work, not faster work
  • Blocking on async code with .Result holds a thread pool thread and degrades requests that are not even involved
  • Every fix has a cost, so the target is fast enough rather than as fast as possible

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Try it yourself

Make the N+1 visible, then remove it

Seed a development database with a few thousand employees across several departments. Write the version that loops and loads each department, turn on SQL logging, and count the queries.

Rewrite it as a single projected query. Record the query count and the timing before and after.

Show solution

The looping version issues one query per employee, so the count tracks the row count almost exactly. That relationship is the signature of an N+1, and once you have seen it in a log you will recognise it immediately in future.

The projected version issues one query. The improvement comes from removing round trips rather than from any change to how fast the database works, which is why it is so large — network latency per query dominates when there are thousands of them.

Seeding thousands of rows is the part people skip, and it is the part that makes the problem appear. With ten rows the looping version is perfectly acceptable, which is exactly how this reaches production.

Think about it

Where is the time going?

An endpoint takes 2.4 seconds at the 95th percentile and 180ms at the median. The code does one database query and one call to an internal payroll service.

What does that gap between median and 95th percentile suggest, and what would you measure first?

Show solution

A large gap between median and 95th percentile means most requests are fine and something occasional is very slow. A uniformly slow endpoint would show both numbers high together.

That pattern points away from your own code, which tends to cost about the same every time, and towards a dependency: the payroll call timing out and retrying, connection pool exhaustion under bursts, or a query whose plan changes with certain parameter values.

Measure the two external calls separately with their own timings before touching anything. A trace that shows how long the payroll call took on the slow requests will usually answer the question in one look.

It is worth noticing that this endpoint has almost no code in it, which is common. Most latency in an API is spent waiting on something else, which is why the profiler beats reading the method.

Saved in this browser only.