Lesson 9 of 17Intermediate16 min

Monitoring and Logs

By the end of this lesson

Collect logs and metrics you can actually act on.

Once the employees API runs on someone else's computer, you cannot attach a debugger to it. What you have instead is whatever the application wrote down while it was working. Monitoring is the practice of deciding, in advance, what it should write down.

Three kinds of signal cover almost everything, and each answers a different question. Metrics answer whether the system is healthy right now. Traces answer where the time went in one request. Logs answer why one specific thing failed. Teams that collect only one of the three end up guessing, usually about the question the missing signal would have answered.

The vocabulary, which is broadly consistent across providers and tools:

Log: A record of one event, with a timestamp and a set of fields. High detail, high volume, and the only signal that can explain a single failure.
Metric: A number sampled over time — requests per second, error count, queue depth, upload duration. Cheap to store, cheap to chart, and it cannot tell you which request failed.
Trace and span: A trace follows one request across services. Each unit of work inside it is a span, with its own duration. This is how you discover that a 900ms endpoint spent 750ms on one database query.
Correlation id: An identifier created at the edge of the system and passed to everything the request touches. Without one, a request that crossed the frontend, the API and a background job is three unrelated piles of lines.
Structured logging: Writing the values of a log line as named fields rather than pasting them into a sentence. It is what makes a log searchable by field instead of by substring.
Sampling: Keeping a fraction of high-volume data, usually traces. Necessary at volume, and it means some of what happened was never recorded.

Logs and metrics get confused because both are numbers on a screen. They answer different questions and they fail in different ways.

	Logs	Metrics
Answers	Why did this particular upload fail at 14:32?	Are uploads failing more than usual today?
Shape	One record per event, with arbitrary fields	A number per time interval, with a few labels
Cost driver	Volume ingested and how long it is kept	Number of distinct label combinations
Good for	Investigating one case, with the detail intact	Dashboards, alerts and trends over months
Poor for	Trends. Counting log lines to chart a rate is slow and expensive	Anything about an individual request. The detail was thrown away on purpose
Sensible retention	Days to weeks for detail, longer for errors	Months to years, because rolled-up numbers are small
Alert on it?	Sparingly, and only on specific known-bad events	Yes. This is what metrics are for

Employees API — logging that can be queried later

// Structured: the placeholders become named fields, not text.
logger.LogInformation(
    "Photo upload completed for employee {EmployeeId} in {ElapsedMs}ms, {Bytes} bytes",
    employeeId, sw.ElapsedMilliseconds, bytes);

// Not this. The values are welded into a unique string.
// logger.LogInformation($"Photo upload completed for employee {employeeId}");

// A scope attaches the same fields to every line written inside it.
using (logger.BeginScope(new Dictionary<string, object>
{
    ["CorrelationId"] = correlationId,
    ["EmployeeId"] = employeeId,
}))
{
    try
    {
        await photos.StoreAsync(key, stream, ct);
    }
    catch (StorageException ex)
    {
        // Pass the exception, not ex.Message — the stack trace is the useful part.
        logger.LogError(ex, "Photo upload failed for key {Key}", key);
        throw;
    }
}

The named placeholders are the whole point. A collector stores EmployeeId as a field, so you can ask for every upload for one employee, or the slowest one per cent by ElapsedMs. With an interpolated string, every line is unique text and you are reduced to substring searches.
The commented-out line is the single most common logging mistake in .NET, and it looks correct. Interpolation happens before the logger sees it, so the fields are gone and the message template can no longer be used to group similar events.
A scope means you write the correlation id once and every line inside inherits it. That is what turns a pile of lines into the story of one request.
Passing the exception object rather than its message keeps the stack trace and the inner exceptions. A log line reading Storage failed with no frames is a reminder that something broke and no help in fixing it.
What is deliberately absent: the photo bytes, the user's token, and anything personal beyond an id. Logs are copied into a search system that more people can read than can reach production, and they outlive the incident by weeks.

The same line, as the collector stores it

JSON

{
  "timestamp": "2025-03-04T14:32:07.812Z",
  "level": "Information",
  "message": "Photo upload completed for employee 4821 in 412ms, 1841204 bytes",
  "messageTemplate": "Photo upload completed for employee {EmployeeId} in {ElapsedMs}ms, {Bytes} bytes",
  "fields": {
    "EmployeeId": "4821",
    "ElapsedMs": 412,
    "Bytes": 1841204,
    "CorrelationId": "b7f1e2c4-9d3a-4c11-9f0e-2a8d5c6b1e77",
    "Environment": "production",
    "Version": "1.4.0+9f4c2ab"
  },
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736"
}

The rendered message is there for humans and the fields are there for queries. Both matter, and you get both from one call because the template was preserved.
The template itself is a field. Grouping by template is how a tool tells you that one kind of event happened 40,000 times today, without pattern-matching the rendered text.
Environment and Version are attached by configuration rather than by each call site. Version is the build metadata from the artifacts lesson, and it is what lets you say a class of error started with one specific release.
The trace id links this line to the distributed trace, so you can move from why did this fail to where did the time go without searching by timestamp.
A metric can be derived from this: count of these events per minute, and a duration histogram from ElapsedMs. Deriving a metric from logs works at small volume and gets expensive. Above a few hundred requests a second, emit the metric directly and keep logs for the detail.

Retention and sampling are where monitoring costs get decided. Set them deliberately, per signal.

Set retention per signal rather than one number for everything. Detailed request logs for two to four weeks covers most investigations; error logs and audit records usually need longer; metrics can be rolled up and kept for a year for very little
Ingestion volume and retention length are the two things you are billed for. A Debug line on a hot path can cost more per month than the compute serving the requests, which surprises people the first time
Sample traces rather than logs. Head-based sampling decides at the start of the request, which is cheap and sometimes discards the failure you wanted. Tail-based sampling buffers the trace and keeps it if it errored or was slow, which is more useful and needs a collector that can hold it
Keep every error and sample the successes. A one per cent sample of healthy traffic still shows you the shape of normal, and losing one per cent of failures means losing whole classes of rare bug
Keep identifiers out of metric labels. An employee id as a metric dimension creates one time series per employee, which multiplies storage and cost without limit. Ids belong in logs and traces, where they cost one field
Review what you are collecting every few months. Log lines added during an incident two years ago are still being written and billed, and nobody has read them since

Common mistake

Common mistakes when starting out

Interpolating values into log messages. It reads better in the editor and it destroys every query you would want to run later.

Logging ex.Message instead of the exception. You lose the stack trace and the inner exception, which are the two things that would have told you where the failure came from.

Alerting on causes rather than symptoms. High CPU is sometimes fine and sometimes fatal, so an alert on it fires when nothing is wrong. Alert on the error rate, on latency, and on failed uploads — things a user would notice — and use CPU to investigate once you are already looking.

Having no correlation id. Diagnosing a cross-service problem then means matching timestamps by eye, which is slow and quietly unreliable when several requests overlap.

Turning logging down to cut the bill, with no other change. The next incident arrives with no data. Reduce volume by removing lines nobody reads and by sampling, not by disabling the signal you will need.

Writing personal data or tokens into logs. A log store is a copy of that data with its own access list and its own retention, and it is frequently forgotten when someone asks where personal data lives.

Building a dashboard nobody opens. A dashboard is for investigating; an alert is for being told. If a chart is the only thing standing between a failure and a customer noticing, the failure will be noticed by the customer.

Summary

Metrics tell you whether something is wrong, traces tell you where the time went, logs tell you why one case failed
Structured logging keeps values as named fields, which is what makes a log queryable at production volume
A correlation id turns scattered lines from several services into the story of one request
Retention and ingestion volume drive the bill, so set them per signal and keep identifiers out of metric labels
Alert on symptoms a user would notice, not on causes like CPU that are sometimes perfectly normal

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Think about it

Which signal answers it?

For each question, say which signal you would reach for first: are photo uploads slower than last week; why did employee 4821's upload fail at 14:32; which part of the employee search endpoint takes the longest; did the deployment at 09:00 change the error rate.

Show solution

Slower than last week is a metric. It is a trend over a period longer than any log retention you want to pay for, and duration histograms are built for exactly this.

One employee's failure at a known time is a log query, filtered by EmployeeId and the time window. No other signal keeps the detail of one event.

Where the time goes inside one endpoint is a trace. Spans break the request into its parts, which a single duration metric cannot do.

The deployment question is a metric, with the version as a label or an annotation on the chart. This is the practical argument for attaching the build version to everything you emit — without it, comparing before and after means comparing timestamps and hoping nothing else changed.

Try it yourself

Make one endpoint answerable

Take one endpoint you have written. Add a correlation id at the entry point, convert its log lines to structured fields, and emit one metric that would tell you it was failing.

Then write down the query you would run to find every failure for one employee yesterday, and check that your fields make it possible.

Show solution

Writing the query first is the useful discipline. Most instrumentation is added by deciding what looks worth logging, which produces lines that cannot answer the questions people actually ask during an incident.

A workable arrangement is one metric for the outcome — a counter of successes and failures, labelled by result and nothing high-cardinality — plus structured logs carrying EmployeeId and CorrelationId for the detail. The metric tells you something is wrong; the logs tell you what.

If your query needs a substring match on a message, the fields are not right yet. That query works on a test machine with a thousand lines and times out on production with a hundred million.

There is a defensible alternative for a low-traffic internal service: derive the rate from logs and skip the metric. Know why you chose it, because that decision stops working as traffic grows, and it usually stops working quietly by getting slower.

Saved in this browser only.