Lesson 16 of 17Advanced17 min

Monitoring and Alerting

By the end of this lesson

Detect problems before users report them.

The earlier monitoring lesson was about collecting signals: structured logs, metrics, traces, a correlation id. Collecting is not detecting. A system can emit perfect telemetry into a search index that nobody opens for three days, which is the same as noticing nothing.

An alert is a decision made in advance that a particular pattern in those signals is worth interrupting a person for. That framing decides everything else in this lesson. Interrupting a person is expensive, it is finite, and it stops working if you do it too often — so the question is never what could we alert on, it is what is worth waking someone up for.

The answer is narrower than most teams expect, and the work is mostly subtraction.

The vocabulary, and the product names where they differ:

Alert: A rule that evaluates a signal and notifies a person when it crosses a line. Azure Monitor calls it an alert rule with an action group; AWS calls it a CloudWatch alarm with an SNS topic. One thing they share: the rule is only half of it, and the action is the other half.
Symptom and cause: A symptom is something the user experiences: requests failing, pages taking eight seconds, the field staff app unable to upload. A cause is why: a full disk, an exhausted connection pool, a slow query. Alert on symptoms, investigate causes.
Dashboard: A set of charts you open deliberately, usually because an alert already told you something is wrong. It answers where and why. It is not a detection mechanism, because detection requires someone looking.
Service level indicator and objective: An indicator is a measurement of something a user cares about, phrased as a proportion of good events — for example, the share of employee search requests that succeed within 500ms. An objective is the target you commit to, such as 99.5 per cent of them over 30 days. Plain arithmetic, deliberately.
Error budget: The failure the objective allows. A 99.5 per cent objective permits half a per cent of requests to be bad, and that remainder is a quantity you can spend. It turns arguments about whether something is bad enough to act on into a calculation.
Runbook: The written first response for one alert: what it means, what to check, what to do, and who to involve if that does not work. An alert without one asks the person it woke to invent a procedure at their worst moment.
Alert fatigue: What happens when alerts fire often enough to be background noise. People stop reading them, then stop hearing them, then mute the channel. The alerts still fire and the system is no longer monitored.
Synthetic check: A request your monitoring makes on a schedule from outside your network, following a path a user follows. Azure Monitor availability tests and CloudWatch Synthetics canaries both do this.

These get built for each other's purpose constantly, which produces dashboards nobody watches and alerts nobody can act on.

	An alert	A dashboard
Purpose	Tell a person something is wrong now	Help a person who is already looking work out where and why
When it is read	When it fires, whether or not that is 3am	During an investigation, a review, or a change
What belongs on it	A small number of symptoms with a defined response	Causes, breakdowns, comparisons, anything that helps narrow a problem down
Processor usage	No. It moves for harmless reasons and it is normal at 90 per cent on a busy service	Yes. Once you know something is wrong, it helps explain it
How many you want	Few enough that every one is read. For one service, single digits	As many as are useful, because an unread chart costs nothing but the effort to maintain it
Cost of getting it wrong	High. Too many and they are all ignored, which removes your detection entirely	Low. A poor dashboard wastes a little time during an incident
Test of whether it should exist	Can you name what the person receiving it will do?	Would someone open it during an incident or a review?

The alerts most services actually need, and the ones to leave off. Four or five rules, each tied to something a user would notice:

Is it serving at all. A synthetic request from outside your network, on the path a real user takes, every minute or two. This is the only check that catches an expired certificate, a broken DNS record or a misconfigured ingress, because all three are invisible from inside the application
Error rate, as a proportion of requests rather than a count. A count fires at peak traffic and stays quiet during a quiet-hours outage. A proportion behaves the same at any volume
Latency at a high percentile, not the average. If one request in twenty takes nine seconds, the average barely moves and one user in twenty is having a bad time. The 95th or 99th percentile is what they feel
One business-level check per critical journey. For the field staff app: zero successful photo uploads in the last hour during working hours. This catches the failures that produce no errors, such as an upload path that returns success and quietly writes nothing
Saturation that becomes an outage with a known lead time. Database disk at 90 per cent and climbing, a connection pool at its limit, a queue growing faster than it drains. These are causes rather than symptoms, and they earn a place because the consequence is certain and the lead time makes them actionable
Leave off: processor and memory percentages, single failed requests, one instance restarting, anything the platform recovers from by itself, and anything you could not name a response for. Each of these belongs on a dashboard, in a metric you query during an investigation, or nowhere
Alert on the deployment too. Most new problems are caused by the most recent change, so a rule that compares the error rate before and after a release catches faults faster than a threshold that has to wait for the overall rate to move

One alert rule, with the fields that decide whether it is useful

JSON

{
  "name": "employees-api-error-rate",
  "description": "Server errors above 2 percent of requests for 10 minutes",
  "signal": {
    "numerator": "http_responses_5xx",
    "denominator": "http_requests_total",
    "filter": { "environment": "production", "service": "employees-api" }
  },
  "evaluation": {
    "window": "10m",
    "frequency": "1m",
    "operator": "GreaterThan",
    "threshold": 0.02,
    "onMissingData": "notify"
  },
  "severity": "page",
  "notify": ["employees-api-oncall"],
  "runbook": "https://runbooks.internal.example/employees-api/error-rate",
  "autoResolveAfter": "20m"
}

The exact schema differs between providers. These fields appear in all of them under different names, and every one of them is a decision rather than a default worth accepting.
The signal is a ratio. Two failures out of four requests at 4am is a 50 per cent error rate and matters; two hundred failures out of a million at midday might be a third-party timeout you already know about. A count cannot tell those apart.
The window buys quiet at the cost of detection time. Ten minutes means you will not be woken by a thirty-second blip, and it also means ten minutes of a genuine outage pass before the alert fires. Write that number down as a choice, because it is the main lever between noise and speed.
The missing-data setting is the one people discover during an incident. If the service stops emitting metrics entirely — the container will not start, the whole revision is down — a rule waiting for a threshold has nothing to evaluate and stays silent. Treating no data as a reason to notify is what turns this from a partial check into a real one.
Severity is not decoration. It is the mapping from this rule to a human action: page someone now, or raise a ticket for the morning. If everything is a page, nothing is, and the fastest way to create alert fatigue is to skip this field.
The runbook link is what makes the alert actionable for whoever receives it, including the person who did not write the service. The auto-resolve stops someone investigating a spike that recovered forty minutes ago, which is a common way to waste an on-call evening.

Watching the deployment, because the deployment is the likely cause

YAML

      - name: Deploy, then watch before declaring success
        run: |
          set -uo pipefail
          ./scripts/deploy.sh employees-api-prod "$(cat digest.txt)"

          # Mark the charts, so the next person can see what changed and when.
          ./scripts/annotate.sh "deployed $(cat version.txt)"

          # Ten minutes of evidence from real traffic before the pipeline goes green.
          for attempt in $(seq 1 20); do
            rate=$(./scripts/error-rate.sh employees-api-prod 120)
            echo "check $attempt of 20: error rate $rate percent"
            if [ "$(printf '%.0f' "$rate")" -gt 2 ]; then
              echo "error rate above threshold after deployment" >&2
              ./scripts/rollback.sh employees-api-prod
              exit 1
            fi
            sleep 30
          done
          echo "no error rate change detected in the watch window"

A deployment is the most likely cause of a problem that did not exist an hour ago, so the pipeline is the right place to look for one. It already knows a change happened, which the alerting system does not.
The annotation is small and repays itself every incident. A chart showing a step change with a release marker next to it answers the first question anyone asks, and reconstructing deployment times from pipeline history costs minutes you will want back.
Twenty checks at thirty seconds gives ten minutes of real traffic. Pick the length from how long it takes a fault to appear in your service, and be honest that a low-traffic internal API may not produce enough requests in ten minutes to conclude anything.
This watch does not replace the alert rule. Plenty of problems start hours after a release, from a scheduled job, a cache expiring, or a slow data growth curve. The watch catches the fast, obvious regressions early; the alert catches everything else.
The automatic rollback here is only as good as the script it calls, and a rollback script nobody has run is a hope rather than a control. The next lesson is about making that step real.
The threshold is deliberately cruder than the alert rule: one number, one window, no ratio subtleties. A gate in a pipeline should be simple enough that the reason it failed is obvious from the log.

Common mistake

Common mistakes when starting out

Alerting on processor usage. It is high during a normal busy period, high during a healthy background job, and sometimes low during a complete outage. It fires when nothing is wrong, which trains people to dismiss it.

Creating an alert nobody can act on. If the recipient's only response is to look at it and hope, it is a notification with a siren attached. Either define the action, or send it somewhere people read at their convenience.

Routinely silencing or filtering an alert channel. This is the clearest signal that monitoring has already failed. Alerts that are muted do not detect anything, and a team that mutes one channel out of habit will mute the next one.

Adding an alert after every incident and never removing one. The set only grows, the noise floor rises, and the useful alerts end up indistinguishable from the rest. Deleting alerts is the main way to improve alerting, and it feels wrong the first few times.

Alerting on a raw count instead of a proportion. Fifty errors a minute is normal at peak and a serious incident at 3am, and one threshold cannot describe both.

Not handling missing data, so the alert cannot fire during the outage it was written for. A rule watching for a high error rate is silent when the service is emitting nothing at all.

Sending alerts to a group rather than a person or a rota. Everybody assumes somebody else is on it, and for a while everybody is right.

Alerting on the average response time. Averages hide the tail, and the tail is where the unhappy users are.

Treating a dashboard as detection. If a chart is the only thing between a failure and a customer noticing, the customer will get there first, because nobody watches a screen at 2am.

Trade-off

What every alerting decision costs

Sensitivity against noise. A tighter threshold and a shorter window catch more real problems and produce more false alarms. There is no setting that avoids both, so pick deliberately and revisit it with evidence from what actually fired.

Speed against confidence. Detection time is roughly the evaluation window plus the frequency. Shorter means faster and noisier. This is the same trade as above, seen from the clock rather than the inbox.

An error budget is a useful way to settle these arguments and it takes real work to set up: you need an agreed objective, a trustworthy measurement of it, and a team that accepts the consequence. The consequence is the point — when the budget is spent, reliability work takes priority over features. Without that agreement the objective is a number on a wiki.

Synthetic checks cost a small amount per run and catch failures nothing else sees. Run them on the few journeys that matter, from more than one location if the service is public, and accept that they add traffic to your own metrics.

On-call is paid for by people, in interrupted evenings and worse sleep. That is a genuine cost and the right response to it is fewer, better alerts with written runbooks, not a larger rota to absorb the noise.

One or two objectives per user-facing journey is plenty. Every additional one needs maintaining, and a service with fifteen objectives has none that anyone remembers.

Summary

Collecting signals is not detecting problems: an alert is a decision, made in advance, to interrupt a person
Alert on symptoms a user feels — not serving, error rate, latency, a business-level check — and investigate causes like processor usage afterwards
Every alert needs a named action and a runbook; alert fatigue removes your monitoring entirely, and the fix is deleting alerts rather than adding them
An alert wakes someone, a dashboard is opened on purpose, and objectives with error budgets turn is this bad enough into arithmetic
Watch the deployment itself, because a change made minutes ago is the most likely cause of a problem that is new

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Think about it

Which of these should wake someone?

Decide for each: page now, ticket for the morning, or neither. Processor at 92 per cent for ten minutes on two of six replicas. Photo uploads from the field staff app failing for every user for five minutes. The nightly employee export job failed once. The employees database disk at 91 per cent, growing about one per cent a week. A single unhandled exception on one search request.

For every one you chose to page on, write the first sentence of its runbook.

Show solution

Page: the photo uploads. Every user of one journey is failing, it is a symptom rather than a cause, and there is a clear action — check the recent release, check the storage account, roll back if a deployment lines up with the start time.

Ticket: the disk and the export job. The disk is a cause, and it earns a place because the outcome is certain and there is roughly nine weeks of lead time, which makes it work rather than an emergency. The export failing once needs someone to look at why and possibly re-run it; it does not need anyone awake, unless something downstream depends on it by a specific hour, which is worth knowing before you decide.

Neither: the processor and the single exception. Two replicas working hard is a service doing its job, and the platform may already be scaling. One exception is a log line and a possible bug report, and alerting on it guarantees a stream of notifications that will bury the upload alert when it fires.

The runbook sentence is the real exercise. If you cannot write the first instruction, the alert is not ready, because you have just discovered that the person you were going to wake would have nothing to go on.

There is a defensible variation: at a very small scale, a single unhandled exception may genuinely be rare enough to be worth seeing. The test is your actual volume — if it would fire twice a week, it will be ignored within a month.

Challenge

Delete some alerts

For a system you work on, list every alert that currently notifies a human. For each one write down: who receives it, what they do when it arrives, when it last fired, and whether that firing led to an action.

Propose the set to delete or downgrade, and the one or two symptom alerts that should exist and do not.

Show solution

Most lists come back with a majority of rows where the action column is blank or reads acknowledge it. Those are the ones to remove. An alert that has never led to an action is not protecting anything, and it is actively harmful because it consumes the attention the useful alerts need.

Downgrading is usually better than deleting for anything with a real lead time. Move it from a page to a ticket or a weekly review and it keeps its value without costing anyone their evening.

The gap in almost every audit is the outside-in check. Teams have plenty of internal metrics and no synthetic request on the path a user takes, so a certificate expiry or a routing change presents as total silence from monitoring while the service is completely unreachable.

Expect this to feel uncomfortable. Deleting an alert feels like removing a safety measure, and the honest framing is the opposite: a set of alerts small enough to be read is a safety measure, and thirty that people filter into a folder is not. If it helps, delete in one batch and keep the list, so restoring anything is easy if the next month proves you wrong.

One caution worth naming. Do not delete an alert on the grounds that it never fires — a quiet alert on a real symptom is doing its job. The test is whether a firing would tell you something you would act on, not how often it goes off.

Saved in this browser only.