Security Logging
By the end of this lesson
Log security-relevant events without recording sensitive data.
Something happened three weeks ago. An order was approved at a price nobody can account for, and the employee named on it says it was not them. Whatever you can reconstruct now, you reconstruct from your logs.
That is the purpose of security logging, and it is a different purpose from diagnostics. Diagnostic logging helps you fix a fault while it is in front of you. Security logging answers questions after the fact, for people who were not there: who did this, to what, when, from where, and what did the system decide at the time.
Two failures are common, and they pull in opposite directions. Logging too little leaves you with nothing to investigate, which is discovered at the worst possible moment. Logging too much turns your log store into a copy of your most sensitive data, held somewhere with broader access and longer retention than the database it came from.
This lesson is about landing between the two: recording the events that matter, in a form you can query, without the store becoming a liability of its own.
Events worth logging in the orders application. The test for each one is whether you would want it in front of you during an investigation:
- Sign-in success — the account, the time, and the source address. Successes matter as much as failures, because the question is usually when access began rather than whether someone guessed
- Sign-in failure — the account that was attempted and a reason category such as unknown account, wrong credential or locked out. Never the credential supplied
- Sign-out, session end, and token or session revocation, which is how you establish when access stopped
- Authorisation denials — the 403s. One is ordinary. Forty in a minute from one account is a pattern, and it is only visible if each one was recorded
- Permission and role changes: who was granted what, by whom, and when. This is the single most valuable audit record in most business applications
- Account lifecycle events — created, disabled, locked, unlocked, password reset requested and completed, second factor enrolled or removed
- Anything that moves money or changes access: order approval, price override, discount beyond a threshold, refund, credit limit change, bank detail change
- Administrative and configuration changes, including deployments — who released what, when, and from which pipeline run
- Data exports and bulk reads. A large export is what a leak looks like from the inside, and it is indistinguishable from ordinary work unless you record the volume
- Failures of your own security controls: token validation rejections, anti-forgery failures, rate limits triggered, requests refused by validation. These are the events that say the machinery is being exercised
public sealed partial class OrderApprovalService
{
private readonly ILogger<OrderApprovalService> _logger;
private readonly IOrderRepository _orders;
public OrderApprovalService(ILogger<OrderApprovalService> logger, IOrderRepository orders)
{
_logger = logger;
_orders = orders;
}
public async Task<bool> ApproveAsync(
int orderId, ClaimsPrincipal user, CancellationToken ct)
{
var employeeId = user.FindFirst("employee_id")?.Value ?? "unknown";
// FLAWED SHAPE, commented out deliberately. Serialising a whole request
// logs every field it has today and every field it gains later, which is
// how a token or a customer's bank details reach a log aggregator.
// _logger.LogInformation("Approval request {Request}", request);
var order = await _orders.FindAsync(orderId, ct);
if (order is null || !user.IsInRole("OrdersManager"))
{
LogApprovalDenied(orderId, employeeId);
return false;
}
order.Approve(employeeId);
await _orders.SaveAsync(order, ct);
LogOrderApproved(orderId, employeeId, order.Total, order.Currency);
return true;
}
[LoggerMessage(EventId = 4101, Level = LogLevel.Information,
Message = "Order {OrderId} approved by employee {EmployeeId} for {Total} {Currency}")]
private partial void LogOrderApproved(
int orderId, string employeeId, decimal total, string currency);
[LoggerMessage(EventId = 4102, Level = LogLevel.Warning,
Message = "Approval of order {OrderId} denied for employee {EmployeeId}")]
private partial void LogApprovalDenied(int orderId, string employeeId);
}- Each field is named and chosen. That is the whole discipline: you decide what goes in, rather than handing an object to a serialiser and finding out later what it contained.
- The commented line is the shape to recognise in review — any log call whose argument is a whole request, entity, options object or configuration section. It is not wrong today; it is wrong the moment somebody adds a field, and nobody revisits a log line when adding a property.
- Stable event ids let you search and alert on one specific event without matching message text. Message wording gets improved; an id does not, and a dashboard built on the id keeps working.
- The employee id comes from the authenticated principal, not from the request. An id supplied by the caller would make the audit record a statement by the caller about itself, which is worth less than nothing in an investigation.
- A denial is logged at Warning rather than Information, because volume is the signal. One denial is someone clicking the wrong thing; a burst is worth waking up for, and an alert can only be built on events that were recorded.
- The total and currency are here because this event is about money, and knowing the amount is the point of the record. That is the test for any field: name what the investigation will need, and stop.
- LoggerMessage source generation produces the logging method at compile time, so nothing is formatted or allocated when the level is disabled. A plain call with an interpolated string builds the string whether or not anything consumes it, which matters on a hot path. It also has a useful side effect here: the fields are declared as parameters, so adding one is a code change somebody reviews.
What never goes in a log line, and what to record instead. The replacement column is the part that keeps the log useful:
- Passwords and secrets
- Never, in any form, including hashed and including on a failure. Log the account and a reason category. The login endpoint deserves particular attention, because a single request-body log line there puts every password into the store.
- Tokens, API keys and session identifiers
- A logged token is a working credential sitting in a store many people can read. If you need to correlate, log the token's own identifier claim, or its expiry, or nothing. The same applies to the Authorization header, which is why headers should be logged from an allow-list rather than wholesale.
- Full card numbers and bank details
- Do not hold them in logs at all. Record the payment processor's reference, which is what you would use to investigate anyway. If a partial card number is genuinely needed for support, last four digits only, and check what your payment obligations require before adding even that.
- Personal data beyond what you can justify
- An internal employee or customer id identifies the record for an investigation without putting names, addresses, dates of birth or health information into a store with wide access and long retention. Where a name genuinely helps, log it at the one event that needs it rather than on every line.
- Whole request and response bodies
- This is the one that causes most accidental exposure, because it is added as a temporary debugging measure and then stays. Log a correlation id instead: the request is identified, and whoever investigates can ask for the specific detail rather than finding it already stored.
- Query strings, verbatim
- Query strings collect things they should not — a reset token, an invitation code, a search over customer names. Log the route template and the parameters you chose, rather than the raw string.
- Exception detail with values in it
- An exception message often contains the data that caused it, and a database error can carry a fragment of the statement. Log the exception type, the correlation id and a safe message, keep full detail where access is tighter, and never return it to the caller.
Diagnostic and security logging are both useful and they are not the same stream. Confusing them is why security events get sampled away or deleted early:
| Diagnostic logging | Security and audit logging | |
|---|---|---|
| Question it answers | Why is this failing right now? | Who did what, to what, and when? |
| Written for | The engineer on call today | Whoever investigates in six months, who was not there |
| Typical retention | Days to weeks | Months to years, decided deliberately rather than by default |
| Sampling | Reasonable. A representative slice is often enough | Not reasonable. A missing event is the one you needed |
| Who may delete or change it | Whoever manages the pipeline | As few people as possible, and not the application itself |
| What belongs in it | Timings, state, stack traces, enough to reproduce the fault | Actor, action, target, outcome, time, source, correlation id |
| Cost of getting it wrong | A harder debugging session | An investigation with nothing to work from, or sensitive data in a widely read store |
Summary
- Security logging answers who did what, to what, and when, for someone investigating long after the event
- Log authentication outcomes, authorisation denials, permission changes, and anything that moves money or access
- Logs are a data store with the same protection and retention obligations as your database, and usually weaker controls
- Never log passwords, tokens, full card numbers or unjustified personal data, and never log whole request bodies
- A correlation id lets you investigate without recording payloads, and stable event ids let you alert without matching text
Practice
Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.
Try it yourself
Rewrite the sign-in logging
A sign-in handler currently logs the whole request body on failure so support staff can see what the employee typed, and logs nothing at all on success.
Rewrite what it records. Say what each field is for, and what you removed.
Show solution
Remove the body. It contains the password on every failed attempt, and a failure caused by a typo in the email means the password is a real one from another attempt. There is no configuration of a log store that makes this acceptable.
Log the failure as: the account that was attempted, a reason category, the source address, the correlation id, and the time. The category is what support actually needed — unknown account, wrong credential, locked out, expired password — and it answers their question without storing the credential.
Add the success, which was missing and matters more than people expect. An investigation almost always starts with when access began, not with whether someone failed first. Record the account, the source, the time, and the correlation id.
Keep both at a level that survives. Failures at Warning, so a burst can be alerted on; successes at Information, since they are the baseline. Give each a stable event id so an alert does not depend on message wording.
One design point worth naming: logging the attempted email address is a judgement call, because someone typing their personal address into the wrong form has now put it in your log store. Logging the account id when the account exists, and a hash or a truncated form when it does not, is a defensible refinement. Logging the full body is not defensible under any reading.
[LoggerMessage(EventId = 1001, Level = LogLevel.Information,
Message = "Sign-in succeeded for account {AccountId} from {SourceAddress}. Correlation {CorrelationId}")]
private partial void LogSignInSucceeded(
string accountId, string sourceAddress, string correlationId);
[LoggerMessage(EventId = 1002, Level = LogLevel.Warning,
Message = "Sign-in failed for account {AccountId} from {SourceAddress}, reason {Reason}. Correlation {CorrelationId}")]
private partial void LogSignInFailed(
string accountId, string sourceAddress, SignInFailureReason reason, string correlationId);
// SignInFailureReason is an enum: UnknownAccount, InvalidCredential,
// LockedOut, PasswordExpired, SecondFactorRequired. A closed set, so no
// free-text message can carry a value nobody reviewed.Think about it
The deletion request reaches the log store
A customer asks for their personal data to be removed. Your team deletes their records from the database and confirms it is done. Someone points out that customer names, addresses and delivery notes appear across eighteen months of application logs.
What does this tell you about the design, and what would you change? Say plainly which parts of this are engineering decisions and which are not.
Show solution
The design problem is that the logs are a second copy of the data, with none of the structure that made the first copy manageable. The database had one row to delete. The logs have the same information smeared across millions of lines, in free text, in a store that may not even support deletion by content.
What to change, in order. First, stop adding to it: log identifiers rather than payloads, so a log line references a customer instead of describing one. Second, set a retention period per stream and enforce it, so the problem is bounded to that window instead of being unbounded. Third, keep the small number of events that genuinely need personal detail in a separate stream with tighter access and a retention period chosen for it.
For the eighteen months already written, the options are limited and worth being honest about: rely on the retention window expiring, delete by time range, or work with whatever redaction your aggregator supports. None of them is as clean as deleting a row, which is the argument for the design change rather than a remedy for this request.
The engineering decisions are: what gets logged, in what shape, for how long, and who can read it. Those are entirely yours and this lesson is about making them deliberately.
What is not an engineering decision is whether the logs are in scope for the request, what the retention period has to be, and what counts as complete. That depends on obligations that vary by jurisdiction and by the kind of data, and it belongs with whoever owns that in your organisation. Bring them the accurate technical picture — here is what we hold, where, for how long, and who can read it — and let them decide. Guessing at the answer yourself is the mistake to avoid.
Knowledge check
Nothing is recorded and there is no score. The explanation appears either way.
Saved in this browser only.