Lesson 19 of 22Advanced18 min

Guardrails

By the end of this lesson

Constrain behaviour and handle unacceptable output safely.

A guardrail is a check you run around a model call, not inside it. There are two places to put one: before the request goes out, and after the response comes back. Nothing about the model changes; you are deciding what reaches it and what reaches the user.

The reason to separate the two is that they solve different problems. An input check stops work you do not want to do and do not want to pay for. An output check stops a response you would not stand behind from being shown as though you would.

One thing to settle before any of the code: a guardrail changes how often something happens. It does not make it impossible. That sentence is the difference between a guardrail used well and a guardrail used as an excuse, and the rest of this lesson keeps returning to it.

The order the checks run in, around the grounded pipeline from module 3:

Check the input, before you spend anything
Empty, enormous, off-topic, or carrying personal data that should not leave your network. All four are decided in your own code in microseconds, and each one avoided is a request you did not pay for.
Retrieve and generate as normal
Nothing new here. The grounded prompt, the abstain path when nothing is retrieved, the output cap, the citation resolution. Guardrails wrap this step; they do not replace anything in it.
Check the output, before the user sees it
Structure first, because it is cheap and exact: does it parse, does it validate, is it inside the length you allow, do the citations resolve. Then the judgement checks, which cost more and are less certain.
Decide what a failure means
Three outcomes: refuse, retry, or escalate to a person. This decision belongs in code, per failure type, written down in advance. Deciding it in the moment produces an interface that behaves differently every time something goes wrong.
Give the user something to do next
Whatever the outcome, the person asked a question. A block, a refusal and an escalation all need a message that says what happened and what to try, or they are dead ends that send people back to email.
Log the decision, not only the failure
Every block, refusal, retry and escalation, with the check that fired. These counts are how you find out that your topic restriction is rejecting a fifth of legitimate questions.

What is worth checking on the way in:

Length, in both directions: An empty question and a pasted forty-page document both need handling, and the second is the expensive one. A character cap on the user's message is one line and it removes the most common way an input cost forecast becomes wrong.
Topic restriction: The assistant answers questions about policy documents and the caller's own claims. A question about anything else should not reach the model. A small classifier or a keyword pass over a short allow-list of topics does this for a fraction of a penny, where asking the model to decide costs a full request.
Requests the feature is not for: Distinct from off-topic, and more important. Legal or medical questions, anything about another employee's pay, anything asking for a decision the assistant is not authorised to make. These get a specific message naming the right route, because the person has a real need and it is not this.
Personal data that should not leave your network: Bank details, national insurance numbers, card numbers. A pattern check that blocks the request and explains why is better than redaction, because silent redaction changes the question and the user never learns not to paste it.
Rate, per user: Not a content check, and it belongs here anyway. A per-user request limit bounds both your bill and the damage any single account can do. It is also the only check on this list that is a hard control rather than a judgement.

The input gate. Cheap checks first, and a useful message on every refusal

Python

MAX_QUESTION_CHARS = 600
IN_SCOPE_TOPICS = {"expenses", "travel", "policy", "leave"}

OUT_OF_SCOPE = (
    "I answer questions about company policy documents and your own expense "
    "claims. For anything else, the service desk can help: servicedesk@example.com"
)
WRONG_ROUTE = (
    "I cannot help with this one. Questions about pay, contracts or another "
    "employee's records go to your HR business partner."
)

class Verdict(BaseModel):
    allowed: bool
    message: str = ""
    reason: str = ""

def check_input(question: str, caller: User) -> Verdict:
    text = question.strip()

    if not text:
        return Verdict(allowed=False, message="Please type a question.", reason="empty")

    if len(text) > MAX_QUESTION_CHARS:
        return Verdict(
            allowed=False,
            message=f"Please shorten this to under {MAX_QUESTION_CHARS} characters, or ask about one thing at a time.",
            reason="too_long",
        )

    if contains_payment_or_id_number(text):
        return Verdict(
            allowed=False,
            message="Please remove bank, card or national insurance numbers, then ask again.",
            reason="personal_data",
        )

    topic = classify_topic(text)                 # small local classifier, five labels
    if topic == "hr_or_legal":
        return Verdict(allowed=False, message=WRONG_ROUTE, reason="wrong_route")
    if topic not in IN_SCOPE_TOPICS:
        return Verdict(allowed=False, message=OUT_OF_SCOPE, reason="off_topic")

    if not rate_limiter.allow(caller.id):
        return Verdict(
            allowed=False,
            message="You have reached the hourly limit. Please try again shortly.",
            reason="rate_limited",
        )

    return Verdict(allowed=True)

The checks run cheapest first. Length and pattern checks are free; the classifier costs something; the rate limiter touches a shared store. Ordering them this way means the common rejections cost nothing, which matters when a fifth of traffic is a paste.
Every refusal carries a message written for the person and a reason written for your logs. The two are separate fields on purpose: the message can be rewritten by whoever owns the copy without breaking the dashboard that counts reasons.
The personal data check blocks rather than redacts. Redaction silently changes the question, so the answer may address something the user did not ask, and they learn nothing about what not to paste. Blocking is the less clever option and the more honest one.
hr_or_legal is separated from merely off-topic because the response differs. Somebody asking about their contract has a genuine need and a correct route; sending them the generic out-of-scope line wastes their time.
The classifier is small and local. Asking the main model whether a question is in scope costs a full request and an extra wait to reject something you were not going to answer, and its verdict is no more dependable than a classifier trained on your own traffic.
What this gate does not do is look for manipulative phrasing. That belongs to the security lesson, and the short version is that a phrase list is not where that problem is solved.

The output gate, and the three things a failure can mean

Python

RETRYABLE = {"too_long", "unresolved_citation", "schema_invalid"}
ESCALATED = (
    "I am not confident enough to answer this one, so I have passed it to the "
    "finance team. Reference {ref}. They usually reply within a working day."
)

def check_output(result: Answer) -> list[str]:
    problems = []
    if not result.grounded:
        problems.append("not_grounded")
    if result.unresolved_citations:
        problems.append("unresolved_citation")
    if len(result.text.split()) > MAX_ANSWER_WORDS:
        problems.append("too_long")
    if result.schema_name and not validates_against(result.text, result.schema_name):
        problems.append("schema_invalid")
    if reads_as_personal_advice(result.text):
        problems.append("gives_advice")
    return problems

def guarded_answer(question: str, caller: User) -> Reply:
    verdict = check_input(question, caller)
    if not verdict.allowed:
        log.info("input blocked", extra={"caller": caller.id, "reason": verdict.reason})
        return Reply(text=verdict.message, outcome="blocked")

    problems: list[str] = []
    for attempt in (1, 2):
        # strictness=2 tightens the instruction and lowers the output cap. Retrying
        # an identical request at temperature 0 returns the same answer and bills twice.
        result = answer(question, caller, strictness=attempt)

        if result.abstained:
            return Reply(text=helpful_refusal(question, caller), outcome="abstained")

        problems = check_output(result)
        if not problems:
            return Reply(text=result.text, sources=result.citations, outcome="answered")

        log.warning("output check failed", extra={"attempt": attempt, "problems": problems})
        if not all(problem in RETRYABLE for problem in problems):
            break

    ref = escalations.create(question=question, caller=caller.id, problems=problems)
    return Reply(text=ESCALATED.format(ref=ref), outcome="escalated")

check_output returns the list of what failed rather than a boolean, because the action depends on which check fired. A structural failure is worth one retry; an answer that reads as personal financial advice is not, and retrying it spends money to produce something similar.
The retry changes the request. This is the detail most implementations get wrong: at temperature 0 a second identical call returns effectively the same answer, so a retry loop with nothing varied is a billing feature. Tighten the instruction, lower the cap, or send fewer passages.
Two attempts, not five. Each one is a full model call with its own latency, and a person is waiting. If two constrained attempts cannot produce a valid answer, a human should see the question.
The abstain path is checked before the output checks and returns early. A legitimate refusal is a correct outcome, and running it through the failure machinery would escalate every question your documents do not cover.
helpful_refusal is the function worth writing carefully. It should say the documents do not cover this, list the closest documents it did find with links, and give one route to a person. A bare "I could not find this" is technically honest and practically a dead end.
Escalation creates a real record and hands back a reference the user can quote. An escalation the user cannot see the status of is indistinguishable from being ignored.

The distinction this lesson is built on. Both belong in the system, and only one of them enforces anything:

	A guardrail	A hard control
What it is	A check on text, sometimes made by another model	Code, a permission, or a credential boundary
What it gives you	A lower rate of something unwanted	An outcome that cannot occur
In this assistant	Topic classification, advice detection, length and structure checks	The permission check in the expenses API, a read-only database role, a human approving a payment
How it fails	Quietly, on the inputs nobody tested, at some rate you can only measure	Loudly, as a refusal, with a log line
Under deliberate pressure	Weakens, because it is a judgement about text	Holds, because it does not consult the text
Honest claim	"Advice-shaped answers are down to roughly 1 in 300, measured"	"No caller can read another employee's claims"

Watch out

Guardrails are probabilistic. They are not a security boundary

Every check in this lesson that looks at text is a judgement, and judgements about language have an error rate. A topic classifier misfiles questions. An advice detector misses phrasing it has not seen. A model asked to review another model's output is running the same technique with the same failure mode, so its verdict is an opinion with a rate attached, not a gate.

So a guardrail reduces how often something bad happens. It never makes it impossible. Never place one where a hard control is required. If the requirement is that a caller cannot read another employee's claims, that requirement lives in the authorisation check inside your API, where it is enforced by code that does not read any text. A prompt instruction and an output filter are, at best, two more layers in front of it.

The practical test, and it is a short one: if this check fails silently on one request in a thousand, is that acceptable? Sometimes yes — an over-long answer is a poor experience. Sometimes absolutely not — one leaked salary figure is an incident, and no measured rate makes it acceptable. Requirements in the second category need a control that cannot be argued with.

The same applies to how you describe the feature. "Guardrails prevent the assistant from discussing salaries" is a claim you cannot support. "Salary questions are classified out of scope, and the underlying API refuses salary reads for every caller" describes what is actually true, and names the part doing the real work.

One more consequence worth stating: because guardrails have a rate, you have to measure it. A check nobody has counted is a check nobody knows the value of, and the evaluation set from the previous lesson is where those counts come from.

Common mistake

The guardrail mistakes that reach users

The dead-end refusal. "I cannot help with that" and nothing else is the most common guardrail failure in production, and it is a design decision nobody made. Every block needs to say what happened, why, and one thing to try next. A refusal that routes somebody usefully is a good outcome; a refusal that stops the conversation sends them to email and teaches them the assistant is not worth using.

Checking output and never input. Output checks are the interesting engineering, so they get written first. Input checks are where the cost savings and the simplest wins are — an empty question, a pasted document, a question about a colleague's pay — and they run before you have spent anything.

Retrying without changing the request. At temperature 0 the second call returns the same answer and doubles the cost. If a retry is the plan, something in the request has to be different.

A keyword blocklist treated as a topic restriction. Blocking on a word list rejects legitimate questions that happen to contain a word and misses everything phrased differently. A small classifier trained on your own traffic is both more accurate and easier to correct when it is wrong.

Silent rewriting. A guardrail that edits an answer before showing it — trimming a sentence, softening a phrase — produces output that matches neither what the model said nor what your logs recorded. When a check fails, refuse, retry or escalate. Do not quietly patch.

No counts. If you cannot say what share of requests are blocked, refused and escalated, and which check fired, you cannot tell a working guardrail from one that has been rejecting a fifth of legitimate questions since March.

Summary

Guardrails are checks around a model call: one set before the request, one set on the response
Input checks stop work you do not want to pay for; output checks stop a response you would not stand behind
Decide per failure type whether the outcome is refuse, retry or escalate, and make a retry change the request
A refusal still needs to be useful — say what happened, show the nearest sources, and give one route to a person
A guardrail lowers the rate of something unwanted; it is not a security boundary, so anything that must never happen needs a hard control

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Try it yourself

Measure your own false positives

Add the input gate to your assistant, logging the reason on every block but allowing the request through regardless for now. Run your evaluation set plus thirty real questions from colleagues through it.

Count how many legitimate questions each check would have blocked. Then decide which checks you would switch on.

Show solution

Running a check in report-only mode before enforcing it is the technique worth taking from this exercise, and it applies well beyond AI features. You get the false positive rate before any user experiences one.

The topic classifier is usually the worst offender. Questions that mix two subjects — a travel question that mentions a contract — land wherever the classifier leans, and a strict allow-list rejects them. Common fixes are to widen the in-scope set, or to let borderline cases through and rely on the abstain path, which fails more gracefully than a block.

The length cap is normally clean, and the surprise is how many pastes it catches. Look at what those pastes are: if people are pasting policy text into a tool that already has the policy indexed, the interface has failed to explain itself, and that is a product fix rather than a guardrail one.

Expect the personal data check to fire on something harmless, such as a claim reference that matches a card-number pattern. Tighten the pattern rather than removing the check.

Why report-only first, rather than shipping and watching: a blocked user does not file a bug, they stop using the feature. A false positive you never hear about is the most expensive kind, and this is the cheapest way to find them.

Think about it

Which of these four can a guardrail satisfy?

Finance gives you four requirements. The assistant must never disclose one employee's claims to another. Its answers must not read as personal financial advice. Answers must be under 200 words. It must not approve a claim.

For each, decide whether a guardrail is the right mechanism, and if not, say what is.

Show solution

Never disclosing another employee's claims is not a guardrail requirement. "Never" needs a hard control: the permission check inside the expenses API, evaluated against the authenticated caller, plus a retrieval filter on audience tags. An output check that scans for other people's names is a weak extra layer and must not be the thing you are relying on.

Advice-shaped answers are a reasonable guardrail. There is no exact definition of advice, the cost of an occasional miss is a poorly worded answer rather than an incident, and the honest report is a measured rate rather than a guarantee. Pair it with a standing line in the interface saying the assistant summarises policy and does not give advice.

Under 200 words is neither a guardrail nor a hard control in the interesting sense — it is a word count. Cap the output tokens, check the length, and truncate or retry. Exact, free, and no judgement involved.

Not approving a claim is settled by not building the tool. There is no approve function in the allow-list, and the service account behind the tools has no write permission. A guardrail that watches for approval language would be theatre; capability the model does not have cannot be talked into existing.

The pattern across the four: ask what the failure costs. Where a single occurrence is unacceptable, you need a control that does not read text. Where occurrences are merely undesirable, a measured guardrail is a reasonable and normal engineering answer.

Saved in this browser only.