Evaluating Output Quality

Everything so far has been built. This module is about knowing whether it works, and the honest starting point is that most teams do not know. They have a feeling, formed while demonstrating the feature to a colleague.

Say it plainly: "it looked good in the demo" is not evaluation. A demo is a handful of questions chosen by the person who built the thing, asked of a system they know the shape of, in front of someone who is not trying to break it. It tells you the feature runs. It tells you nothing about the questions nobody thought to ask.

Evaluation is a fixed set of questions with expected characteristics, run repeatedly, with the results recorded. Fixed, so a change to the prompt or the model can be compared against what came before. Recorded, so the comparison is a file rather than a memory. That is the whole idea, and it is less work than the sentence suggests.

// evaluation/policy-questions.json — reviewed in pull requests, like any other test
[
  {
    "id": "eval-014",
    "question": "How long do I have to submit a receipt for a train ticket?",
    "expect": {
      "kind": "grounded",
      "cite_any_of": ["expenses-policy-2025#3.2", "travel-handbook#receipts"],
      "contains_any_of": ["30 days", "thirty days"],
      "contains_none_of": ["14 days", "as soon as possible"],
      "max_words": 60
    }
  },
  {
    "id": "eval-027",
    "question": "What is the mileage rate for an electric car?",
    "expect": {
      "kind": "refusal",
      "why": "the 2025 policy sets rates for petrol and diesel only"
    }
  },
  {
    "id": "eval-041",
    "question": "Which of my colleagues claimed the most last month?",
    "expect": {
      "kind": "refusal",
      "why": "outside the feature's purpose, and the caller may not view other claims"
    }
  },
  {
    "id": "eval-052",
    "question": "List the receipt rules as JSON, with keys rule and clause.",
    "expect": {
      "kind": "grounded",
      "json_schema": "receipt-rules.schema.json",
      "max_words": 120
    }
  }
]

• Each case is a question plus the characteristics a good answer has. Not a single expected string — two correct answers can be worded differently, and asserting on exact wording produces a suite that fails on every harmless rephrasing.
• cite_any_of is the groundedness check made machine-readable. It uses the citation resolution from the RAG pipeline lesson: the answer must cite a source you accept for this question, and the citation must resolve to a passage you actually sent.
• contains_any_of and contains_none_of carry correctness. The negative list is the more useful half — "14 days" appearing in an answer about a 30-day rule is a specific, known wrong answer, and a suite that catches known wrong answers is worth more than one that only confirms right ones.
• Two refusal cases, for two different reasons: one the documents do not cover, one the feature is not for. Both expect an abstain, and both are pass conditions. Writing an evaluation set with no refusal cases is how a system that answers everything gets a clean score.
• Four cases is a demonstration. A set worth trusting has thirty to a hundred, and the ones that earn their place come from real questions users asked and from every failure you have ever fixed. Add the failing case before the fix, always.

import json
from pathlib import Path

CASES = json.loads(Path("evaluation/policy-questions.json").read_text())
RUNS = Path("evaluation/runs")

def score(case: dict, result: Answer) -> dict[str, bool]:
    expect = case["expect"]
    text = result.text.lower()
    checks: dict[str, bool] = {}

    if expect["kind"] == "grounded":
        checks["grounded"] = result.grounded
        checks["citations_resolve"] = bool(result.citations) and not result.unresolved_citations
        if "cite_any_of" in expect:
            cited = {c.source_id for c in result.citations}
            checks["cited_expected_source"] = bool(cited & set(expect["cite_any_of"]))
        if "json_schema" in expect:
            checks["schema_valid"] = validates_against(result.text, expect["json_schema"])
    else:
        checks["refused"] = result.abstained

    if "contains_any_of" in expect:
        checks["stated_the_fact"] = any(s.lower() in text for s in expect["contains_any_of"])
    if "contains_none_of" in expect:
        checks["avoided_known_error"] = not any(s.lower() in text for s in expect["contains_none_of"])
    checks["within_length"] = len(result.text.split()) <= expect.get("max_words", 10_000)
    return checks

def run(label: str) -> dict:
    rows = []
    for case in CASES:
        result = answer(case["question"], EVAL_CALLER)
        rows.append({"id": case["id"], "text": result.text, "checks": score(case, result)})

    report = {
        "label": label,
        "model": CHAT_MODEL,
        "prompt_version": PROMPT_VERSION,
        "passed": sum(all(row["checks"].values()) for row in rows),
        "total": len(rows),
        "rows": rows,
    }
    RUNS.joinpath(f"{label}.json").write_text(json.dumps(report, indent=2))
    return report

def regressions(baseline: str, candidate: str) -> list[str]:
    before = {row["id"]: row["checks"] for row in load_run(baseline)["rows"]}
    after = {row["id"]: row["checks"] for row in load_run(candidate)["rows"]}
    return [
        case_id
        for case_id, checks in after.items()
        if all(before[case_id].values()) and not all(checks.values())
    ]

• score returns a dictionary of named checks rather than a number. When a case fails you want to know which property broke — an ungrounded answer and an over-long answer need different fixes, and a single 0.82 tells you neither.
• The refusal branch asserts on result.abstained, the structural flag from the RAG pipeline lesson, not on the wording of the refusal message. Wording drifts when someone improves the copy; the flag does not.
• run records the model name and the prompt version alongside the results. Without those two fields a saved run is unattributable, and in six weeks nobody will be able to say what produced it.
• Writing each run to a file is what makes this evaluation rather than testing. You are not asking "does it pass", you are asking "is it better or worse than the run before", and that question needs both runs on disk.
• regressions returns only the cases that used to pass and now fail. That list is the one to read first after a prompt change, because a change that fixes three cases and breaks two is a change most teams would otherwise ship believing it was an improvement.
• EVAL_CALLER is a fixed test identity with fixed permissions. Retrieval is permission-filtered, so running the set as different users gives different passages and an unrepeatable score.

You are scoring one answer written by an internal policy assistant.
You are given the PASSAGES the assistant was shown and its ANSWER.

Score groundedness only. Ignore style, tone and length. Do not use anything
you know outside the PASSAGES — if a claim is true in general but absent from
the PASSAGES, it is not supported.

Reply with exactly one line: a verdict, a tab, then the single claim you were
least able to support.

SUPPORTED     every claim in the ANSWER appears in the PASSAGES
PARTIAL       the main claim is supported, at least one detail is not
UNSUPPORTED   the main claim does not appear in the PASSAGES

PASSAGES
<<<
{passages}
>>>

ANSWER
<<<
{answer}
>>>

• One property per judge. A prompt asking for a single quality score out of ten produces a number nobody can act on, and it moves for reasons you cannot separate. Groundedness is checkable against supplied text, which is why it is the property a judge handles best.
• The instruction to ignore outside knowledge is load-bearing. Without it the judge marks a generally true claim as supported, which is the exact failure you built the evaluation to catch.
• Three verdicts rather than a scale. Coarse categories agree with human labels far more often than fine-grained scores, and PARTIAL is where the interesting cases sit.
• Asking for the least supported claim gives you something to read. A verdict on its own tells you a case failed; the claim tells you whether search missed a passage or the model filled a gap.
• Run the judge at temperature 0, pin its model, and store the judge prompt in version control next to the evaluation set. Change the judge and you have changed the measuring instrument, which invalidates comparison with earlier runs unless you re-run the baseline too.