Cost and Latency

Two numbers decide whether an AI feature survives contact with a finance review: what one request costs, and how long a person waits for it. Both are controllable, and both are usually left to whatever the first working version happened to do.

Cost comes from tokens, counted separately in each direction because they are priced differently. Output is commonly several times the input rate, so a hundred tokens of answer can cost more than a thousand tokens of prompt. Any cost work that treats a token as a token will aim at the wrong half.

Latency has a different shape. Input is processed largely in parallel, so a longer prompt adds modestly to the wait. Output is produced one token at a time, so the length of the answer is the dominant term. That is why a shorter answer is both the cheapest and the fastest change available to you.

# Placeholder rates in pence per million tokens, chosen to show the arithmetic.
# Real prices differ by model and change without warning. Read them from
# configuration and check the provider's published rates before forecasting.
RATES = {
    "gpt-4.1-mini": {"input": 10, "output": 40},
    "gpt-4.1": {"input": 150, "output": 600},
}
DAILY_BUDGET_PENCE = 50_000

def record_cost(model: str, usage, feature: str, caller_id: str) -> float:
    rate = RATES[model]
    pence = (
        usage.prompt_tokens * rate["input"] + usage.completion_tokens * rate["output"]
    ) / 1_000_000

    log.info(
        "model cost",
        extra={
            "feature": feature,                 # "policy_answer", "claim_summary", "eval_judge"
            "model": model,
            "input_tokens": usage.prompt_tokens,
            "output_tokens": usage.completion_tokens,
            "cached_input_tokens": getattr(usage, "cached_tokens", 0),
            "pence": round(pence, 4),
            "caller": caller_id,
        },
    )

    spent = daily_spend.add(pence)               # integer pence in a shared counter
    if spent > DAILY_BUDGET_PENCE and alerts.first_today("ai-budget"):
        alerts.send("AI daily budget exceeded", spent_pence=round(spent))
    return pence

• Input and output are multiplied by separate rates, which is the whole reason this function exists rather than a single token count. Get this wrong and you will optimise the prompt while the answer length is what is costing you.
• The feature field is what makes the log usable. An application-wide total tells you spend went up; a breakdown by feature tells you the evaluation judge is now two thirds of your bill, which is a different conversation and a much easier fix.
• cached_input_tokens is recorded separately because providers bill cached input at a reduced rate. Without the field you cannot tell whether your caching is working — you will see a lower bill and be unable to attribute it. Field names for this differ between providers and have changed, so read the current response reference rather than trusting the attribute name here.
• Cost per request is the number that scales. A monthly total tells you what happened; cost per request multiplied by expected volume tells you what happens when the feature is rolled out to the whole company, which is the question you will be asked.
• The counter holds integer pence. Accumulating sub-penny floats over a million requests drifts, and if the total ever reaches a finance system it needs to be exact rather than close.
• A budget alert is not a cap. It tells a person that spend crossed a line, which is the right first step. If you need spending to actually stop, that is a separate control: refuse new requests, drop to a cheaper model, or queue the work — and it needs deciding before the day it matters.

from functools import lru_cache
import hashlib

MAX_PASSAGES = 4
MAX_HISTORY_TURNS = 4
MAX_ANSWER_TOKENS = 250
ANSWER_TTL_SECONDS = 6 * 3600

@lru_cache(maxsize=8192)
def embed_question(question: str) -> tuple[float, ...]:
    """Distinct questions repeat constantly in an internal tool. One call each."""
    return tuple(embed(question.strip().lower()))

def answer_cache_key(question: str, caller: User) -> str:
    material = "|".join(
        [
            CHAT_MODEL,                                  # a model change invalidates
            PROMPT_VERSION,                              # a prompt change invalidates
            INDEX_VERSION,                               # a re-ingest invalidates
            question.strip().lower(),
            ",".join(sorted(caller.audience_tags)),      # permissions change what is retrieved
        ]
    )
    return hashlib.sha256(material.encode()).hexdigest()

def build_messages(question: str, passages: list[Passage], history: list[dict]) -> list[dict]:
    # Stable content first and byte-identical, so a provider-side prompt cache can
    # reuse the prefix. Anything that varies per request goes last.
    return [
        {"role": "system", "content": SYSTEM},
        *history[-MAX_HISTORY_TURNS * 2 :],
        {
            "role": "user",
            "content": context_block(passages[:MAX_PASSAGES]) + f"\n\nQUESTION\n{question}",
        },
    ]

def cheap_answer(question: str, caller: User) -> Answer:
    key = answer_cache_key(question, caller)
    cached = answer_cache.get(key)
    if cached:
        metrics.increment("answer_cache_hit")
        return cached

    passages = search(embed_question(question), caller, top_k=MAX_PASSAGES)
    result = generate(
        build_messages(question, passages, load_history(caller)),
        max_completion_tokens=MAX_ANSWER_TOKENS,
    )
    if result.grounded:                                  # never cache an abstain or a failure
        answer_cache.set(key, result, ttl_seconds=ANSWER_TTL_SECONDS)
    return result

• The cache key is where answer caching goes wrong, so it is worth reading slowly. Model, prompt version and index version are all in it, which means any change you make invalidates the cache automatically rather than serving answers produced by a system that no longer exists.
• Audience tags are in the key because retrieval is permission-filtered. Leave them out and two callers with different permissions share a cached answer, which turns a performance optimisation into a disclosure. If a cache key omits anything that affects what was retrieved, it is not safe.
• The time to live is short deliberately. Documents change, and a six-hour window bounds how long a superseded policy can be repeated. Pick the number from how often your documents actually change, and add an explicit invalidation when ingestion runs.
• Only grounded answers are cached. Caching an abstain means a question stays unanswerable for six hours after somebody uploads the document that answers it, and caching a failure serves the failure repeatedly.
• Provider-side prompt caching is a different mechanism and it has one requirement: the repeated part must come first and be identical byte for byte. Putting a timestamp or the user's name at the top of your system prompt defeats it completely, and that is a surprisingly common way to lose the discount.
• The trimming is unremarkable and it is where most of the input saving actually comes from. Four passages, four turns, a cap on the answer. Every one of those numbers should be a named constant you can change and re-evaluate, not a literal buried in a function.