Skip to main content
ANVISoftware Solutions
Lesson 4 of 22Beginner15 min

Large Language Models

By the end of this lesson

Explain in plain terms what an LLM does when it responds.

A large language model does one thing: given a piece of text, it produces a probability for every possible next piece of text. That is the entire operation. Everything a model appears to do — answering, summarising, writing code, arguing with you about a date — is that one operation repeated.

Repeated how? It predicts one piece, picks one, sticks it on the end of the text, and predicts again from the longer text. It keeps going until it produces a signal meaning "finished", or until it hits the output limit you set. A paragraph of reply is a few hundred passes through that loop.

What happens between your API call and the reply arriving:

  1. Your text is split into pieces

    The whole prompt — your instructions, the conversation so far, any document extracts you supplied — is broken into tokens, the sub-word units the model works in. The next lesson covers these in detail.

  2. The model scores every possible next token

    One pass produces a number for each token in its vocabulary, representing how likely that token is to come next. Not one answer. A ranked distribution over tens of thousands of options.

  3. One token is chosen

    Selection is where your settings apply. The highest-scoring token may be chosen, or a choice may be sampled from among the plausible ones. This step is the source of variation between runs.

  4. The chosen token is appended and the loop repeats

    The model now predicts from your prompt plus the tokens it has already produced. This is why a reply that starts down a wrong path tends to continue down it — its own earlier words are part of the input.

  5. Generation stops

    The loop ends at an end-of-response token, at your maximum output length, or at a stop sequence you specified. Hitting the length cap truncates mid-sentence, which is worth handling rather than displaying.

The same prompt, twice, at two temperatures
Python
question = "In one sentence, what is the deadline for submitting an expense claim?"

def ask(temperature: float) -> str:
    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        temperature=temperature,
        max_completion_tokens=60,
        messages=[
            {"role": "system", "content": "Answer from the extract only. If it does not say, reply: not stated."},
            {"role": "user", "content": f"Extract: {POLICY_EXTRACT}\n\nQuestion: {question}"},
        ],
    )
    return response.choices[0].message.content

for _ in range(3):
    print("varied:", ask(temperature=1.0))

for _ in range(3):
    print("tight: ", ask(temperature=0.0))
  • temperature controls the selection step. At 0 the model takes the highest-scoring token at each position. Raise it and lower-ranked tokens get a real chance, so wording drifts and occasionally so does substance.
  • The three calls at temperature 1.0 will usually differ in phrasing. That is not a fault — it is sampling doing what it was asked to do.
  • The three calls at 0.0 will usually match, and that is the honest limit of the claim. Temperature 0 makes output much more repeatable, not guaranteed identical. Floating-point arithmetic on parallel hardware, provider-side batching, and silent model updates all move the result.
  • max_completion_tokens caps how long the reply can get. Parameter names for this vary between providers and between API versions, so check the reference for the one you are calling.
  • If you need a value to be stable across runs, do not rely on the model producing the same words. Ask for a constrained output and validate it, which is the subject of a later lesson.

The settings you will actually reach for, and what each one really does:

temperature
Flattens or sharpens the distribution before a token is picked. Low for extraction, classification and anything a program will parse. Higher only when variety is the point, such as offering three alternative phrasings to a human.
top_p
Restricts sampling to the smallest group of tokens whose probabilities add up to p. Another way to limit how adventurous the choice gets. Adjust this or temperature, not both at once, or you will not know which one caused a change.
max output tokens
A hard ceiling on the reply length. It protects you from a runaway response and from an unexpected bill, and it truncates mid-sentence when hit, so handle that case.
stop sequences
Strings that end generation as soon as they appear. Useful when you have asked for one item and the model tends to helpfully continue with a second.
model version
Pin a specific version where the provider allows it. A model identifier that quietly points at something newer will change your output without a line of your code changing.

Summary

  • An LLM produces a probability distribution over the next token, then repeats that step to build a reply
  • There is no document index inside the model, so nothing distinguishes a true statement from a plausible one
  • Temperature controls how adventurous token selection is; 0 is repeatable rather than guaranteed identical
  • "Understanding" is a loose metaphor and misleads as soon as you reason from it
  • Do not build on stable wording — constrain the output format and validate it

Practice

Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.

Try it yourself

Watch the variation

Send the same short question three times at temperature 1.0 and three times at temperature 0. Use something with a definite answer, such as summarising a paragraph you paste in.

Compare the six replies. Which parts changed — wording, ordering, length, or the actual claims being made?

Show solution

At the higher temperature you will usually see wording and ordering move while the substance stays roughly stable. Push further and substance starts moving too, because a lower-ranked token early on sends the rest of the response down a different path.

At temperature 0 the replies are normally the same. If two differ, you have seen first-hand why "deterministic" is the wrong word for this — repeatable is the honest one.

The design lesson: never build a feature on the assumption that generated wording is stable. Build on a validated structure, and let the wording vary.

Think about it

Where did that clause come from?

A colleague asks your assistant about parental leave. It replies with a confident paragraph citing "clause 7.3 of the staff handbook". No such clause exists.

Explain what happened in terms of next-token prediction, and say what change to the system would make this specific failure much less likely.

Show solution

Nothing was retrieved and nothing failed to be retrieved. Handbooks contain numbered clauses, so "clause 7.3" is a high-probability continuation of a sentence about a staff handbook. The model produced likely text, which is all it does.

The change that helps is supplying the actual handbook text in the prompt and instructing the model to answer only from what was supplied and to say when the material does not cover the question. That is retrieval, covered in module 3, and it moves the answer's source from the model's weights to a document you control.

Worth being precise about the limit: supplying context makes ungrounded answers much less likely, not impossible. You still need the instruction to refuse, citations a reader can check, and evaluation to know how often it slips.

Knowledge check

Nothing is recorded and there is no score. The explanation appears either way.

What does a large language model compute in a single forward pass?
You set temperature to 0 and the same prompt still occasionally produces different text. Why?

Saved in this browser only.