Inside a language model: from text to the next token — Part 1

A language model can use a sentence you just typed without learning new weights from it. The prompt changes the calculation's input. During ordinary generation, the model's learned numbers stay fixed. To see how that works, give it the words “The kettle is on the” and follow one prediction.

We will follow a model in the GPT family, short for Generative Pre-trained Transformer. It generates text using already-learned parameters: the numbers its calculations use. We will trace one prediction through that model, then see how repeating the step produces more text. The second article explains how training learns those numbers.

Start with one prediction

After “The kettle is on the”, “stove” is a possible continuation. So is “table”. The model assigns scores to the possible next tokens, which we can turn into probabilities. Choosing a token from those probabilities is a separate step.

A token is a unit of text defined by a tokenizer. It might be a word, part of a word, or punctuation. To keep our example readable, we will pretend each word is one token and omit punctuation. This is a teaching simplification, not the output of a real tokenizer.

The immediate task is small: predict one more token using the text available so far. Repeating that task is how we get a longer answer.

Text becomes numbers

A tokenizer converts text into token IDs. These integers identify entries in a vocabulary. Their numerical order does not tell us anything about meaning: token 400 is not twice as meaningful as token 200.

The model uses each ID to look up an embedding, a vector, or ordered list, of learned numbers. We can think of the embedding table as one row per vocabulary entry. Looking up a token selects its row; it does not yet account for the surrounding sentence.

The IDs and vectors here are made up for illustration. A real tokenizer may split the words differently.

The model also needs information about order. In the GPT-style architecture we are following, it adds a learned position vector to each token’s embedding. Other architectures can represent position differently. Here, the result is a sequence of vectors carrying both token and position information.

Those vectors are the input to the transformer blocks. The text has become something we can calculate with.

Let each position use context

The embedding for “the” starts from the same vocabulary entry whenever that token occurs. But “the” after “The kettle is on” should contribute to a different prediction than “the” in another sentence. Its representation needs context.

Attention provides a way to combine information from different positions. For each position, the model calculates weights for the positions it can use, then combines information from them according to those weights. The weights depend on the input and learned parameters. For our final “the”, attention can bring in information from “kettle”, “is”, and “on” before the model predicts what comes next. That is a description of the available context, not a claim about measured weights in a real model. We do not hand-code a rule saying that a kettle must attend to a stove.

There is a restriction: each position can use itself and earlier positions, but cannot use later ones. This is causal attention.

Filled cells mean access is allowed. They do not show measured attention weights or how much a token matters.

For example, the third position, “is”, can use “The”, “kettle”, and itself. It cannot use the later “on” or “the”. The final position can use the whole prefix. This visibility rule also matters during training, which we will see in part two.

Multiple attention heads perform several learned combinations of context. Their results are combined before the model continues. We do not need to assign a fixed job such as “grammar head” or “meaning head” to each one.

Attention is only part of a transformer block, a group of operations repeated through the model. A feedforward network applies a learned transformation to the representation at each position. Attention has mixed information across positions; this network processes the resulting vectors one position at a time.

Residual connections add a block component’s input back to its output, so the next operation receives both the carried-forward representation and a learned change to it. Layer normalization centers and rescales values within a representation, with learned scale and offset. Stacking blocks lets later calculations work with representations already shaped by earlier ones.

Turn the result into a next token

At the end of the model, an output layer maps the final position’s representation to one score for every token in the vocabulary. Those scores are often called logits. A function called softmax converts them into nonnegative probabilities that sum to one.

The generation program then chooses a token. It could choose the highest-probability token, or sample from the distribution. Sampling gives lower-probability choices a chance too, so the same prompt can produce different continuations.

A probability here expresses the model’s prediction of a continuation. It is not a probability that a statement is factually true. A fluent continuation can still be wrong.

Repeat the step to get more text

Suppose the selected token is “stove”. The program appends it to the context, then asks for the next token using the extended sequence. It keeps doing this until a stopping condition, such as an end token or a length limit, is reached.

This is the conceptual data flow. Implementations can reuse intermediate calculations instead of recomputing everything on each step.

When “stove” is appended, the next prediction has a different input even though the model's parameters are unchanged. This is how a conversation can influence the answer without retraining the model. If we remove that context on the next call, we cannot assume the model will retain what was said.

The remaining question is where the parameters came from. A model with random numbers can perform these calculations too, but its predictions will not yet be useful. Part two follows the training process that changes those numbers.

Sources & references

This two-part introduction draws on Sebastian Raschka’s Developing an LLM: Building, Training, Finetuning. Credit for the teaching material belongs to him; the examples and diagrams here are my own explanations.

See also Raschka’s original slides.