How a language model learns: pretraining and fine-tuning — Part 2
A language model can get a training target from the same text it is reading. Move the text over by one token: each input position now has a next token to predict. No one has to write a separate answer for every prefix. The catch is that the model must not be allowed to peek at that answer. The text already contains a training target Consider the sentence “The kettle is on the stove.” As before, we will pretend each word is one token and leave out punctuation. A real tokenizer may split it differently. After “The”, the next token in this sentence is “kettle”. After “The kettle”, it is “is”. Each position gives us a prediction to compare with the token that actually follows. The input and target sequences are offset by one position. Word-sized tokens are a simplification for this example. This is why we can train a language model from ordinary text without manually writing an answer for every prefix. The text supplies the target. However, later tokens are present in the training s...