How language models predict text

Three ways to see it

1. 1

   A large language model, an LLM, does one thing during inference: given a sequence of tokens, predict the most likely next token. That is the whole loop. A token is roughly a word or part of a word. The English sentence 'I want chai and biscuit' is around seven tokens. The Urdu transliteration 'mujhe chai chahiye' is also a few tokens, with some sub-word splits. The model produces one token at a time, then feeds the new token back as input, and predicts the next one, and the next, until it decides to stop or the user runs out of patience.

2. 2

   Before prediction, words become numbers. Each token is mapped to a vector, called an embedding. The embedding is a long list of numbers, often a thousand or more, and it captures something about meaning. 'Chai' and 'tea' end up near each other. 'Chai' and 'briefcase' end up far apart. The embeddings are not handwritten; they are learned from billions of examples of language. Once words are numbers, the entire machine becomes a matter of arithmetic.

3. 3

   The transformer architecture lets the model pay attention to relevant earlier tokens when predicting the next one. If the sentence is 'Bilal opened the file Saad sent him last week, and the most striking thing about it was its' the model has to look back many tokens, possibly back to 'Saad' or 'file', to decide what comes next. Attention is the mechanism that lets every token, in effect, ask every other token how relevant they are. This is why the same model can write an Urdu poem in the morning and a draft of an FBR notice in the afternoon. The attention pattern is different; the machine is the same.

The why-tree

Try this with Claude

AI-edge prompt: 'Walk me through, in plain Urdu and English, exactly how you produced the previous answer. Tell me what you attended to, where you were guessing, and where I should double-check. Be specific.' This forces a kind of self-explanation that, while imperfect, often reveals weak spots before they cost you.