What are tokens, and why your bill depends on them

Three ways to see it

1. 1

   A token is not a word, not a letter, and not a syllable. It is a unit of text that the model has decided to treat as one indivisible chunk during training. The decision was made by an algorithm called Byte Pair Encoding, or BPE, which scans billions of words and finds the most common chunks. Common English words like the, of, and is each become one token. Less common words like reconciliation are split into pieces, like recon, cilia, tion. Rare names and Urdu text get split even more aggressively because the model saw them fewer times during training.

2. 2

   Way one to picture tokens: think of a shop in Anarkali bazaar. The shopkeeper does not charge per item brought in. He charges per packet. A common item like sugar fits in one packet. A rare item like saffron, because it comes wrapped in tissue and a small box, takes three packets. You pay by packets, not contents. That is exactly how an LLM bills you. Hello world is two tokens. Assalam-o-alaikum dunya is roughly seven. The content is similar; the packaging is different.

3. 3

   Way two: think of a postal sorting machine at Pakistan Post. To move a letter, the machine does not read the words. It looks at the envelope, scans a barcode, slots it into a bin. The barcode is a token id. Inside the model, every token is a number from a fixed vocabulary, usually around fifty thousand to one lakh entries. The model never sees your letters as you wrote them. It sees a sequence of numbers, and it predicts the next number based on the pattern of numbers so far. The word you typed is a costume; the token id is the citizen number behind it.

The why-tree

Try this with Claude

AI-edge prompt: 'I run a five-person research team at a Pakistani think tank. We process around 200 documents a month, mostly Urdu policy papers averaging 8000 words each. Estimate our monthly token consumption if we use Claude Sonnet for summarisation. Give the answer in input tokens, output tokens, and USD at current Anthropic pricing.' Compare what Claude tells you with the calculator you build in lesson two.