Context windows and why size is not free

Three ways to see it

1. 1

   A context window is the maximum number of tokens a model can hold in its head at once. Claude Sonnet 4.5 carries up to two lakh tokens (about 150,000 English words). Some models go further with experimental 1-million-token windows. GPT-4o sits at 128,000. Gemini Pro stretches to a million. These limits are advertised loudly because they sound impressive. They are also where many of the most expensive mistakes in enterprise AI happen, because users confuse 'the model can hold it' with 'the model handles it for free'. The window is the maximum bag size. You still pay for everything you put in the bag.

2. 2

   Way one to feel the cost: attention is quadratic. To predict the next token, the model compares every token in the window with every other token. If your prompt has N tokens, the model does roughly N-squared comparisons. Double the prompt and you quadruple the work. The latest models use clever tricks to soften the curve in practice, but the principle stands: a 100,000-token prompt is not twice as expensive as a 50,000-token prompt. It is closer to four times. You feel this in two places at once, the bill and the latency. A long prompt also takes longer to start answering.

3. 3

   Way two: the lost-in-the-middle problem. Researchers have shown that LLMs pay the most attention to the beginning and the end of a long prompt and lose track of facts buried in the middle. If you paste a 500-page tax ordinance and ask about Section 122 (which lives in the middle), the model is genuinely less reliable than if you had given it only the relevant pages. You pay more and you get less accuracy. This is the cruellest part of the trap, because the user assumes more context means better answers. Sometimes it does. Often it does not.

The why-tree

Try this with Claude

AI-edge prompt: 'I have a 500-page FBR tax ordinance and 200 monthly questions from auditors. Compare three architectures: (a) paste full document each time, (b) RAG with top-5 chunks, (c) fine-tuned smaller model. Estimate monthly cost in PKR and answer-quality risk for each. Recommend one and explain.' This prompt will surface tradeoffs your procurement team has not yet thought through.