RAG vs full context — when to chunk and when not

Three ways to see it

1. 1

   RAG, retrieval augmented generation, is the architecture where you split documents into small chunks, convert each chunk into a vector (an embedding) that represents its meaning, store these vectors in a search index, and at query time fetch only the chunks most relevant to the question. The model sees a slim, targeted context instead of the whole corpus. Full-context is the opposite: paste the whole document each time and let the model find the answer itself. Both are honest engineering choices. The question is which one fits your situation.

2. 2

   The decision rests on three numbers. First, corpus size: the total tokens across all documents you might ever need. Under 100,000 tokens, RAG is overkill; the whole thing fits in a window. Between 100,000 and 2 million, full-context per-document is workable; RAG is optional. Above 2 million tokens of corpus, RAG becomes essential because no single window holds it. HBL with 12,000 contracts is well into RAG territory. A startup with one product manual is not. Second, query volume: a single query a day means RAG infrastructure does not pay back; 10,000 queries a day means the savings dwarf the build cost. Third, accuracy requirement: a hospital triage chatbot cannot tolerate the model missing a chunk; an internal HR FAQ can.

3. 3

   Where RAG shines: large stable corpora, frequent queries, structured questions that map cleanly to chunks (one section answers one question), need for citations (you can show which chunk you retrieved), need for permissions (you can apply ACLs at chunk level). HBL contract review fits all five. Where full-context shines: small or single-document scope, exploratory questions that touch many parts at once ('summarise the overall tone'), one-off analysis where building infrastructure is not justified, cases where the model needs to see global structure (an entire balance sheet, an entire policy). FBR's quarterly press-release drafting from a single ministerial briefing note is full-context territory.

The why-tree

Try this with Claude

AI-edge prompt: 'I run customer support at PIA. We have 5000 internal documents (cancellation policies, fare rules, baggage rules), 800 daily customer queries, and we cannot afford to hallucinate fare amounts. Recommend RAG, full-context, or hybrid, justify with token math, and outline a 6-week build plan. Estimate monthly PKR cost at completion.' Compare the model's recommendation against what your in-house engineers would propose.