RAG vs. long context: choose by failure mode, not by benchmark
Retrieval and million-token windows both promise to put your documents in front of the model. They fail differently, and the failure you can tolerate should decide.
Two years ago the choice was forced: context windows were small, so anything beyond a few pages had to be retrieved. Windows have since grown by orders of magnitude, and a reasonable person now asks whether retrieval-augmented generation (RAG) is worth the machinery. The honest answer is that the two approaches fail in different ways, and the decision should be made by asking which failure you can live with.
How each one fails
Long context fails quietly. Give a model a very long document and it will usually answer well, but attention is not uniform. Research on positional effects showed that information in the middle of a long context is used less reliably than information at the start or end, and the effect persists in newer models even as it shrinks. The model does not tell you it skimmed. It simply answers with slightly less of the document than you gave it.
RAG fails loudly, and earlier. If the retriever does not surface the right passage, the model cannot use it. The failure is visible in the retrieved set, which means it can be measured and fixed, but it also means a mediocre retriever caps the whole system regardless of how good the model is.
Long context hides its misses inside a fluent answer. Retrieval shows you its misses before the model speaks.
The other axes
| Axis | Long context | RAG |
|---|---|---|
| Cost per query | Scales with document size; you pay for every token every time | Roughly constant; only the retrieved passages are billed |
| Latency | Grows with input length | Retrieval adds a step, but the model call stays small |
| Freshness | Requires re-sending the updated document | Update the index; the next query sees it |
| Corpus size | Bounded by the window | Effectively unbounded |
| Debuggability | Hard: the model saw everything | Easier: inspect what was retrieved |
| Setup effort | Minimal | An index, a chunking strategy, and an embedding model to maintain |
A decision rule
Use long context when the corpus is small enough to fit comfortably, changes rarely, and you value simplicity over per-query cost. A contract, a codebase module, a single long report.
Use retrieval when the corpus is larger than a window, changes often, or per-query cost matters. A knowledge base, a ticket history, anything multi-tenant.
Use both when the stakes are high: retrieve widely, then let a long window read everything retrieved instead of only the top few passages. This is the configuration most serious production systems have converged on, and it turns the two failure modes into checks on each other.
What to measure
Whichever you choose, build an evaluation set of questions with known answers and known source passages. Score retrieval recall separately from answer accuracy. If accuracy is low but recall is high, the model is the problem. If recall is low, no model will save you. Our comparisons desk has more on what an evaluation setup should contain.
Sources
Questions readers ask
Is RAG obsolete now that context windows are huge?
No. Long context removes the need for retrieval on small corpora, but retrieval still wins on cost, latency and freshness once the corpus is larger than a window or changes often.
Can I combine the two?
Yes, and most production systems do. Retrieve a generous set of candidates, then give the model a long window so it can read all of them rather than the top three.