Contextual Retrieval addresses a common RAG failure: a system can return a chunk that looks relevant while still missing the information the user needs. The usual culprit is lost context: a sentence survives chunking, but the document, product, customer, date, or definition that makes the sentence meaningful does not.

Anthropic’s answer is Contextual Retrieval. In its engineering write-up, the company reports that the approach reduced failed retrievals by 49% in its evaluation, and by 67% when combined with reranking. Those are reported evaluation results, not a promise that every RAG system will improve by the same amount. The useful idea is simpler: give each chunk enough context to stand on its own before asking a retrieval system to find it.

Key takeaways

• Contextual Retrieval adds a short explanation of where a chunk sits in its source document and what it refers to.
• The added context is used for both embedding-based retrieval and BM25-based retrieval.
• Anthropic’s 67% figure applies when Contextual Retrieval is combined with reranking in its reported evaluation.
• A small knowledge base may be better placed directly in the model prompt than put behind a RAG pipeline.

Why ordinary RAG can retrieve the wrong chunk

Retrieval-augmented generation, or RAG, retrieves relevant material from a knowledge base and appends it to the user’s prompt. To make that practical, a system normally splits source material into smaller chunks, indexes them, retrieves the most promising ones for a question, and sends those chunks to the model.

Chunking is necessary, but it can turn a clear passage into an ambiguous fragment. Consider a chunk that says, Revenue increased by 12%. In isolation, it does not identify the company, reporting period, product line, geography, or even the definition of revenue. A query about a specific business unit may retrieve it because the wording is a close match, even though it answers a different question.

Embedding retrieval is designed to find conceptual similarity. BM25 is a keyword-oriented retrieval method that scores documents according to the query terms they contain. Both can be valuable, but neither can recover details that were removed before the chunk was indexed. Anthropic identifies this loss of context during encoding as a reason traditional RAG can fail to retrieve the relevant information.

What Contextual Retrieval changes before indexing

Contextual Retrieval enriches each chunk with a concise description of its place in the larger source. Instead of indexing the isolated revenue sentence, a system could prepare an enriched version such as:

This chunk is from Acme’s Q2 2025 investor update. It describes revenue for the UK subscription product line.

Revenue increased by 12%.

The example is illustrative, but it shows the key move. The original text remains available, while the added prefix supplies the references that would otherwise be missing. A query about Acme’s UK subscription business now has more signals to match against.

Anthropic describes two parts of the method: Contextual Embeddings and Contextual BM25. The contextualized chunk is embedded for semantic retrieval and also indexed for BM25. This gives both retrieval paths access to the same disambiguating information rather than relying on the bare fragment alone.

Why use embeddings, BM25, and reranking together?

Semantic and keyword retrieval do not make the same mistakes. A semantic search can surface wording with a similar meaning even when the terms differ. A keyword search can be especially useful when a user supplies an exact identifier, product name, or phrase. Using both approaches creates a broader set of candidate chunks for the answer stage.

Reranking is a further selection step. Rather than accepting the initial ordering from retrieval, the system evaluates the candidate set again and promotes the chunks that best match the query. Anthropic’s reported result is important to state precisely: Contextual Retrieval alone reduced failed retrievals by 49% in its evaluation; the 67% reduction was reported when it was combined with reranking.

That distinction matters in planning. Contextualizing chunks may improve the material entering retrieval, while reranking may improve the final choice among the retrieved candidates. Neither replaces a representative evaluation of the documents and questions an organisation actually cares about.

Make the implementation decision with an evaluation

Contextual Retrieval adds a preprocessing step. Someone or something must generate the contextual descriptions, the index must be rebuilt, and the team needs to decide how much extra text is useful without becoming repetitive. These are practical costs in latency, model usage, and operational complexity.

The strongest first test is an A/B retrieval evaluation. Collect real or carefully designed questions, define what a successful retrieval looks like, and compare the baseline against contextualized chunks. Review missed results as well as successful ones. An answer can sound fluent even when the retrieval step selected the wrong evidence, so retrieval quality deserves its own measurement.

This approach is most promising when the corpus contains many similarly worded fragments, long documents, repeated terminology, or facts whose meaning depends on surrounding metadata. It is less compelling when a team has not observed a retrieval problem or cannot afford the additional indexing workflow.

When RAG may be unnecessary

More retrieval machinery is not always the right answer. Anthropic notes that, for a knowledge base smaller than 200,000 tokens, it may be simpler to include the entire knowledge base in the prompt rather than use RAG or a similar retrieval system. That option removes the chunk-selection problem, although it still needs to fit the product’s cost, latency, and context requirements.

The decision rule is practical: start with the simplest approach that reliably gives the model the needed evidence. Add retrieval when the knowledge base or task makes direct inclusion impractical; add contextualization when chunk-level retrieval is demonstrably losing important meaning.

The verdict

Contextual Retrieval is a useful technique for teams with large or context-heavy knowledge bases where relevant chunks are being missed. Its contribution is not a mysterious new search algorithm; it is the disciplined step of preserving a chunk’s meaning before embedding and BM25 indexing.

It is not a universal 67% improvement. Treat Anthropic’s figure as a result from Anthropic’s evaluation, then validate the method against your own documents and queries. The safest first action is to audit a sample of failed searches and run a small A/B test with contextualized chunks and the existing baseline.

Categories: Artificial Intelligence, Software Architecture

Tags: Anthropic, LLM, Artificial Intelligence, Claude

The post Contextual Retrieval: Anthropic’s Approach to Reducing RAG Retrieval Failures appeared first on Alpesh Kumar.