Advanced RAG Techniques: Beyond Basic Retrieval

Basic RAG is simple: chunk your documents, embed them, retrieve the top-k matches, stuff them into a prompt. It works for toy demos. It breaks in production.

Here are 5 advanced techniques that fix it.

The Problem with Naive RAG

Chunking loses context. You split a document at arbitrary token boundaries. The chunk that mentions “the acquisition” doesn’t include the company name three paragraphs earlier.

Retrieval misses relevant docs. Semantic search finds similar meaning but can miss exact terminology. A query about “reducing server costs” might retrieve docs about “cost optimization” but miss one titled “Infrastructure Spend Reduction” if the embedding model doesn’t map those terms closely.

Bad context triggers hallucination. You retrieve irrelevant chunks because the embedding model confused the query intent. The LLM uses that garbage as grounding and confidently hallucinates.

Basic RAG assumes one retrieval step with naive chunking is enough. It’s not.

Technique 1: Query Transformation

What it is: Rewrite the user’s query before you retrieve anything.

Why it works: Users ask questions in natural language. Your documents are written in formal language. The semantic gap causes retrieval to miss.

Three variants:

Query Rewriting: Expand abbreviations, add synonyms, clarify ambiguous terms. “Reduce EC2 costs” becomes “Reduce Amazon EC2 instance costs by optimizing compute spend.”

HyDE (Hypothetical Document Embeddings): Generate a fake answer to the question, then retrieve documents similar to that fake answer. Why? The embedding of an answer is closer to relevant documents than the embedding of a question.

Step-Back Prompting: Ask a broader version of the question first. Instead of “How do I configure SSL in Nginx 1.20?”, retrieve docs for “How does SSL work in Nginx?” first — the broader context helps answer the specific version.

When to use it: Complex queries, domain-specific jargon, or when your retrieval recall is low (you’re missing relevant docs).

Technique 2: Hierarchical Chunking

What it is: Store chunks at multiple granularities. Small chunks for precise retrieval. Large parent chunks for full context.

Why it works: Small chunks match queries precisely but lack context. Large chunks have context but drown the LLM in noise. Hierarchical chunking gives you both.

How it works:

• Split documents into small chunks (100-200 tokens) for embedding/retrieval
• Keep the large parent chunk (1000+ tokens) they came from
• At retrieval time: match on small chunks, return the parent chunks to the LLM

Example: User asks “What’s the refund policy for cancellations?” You retrieve a small chunk: “Cancellations within 24 hours are eligible for refunds.” But you return the full parent section that includes the steps to request a refund, the timeline, and the exceptions.

When to use it: When retrieved chunks feel incomplete or when the LLM asks for clarification that’s in surrounding text.

Technique 3: Hybrid Search

What it is: Combine semantic search (vector embeddings) with keyword search (BM25). Platforms like Elasticsearch or OpenSearch can run both.

Why it works: Semantic search understands meaning but misses exact matches. Keyword search catches exact terms but misses paraphrasing. Together, they cover both.

How it works:

• Run the query through both a vector index (embeddings) and a keyword index (BM25)
• Combine the results using Reciprocal Rank Fusion (RRF) or a weighted score
• Return the top-k from the merged ranking

Example: Query: “Einstein Analytics deprecation timeline.” Semantic search finds docs about “Tableau CRM sunsetting” (same concept, different words). Keyword search finds the one doc that uses the exact phrase “Einstein Analytics deprecation.” You need both.

When to use it: When you have domain-specific terminology, proper nouns, or acronyms that embedding models don’t handle well.

Technique 4: Re-ranking

What it is: Retrieve 50-100 candidates with fast retrieval, then re-rank them with a slower, more accurate model.

Why it works: Fast retrieval (cosine similarity on embeddings) optimizes for recall — cast a wide net. Re-ranking optimizes for precision — pick the best.

How it works:

• Initial retrieval: Use a bi-encoder (separate embeddings for query and docs). Fast. Returns 50-100 candidates.
• Re-ranking: Use a cross-encoder (query + doc together through a model). Slow. Scores each candidate for relevance.
• Return the top-5 after re-ranking.

Models: Cross-encoders like ms-marco-MiniLM or bge-reranker are trained specifically for this.

When to use it: When your top-k results include too much noise, or when precision matters more than latency (e.g., legal research, compliance checks).

Technique 5: Agentic RAG

What it is: Give an agent control over retrieval. The agent decides when to retrieve, what to retrieve, and whether to reformulate the query based on what it found.

Why it works: Static RAG retrieves once and hopes it’s enough. Agentic RAG treats retrieval as a tool — the agent can call it multiple times, refine the query, or skip it entirely if it already knows the answer.

Example workflow:

1. User asks: “Compare pricing for Heroku vs AWS for a Node.js app with 5GB DB.”
2. Agent retrieves Heroku pricing docs → not enough info on AWS
3. Agent reformulates: “AWS pricing for Node.js hosting” → retrieves AWS docs
4. Agent retrieves DB pricing separately for both platforms
5. Agent synthesizes a comparison table from multiple retrievals

This is multi-hop reasoning with retrieval as a tool, not a static pipeline.

When to use it: Complex questions that require multiple sources, comparisons, or when the query intent isn’t clear until you see initial results.

When to Use What

Stack them. Advanced RAG in production uses 2-3 of these together:

• Query rewriting + Hybrid Search + Re-ranking
• Hierarchical Chunking + Agentic RAG
• HyDE + Hybrid Search + Hierarchical Chunking

The Bottom Line

Basic RAG is “retrieve and pray.” Advanced RAG is an intelligent pipeline that:

• Transforms queries to match how documents are written
• Retrieves at multiple granularities and modalities
• Re-ranks for precision
• Lets agents decide when and how to retrieve

The shift from static retrieval to agentic retrieval is the same shift we saw from static websites to web apps. One retrieval step was never enough.

Further Reading:

• LangChain has implementations of most of these techniques
• The RAGatouille library implements ColBERT for late-interaction retrieval and re-ranking
• LlamaIndex supports hierarchical indexing and agentic retrieval patterns