Advanced RAG: Re-ranking, Hybrid Search, HyDE, Multi-Query, Self-RAG & CRAG

The standard RAG pipeline — embed the query, retrieve candidate chunks, optionally re-rank them, and generate — is well covered ground, and it's a strong default. But plenty of production systems ship that pipeline and still get retrieval wrong often enough that users notice. This article picks up where basic RAG leaves off: a set of techniques that change what gets embedded, how many retrieval passes run, and whether the pipeline trusts its own results before generating an answer.

Re-ranking, one level deeper: why it's a separate stage

It's tempting to ask why retrieval doesn't just be precise in the first place, skipping the second re-ranking pass entirely. The answer is architectural. A bi-encoder — the model that powers your vector search — embeds the query and every document independently, ahead of time, into the same vector space. That's what makes it fast: you can pre-compute millions of document embeddings once and compare a new query against all of them with a single approximate-nearest-neighbor lookup. The cost of that speed is that the model never sees the query and a specific document together — it can't reason about their interaction, only about how close two independently-computed points happen to land.

A cross-encoder re-ranker does see them together — it takes the query and one candidate document as a single input and outputs a relevance score conditioned on both. That joint attention makes it dramatically more accurate, but it also means it can't be pre-computed: every query/document pair has to be scored fresh, which is far too slow to run over a corpus of millions. So the standard architecture uses each model for what it's good at — a cheap bi-encoder casts a wide net over the whole corpus, and an expensive cross-encoder re-scores only the 20–50 survivors. Trying to make the first-stage retriever itself perfectly precise would mean giving up the pre-computation that makes it fast at scale in the first place.

Hybrid search, one level deeper: fusing two ranked lists

Dense retrieval and sparse (BM25-style) retrieval don't just differ in what they're good at — they produce scores on completely different, incomparable scales. A cosine similarity of 0.82 and a BM25 score of 14.3 can't be averaged meaningfully. That's the actual problem hybrid search has to solve: not just "use both signals," but fuse two rankings that don't share a scale.

Reciprocal Rank Fusion (RRF) is the standard answer. Instead of combining raw scores, it combines rank positions. For each document, it computes 1 / (k + rank) in the dense list and the same in the sparse list, then sums the two — where k is a small constant (commonly 60) that softens the effect of rank-1 dominating everything. A document that ranks highly in either list gets a strong combined score; a document that ranks moderately well in both often beats one that's ranked first in only one of them. It requires no score calibration between the two retrieval methods at all, which is exactly why it's become the default fusion technique for hybrid search.

HyDE: embed the answer, not the question

Dense retrieval has a subtle structural weakness: queries and documents are usually written in different registers. "How do I reset a forgotten password?" is a question. The document that answers it is a statement: "To reset your password, navigate to Account Settings and select Forgot Password." Even though they're semantically related, a query embedding and a document embedding are, in a real sense, two different kinds of text — and that phrasing mismatch shows up as lost recall.

HyDE (Hypothetical Document Embeddings) closes that gap by never embedding the raw query at all. Instead, it asks the LLM to first write a hypothetical answer — a made-up passage that would answer the query, even though the model has no guarantee it's factually correct — and embeds that instead. Because the hypothetical answer is phrased like a statement, in roughly the same register as the real documents in your corpus, its embedding lands much closer to the actual answer's embedding than the original question's ever would.

def hyde_retrieve(query: str, top_k: int = 8):
    # 1. Ask the LLM to imagine an answer, not to search for one.
    hypothetical_doc = llm.generate(f"""Write a short passage that
would answer the question below. Do not mention that you're
unsure — write it as if it were a confident excerpt from a
reference document.

Question: {query}""")

    # 2. Embed the *hypothetical answer*, not the raw query.
    hyde_vector = embed(hypothetical_doc)

    # 3. Retrieve real documents that live near that point in
    #    embedding space — answers cluster near other answers.
    candidates = vector_store.search(hyde_vector, top_k=top_k)

    return candidates

The hypothetical document itself is never shown to the user and never used as context for the final answer — it exists purely as a better-shaped probe into the vector space. It also doesn't need to be accurate; it just needs to be stylistically closer to a real answer than the raw question is, which is a much lower bar to clear.

Multi-query retrieval: cover more phrasings, not just one

A single query is a single point in phrasing space, and the documents that answer it may use entirely different wording than the one you happened to type. "What's our refund policy?" and "Can I get my money back after 30 days?" are the same underlying question, but they'll pull different neighborhoods of a vector index.

Multi-query retrieval asks the LLM to generate several rephrased or complementary variants of the original query — different angles, different vocabulary, sometimes narrower sub-questions that decompose a broad query — and runs retrieval independently for each variant. The result sets are then merged and deduplicated (RRF works well here too) into one combined candidate pool before re-ranking. Where HyDE tries to close the query/document phrasing gap with one better-aimed embedding, multi-query attacks the same gap from the other direction: instead of one good shot, it takes several shots that collectively cover more of the space a single phrasing would miss.

Self-RAG: retrieval as a decision, not a fixed step

Every technique so far still assumes retrieval always runs. But not every query needs it — "summarize what you just said" or "what's 15% of 80?" gains nothing from a trip to the vector store, and forcing one just adds latency and a chance of irrelevant context leaking into the prompt.

Self-RAG reframes retrieval and verification as decisions the model makes rather than stages a pipeline always executes. The model is trained or prompted to first judge whether retrieval would actually help this query, and only fetches context when it decides it would. After retrieval, it takes a second reflective pass: does the retrieved content actually support the answer it's about to give, or is it tangential, contradictory, or simply insufficient? Both judgments — whether to retrieve, and whether what was retrieved is trustworthy — become explicit reasoning steps the model produces alongside its answer, rather than fixed plumbing that runs unconditionally on every request.

Corrective RAG (CRAG): grade before you generate

CRAG tackles a narrower but very common failure: retrieval running successfully and returning documents that are simply not good enough — off-topic, outdated, or too thin to actually answer the query — with the pipeline generating from them anyway because nothing ever checked. CRAG inserts an explicit grading step right after retrieval and before generation: a lightweight evaluator (often a smaller model or a focused prompt) scores the retrieved set as sufficient, ambiguous, or poor.

A sufficient set proceeds to generation as normal. A poor set triggers a fallback instead of silently degraded output — that might mean a broader web search, an automatically reformulated query fed back into retrieval, or an explicit "I don't have enough information to answer this confidently" response surfaced to the user. The point isn't that CRAG retrieves better documents; it's that it refuses to generate confidently from bad ones.

Composing these techniques — and knowing when to bother

None of these techniques are mutually exclusive. A production pipeline might run hybrid search with RRF fusion, embed a HyDE passage instead of the raw query, expand into a handful of multi-query variants, merge everything, and re-rank the combined pool with a cross-encoder before generating — each stage adding a bit more precision on top of the last. That's a reasonable architecture for a hard retrieval problem, and also a genuinely slow and expensive one: every added stage is another LLM call, another retrieval pass, or another model invocation on the request's critical path.

The right approach is incremental, not aspirational. Ship basic RAG, measure retrieval quality directly — not just whether answers look plausible — and only reach for the specific technique that addresses the specific failure mode you've actually observed. If recall is the problem, multi-query or HyDE earns its latency. If bad context is slipping through untouched, CRAG's grading step is the one worth adding. Layering all of them in before you've confirmed retrieval is even your bottleneck just adds cost without a clear return.