Retrieval-Augmented Generation (RAG), explained

A language model only knows what was in its training data — frozen at some cutoff date, and never containing your product docs, your support tickets, or last week's meeting notes. Fine-tuning can teach it new behavior, but it's a slow, expensive way to teach it new facts. Retrieval-Augmented Generation (RAG) solves this differently: instead of baking knowledge into the model's weights, you fetch the relevant facts at request time and hand them to the model as context.

The result is a model that answers from your data, cites where the answer came from, and stays current the moment you update the underlying documents — no retraining required.

The pipeline, end to end

Every RAG system, however sophisticated, is a variation on the same five steps: embed the query, retrieve candidate chunks, optionally re-rank them, stuff the survivors into a prompt, and generate.

Chunking: the decision that quietly controls everything

Before any of this works, your documents need to be split into chunks small enough to embed meaningfully and retrieve precisely. Chunk too large, and you dilute a chunk's embedding with irrelevant text — a 2,000-word page about pricing and onboarding won't score well against a query about either. Chunk too small, and you lose the surrounding context a chunk needs to make sense on its own.

• Fixed-size chunking — split every N tokens with a small overlap. Simple, fast, and a reasonable default.
• Recursive/structure-aware chunking — split along headings, paragraphs, or code blocks first, and only fall back to fixed-size splits when a section is still too long.
• Semantic chunking — split where the topic actually shifts, using embedding similarity between adjacent sentences. More expensive to compute, often worth it for long, dense documents.

Retrieval: dense, sparse, or both

Dense retrieval embeds the query and every chunk into the same vector space and ranks by cosine similarity — great at matching meaning ("how do I cancel?" ↔ a chunk about subscription refunds) even when the wording is completely different. Sparse retrieval (BM25 and friends) ranks by exact keyword overlap — great at matching a specific error code, SKU, or acronym that a dense embedding might blur away.

Re-ranking: cheap recall, expensive precision

Vector search is fast but only roughly right — it's tuned to quickly narrow millions of chunks down to a few dozen candidates, not to perfectly order them. A cross-encoder re-ranker looks at the query and each candidate together (rather than as independent vectors) and produces a much more reliable relevance score. It's too slow to run over the whole corpus, but perfect for re-scoring the 20–50 candidates retrieval already found.

Putting it together

def answer(query: str) -> str:
    query_vector = embed(query)

    # 1. Retrieve the most relevant chunks
    candidates = vector_store.search(query_vector, top_k=20)

    # 2. Re-rank for precision (cheap recall, expensive precision)
    context = reranker.top_n(query, candidates, n=4)

    # 3. Ground the prompt in retrieved context
    prompt = f"""Answer using ONLY the context below.
If the answer isn't there, say you don't know.

Context:
{format(context)}

Question: {query}"""

    return llm.generate(prompt)

Notice the prompt explicitly tells the model to answer only from the provided context, and to admit when it doesn't know. That one instruction is doing a lot of work — without it, the model will happily blend retrieved facts with things it half-remembers from training, which is exactly the hallucination RAG was supposed to prevent.

Where RAG tends to break

The failure modes are rarely about the LLM itself — they're almost always upstream, in retrieval. A wrong or missing chunk guarantees a wrong or unhelpful answer no matter how good the model is. That's exactly why teams building RAG systems invest so heavily in evaluation — measuring retrieval quality and answer faithfulness separately, which is the subject of the evals article.