Building RAG Pipelines

The discipline in one sentence

Most RAG systems fail for a reason nobody measured, because the team optimised the half that was already working.

So the organising principle of this lesson is: retrieval and generation are two systems with two failure modes and two sets of metrics. Instrument them separately or you will tune blind.

Measure first

Before any tuning, build an evaluation set: 50–100 questions with the passage that should answer each one, drawn from real usage where possible.

That labelled passage is what makes everything else measurable.

Retrieval metrics

Recall@k is the one to fix first. If the right passage is not in the top k, nothing downstream can recover. Everything else is secondary.

Generation metrics

Assuming retrieval succeeded:

• Faithfulness — is every claim supported by the retrieved context? Catches hallucination over correct evidence.
• Answer relevance — does it address the question actually asked?
• Correctness — does it match ground truth? Needs labelled answers.
• Refusal accuracy — does it decline when the corpus genuinely lacks the answer? Routinely omitted, and it is where trust is won or lost.

The diagnostic that resolves most confusion:

Right passage retrieved?
├── No  → retrieval problem: chunking, embeddings, query, k
└── Yes → generation problem: prompt, model, context ordering

Ingestion

Quality is decided here, before any clever retrieval.

Parsing. Preserve structure — headings, tables, lists carry meaning that flat text extraction destroys. A table flattened into a wall of numbers is unretrievable and, worse, misleading when it does get retrieved. For PDFs specifically, budget real time: multi-column layouts, headers and footnotes are genuinely hard, and a bad parse is an invisible ceiling on the entire pipeline.

Chunking. One vector represents one chunk, so boundaries determine what is findable.

Sensible starting point: recursive splitting at 300–500 tokens with 10–15% overlap. Then measure recall@k and adjust — this is a parameter to tune, not a setting to get right first time.

The parent-child pattern deserves particular attention. Small chunks embed precisely; large chunks give the model enough context to answer. Doing both — retrieve on the small chunk, pass the parent to the model — resolves a tension that otherwise forces a compromise.

Metadata. Attach source, section, date, permissions, version to every chunk. This enables filtered retrieval, recency weighting, citation, and — critically — access control. A RAG system that retrieves across permission boundaries is a data breach, and metadata filtering is how you prevent it. Filter before the vector search, not after.

Embeddings

Selection criteria in order of practical importance: retrieval quality on your data, dimensionality (memory and speed), sequence limit, and whether it can be self-hosted for privacy.

Two rules that cause real incidents:

• The same model must embed queries and documents. Different models produce incompatible spaces. Mixing them fails silently and badly.
• Changing the embedding model means a full re-index. Plan for it: version your index, and be able to run two in parallel during migration.

Do not choose on a public leaderboard alone. Benchmarks measure general retrieval; your corpus may be legal contracts, or code, or clinical notes. Take twenty labelled query-passage pairs from your own data and compare candidates on those. It takes an hour and regularly overturns the leaderboard ordering.

Retrieval

Vector search alone

Query embedding, nearest neighbours, done. Good on paraphrase and conceptual similarity. Weak on exact tokens — error codes, part numbers, surnames, API names. Embeddings blur precisely the things you needed matched exactly.

Hybrid: vector plus lexical

Run BM25 alongside vector search and fuse the rankings. Reciprocal Rank Fusion is the standard method because it needs no score normalisation between two incomparable scales:

$\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}$

where $r(d)$ is the document’s rank in result set $r$, and $k \approx 60$ damps the influence of top positions.

Hybrid is close to a free win. It reliably improves recall on real corpora because lexical and semantic matching fail on different queries.

Reranking

The strongest single improvement available, and the reason is architectural.

Vector search uses a bi-encoder: query and document are embedded independently, so the score is a distance between two fixed vectors. Fast enough for millions of documents, but the two never interact.

A cross-encoder processes query and document together, attending across both. Far more accurate, and far too slow to run over a corpus.

So: use the bi-encoder to get 50–100 candidates, then the cross-encoder to rerank down to 5–10.

query → hybrid retrieval (top 50) → cross-encoder rerank → top 8 → generate

Typical cost is 100–300ms. On most pipelines it is the highest-value latency you will spend.

On k

Raising k does not monotonically improve answers. Recall rises, precision falls, and extra marginal passages give the model more chances to ground its answer in the wrong place. This is why reranking a wide candidate set beats simply widening k — you get the recall of a wide net with the precision of a narrow one.

Generation

Context assembly matters as much as the prompt. Order by relevance with the strongest passages at the beginning and end, since recall dips in the middle of a long context. Label every chunk with an identifier so citation is possible. And keep a [1]-style marker in the text so a claim can be traced to a source.

Answer using ONLY the numbered sources below.
Cite the source number after each claim, like [2].
If the sources do not contain the answer, reply exactly:
"I don't have information about that in the available documents."

[1] (handbook.pdf, "Parental Leave", 2026-01-14)
{chunk}

[2] (policy.pdf, "Entitlements", 2025-11-02)
{chunk}

Question: {query}

Three deliberate choices there. Only is stated positively rather than as a prohibition. Citation is required inline, which makes verification cheap and measurably reduces unsupported claims. And the refusal is given exact wording, so it is detectable in logs and testable in evals — a refusal you cannot detect is a refusal you cannot measure.

Failure modes and their fixes

Where standard RAG stops

Top-k retrieval answers local questions — the answer sits in one or a few passages. It structurally cannot answer global ones: “how has this policy evolved?”, “what themes recur across these 400 reports?”

No amount of reranking fixes this, because the answer is not in any k passages. It requires a different shape: hierarchical summarisation over the corpus, or a graph index built by extracting entities and relationships and querying over community summaries. That is real additional machinery, and worth building only once you have confirmed your users actually ask global questions.

Knowing the boundary is the senior skill. Recognising “this is a global question and my architecture is local” saves months.

Build order

1. Evaluation set: 50-100 questions with expected passages
2. Baseline: recursive chunking + vector search. Record recall@k
3. Add hybrid retrieval. Re-measure
4. Add reranking. Re-measure
5. Tune chunk size against recall@k
6. Only now tune the generation prompt
7. Add refusal handling and measure refusal accuracy

Steps 2 through 5 are all retrieval. The generation prompt comes sixth deliberately — it is what teams tune first, and it is what matters least while recall is poor.

Try this: Log the retrieved chunk IDs for a hundred real queries and hand-check twenty against what should have been retrieved. That recall@k number is the ceiling on your whole system, and most teams find it far lower than they assumed.