RAG: Teaching AI Your Data

The problem RAG solves

A model knows what was in its training data, frozen at a cutoff. It does not know your company handbook, last week’s incident report, or anything behind your login.

There are three ways to fix that, and picking the right one matters:

The common error is reaching for fine-tuning to add knowledge. Fine-tuning changes how a model writes, not reliably what it knows. For facts, RAG is almost always the right instrument.

And now that context windows reach a million tokens, a fair question: why not paste everything? Because attention is uneven, latency scales with input, and you pay for every token on every request. Selecting the right 5,000 tokens is faster, cheaper and more accurate than supplying 500,000.

Two separate pipelines

Almost all RAG confusion comes from mixing these up. They run at different times.

Indexing — offline, once per document

1. Load — pull text out of PDFs, HTML, docs. Preserve headings and tables; structure carries meaning.
2. Chunk — split into passages of a few hundred tokens.
3. Embed — convert each chunk to a vector.
4. Store — keep vectors plus the original text and metadata.

Querying — online, on every question

1. Embed the question using the same model.
2. Search for the nearest chunk vectors.
3. Assemble a prompt containing the question and the retrieved text.
4. Generate an answer grounded in that text.

What an embedding actually is

An embedding is a list of numbers — commonly several hundred to a few thousand — positioning a passage in space so that similar meanings land near each other.

“How do I get my money back?” and “What is the refund process?” share almost no words. Their embeddings sit close together, because the model that produced them learned meaning rather than spelling. That is the whole trick, and it is why RAG beats keyword search on real questions.

Two operational rules:

• The same model must embed both documents and queries. Different models produce incompatible coordinate systems. Mixing them yields nonsense, silently.
• Changing the embedding model means re-indexing everything. Budget for it.

Chunking is where quality is won or lost

One vector represents an entire chunk, so chunk boundaries decide what can be found.

• Too small (a sentence) — “It expires after 30 days” is unfindable, because the chunk no longer says what it is.
• Too large (a whole chapter) — several topics average into one vague vector that matches everything weakly and nothing strongly.

A sane default: split on structure first — headings, then paragraphs — targeting roughly 300–500 tokens with a small overlap so a sentence spanning a boundary is not lost.

Always attach metadata: source document, section heading, date. It enables filtering (“only current policies”) and, crucially, citation.

The four failure modes

This is the part most introductions omit, and it is what separates a demo from something usable.

1. It answers when it should refuse. Vector search returns nearest neighbours. If nothing relevant exists, it returns the nearest irrelevant thing, and the model — being obliging — writes an answer from it. Fix: enforce a similarity floor, and instruct the model explicitly to say it does not know.

2. Retrieval is right, the answer is wrong. The correct passage was supplied and the model ignored it or embellished. This is a generation problem, not a retrieval one. Fix: strengthen the prompt — answer only from the context, cite which chunk each claim came from.

3. Contradictory chunks. The 2023 policy and the 2026 policy both retrieve. The model picks one, or blends them. Fix: metadata filtering and recency weighting.

4. The question needs synthesis across many documents. “How has our refund policy changed over five years?” needs twenty documents compared, and top-k retrieval returns five. Standard RAG cannot do this well. Fix: a different architecture, or acknowledge the limit.

Diagnosing, in the right order

When an answer is bad, always check retrieval first. Log the chunks that were retrieved and read them.

• Wrong chunks retrieved → chunking, the embedding model, or the query.
• Right chunks, wrong answer → the generation prompt.

Skipping this step and tuning the prompt when the retriever is broken is the single most common way teams waste weeks. The expert tier covers how to measure both halves properly.

Try this: Take ten questions real users would ask of a document set you know well. For each, write down the passage that should answer it. That list is now your evaluation set — and you have done the most valuable part of building a RAG system before writing any code.