How to Build a Production-Ready RAG Document Q&A System in 2026

The one sentence that matters

A retrieval-augmented generation demo takes an afternoon. A RAG system you can put in front of paying users takes months, and the months go somewhere specific: retrieval quality.

Almost every struggling RAG project I have seen has the same shape. Answers are wrong or vague, so the team rewrites the prompt. Then rewrites it again. Then tries a bigger model. Quality moves a little, plateaus, and nobody can explain why. The actual problem is upstream — the right passage never reached the model — and no amount of prompt engineering fixes a retriever that did not retrieve.

This article is the architecture I would build today for document Q&A, in the order the decisions actually matter. The organising principle throughout:

Retrieval and generation are two systems with two failure modes and two sets of metrics. Instrument them separately or you will tune blind.

Everything below follows from that.

Contents

1. Architecture, in one diagram
2. Ingestion: where your quality ceiling is set
3. Chunking: the decision people get wrong
4. Embeddings and the vector store
5. Retrieval: hybrid search and reranking
6. Generation: grounding, citations, refusal
7. Evaluation: the part that gets skipped
8. Serving: latency, streaming, async
9. Observability and cost
10. Security, tenancy, and compliance
11. Failure modes and what to build first

---

1. Architecture, in one diagram

RAG has two pipelines that run at different times. Conflating them is the source of a surprising amount of confusion.

Indexing (offline, once per document):

source docs → parse → chunk → embed → store (vectors + text + metadata)

Querying (online, every request):

question → embed ──┐
                   ├→ hybrid retrieve → rerank → assemble context → generate → validate → stream
question → BM25 ───┘

A reasonable service decomposition, deliberately without version numbers because they rot faster than the ideas:

Do not start here. This is where you end up, not where you begin. Start with Postgres plus pgvector and a single service; you can serve a surprising amount of traffic that way, and you will learn what you actually need before you buy it. Splitting into microservices before you have a working retriever just gives you a distributed version of the same bad answers.

---

2. Ingestion: where your quality ceiling is set

Nothing downstream can recover information that parsing destroyed. This stage sets a hard ceiling on the whole system, and it is the least glamorous part of the build.

Budget real time for PDFs. Multi-column layouts, headers and footers bleeding into body text, tables flattened into number soup, scanned pages needing OCR. A bad parse is an invisible ceiling: retrieval looks fine, answers are subtly wrong, and the cause is three stages upstream.

Preserve structure, because structure is meaning. A heading tells you what the section is about. A table’s column headers make its rows interpretable. If your parser emits a flat wall of text, you have thrown away the signal that makes a chunk retrievable at all.

Attach metadata to every chunk. At minimum: source document, section heading, page, effective date, and — critically — whatever access-control key your application uses. Metadata is what makes filtered retrieval, recency weighting, citation, and tenant isolation possible later. Adding it retroactively means reindexing everything.

Make ingestion replayable. Store the raw source and log each transformation as structured events. When you change chunking strategy — and you will — you need to rebuild the index without re-fetching from source systems. This also gives you the audit trail that regulated deployments require.

---

3. Chunking: the decision people get wrong

One vector represents one chunk. That single sentence explains every chunking mistake.

Too small and the passage loses the context that makes it interpretable. A chunk reading “It expires after 30 days” is unretrievable, because nothing in it says what it is.

Too large and several topics average into one vague vector that matches many queries weakly and none strongly. Embeddings of long passages drift toward the mean and lose the specificity you need.

A defensible default to start from and then tune:

• Split on structure first — headings, then paragraphs, then sentences. Never split mid-sentence.
• Target roughly 300–500 tokens, with 10–15% overlap so a sentence spanning a boundary is not lost.
• Keep tables intact where you can, and prepend the column headers to each row you emit.
• Prepend the section heading to each chunk’s embedded text. This one trick measurably improves retrieval on structured documents and costs nothing.

The pattern worth adopting: parent–child. Embed small, precise chunks for retrieval; return the larger parent section to the model for context. This resolves the tension directly instead of forcing a compromise between findability and comprehensibility.

Treat chunk size as a parameter you tune against measured recall, not a setting you get right on the first try. Which requires that you are measuring recall — see section 7.

---

4. Embeddings and the vector store

Choosing an embedding model

Two rules that cause real incidents when ignored:

1. The same model must embed both documents and queries. Different models produce incompatible vector spaces. Mixing them fails silently and produces plausible-looking nonsense.
2. Changing the embedding model means reindexing everything. Version your index and be able to run two in parallel during a migration.

Do not pick from a public leaderboard alone. The MTEB leaderboard measures general retrieval; your corpus might be supply contracts, or clinical notes, or Django tracebacks. Take twenty labelled query-passage pairs from your own data and compare candidates on those. It takes an hour and regularly overturns the leaderboard ordering.

Choosing a store

The honest guidance is less exciting than the vendor comparison tables suggest:

• Already on Postgres, under a few million vectors? Use pgvector. One database, transactional consistency between your rows and your vectors, no new operational surface. This covers far more production systems than the discourse implies.
• Need heavy metadata filtering, higher scale, or hybrid search built in? A dedicated store — Qdrant, Milvus, Weaviate — earns its keep.
• Do not want to run it? A managed service is a legitimate answer; you are buying back operational time.

What actually matters more than the brand: filtering must happen before or during the vector search, not after. Post-filtering a top-k result set will silently return fewer results than you asked for, and on a multi-tenant system it is how you leak data across tenants.

Most implementations use HNSW for approximate nearest-neighbour search (Malkov & Yashunin, 2016). It is approximate by design — the recall/latency tradeoff is a tunable parameter, not a fixed property, and it is worth knowing which knobs your store exposes.

---

5. Retrieval: hybrid search and reranking

This is the section that decides whether your system works.

Vector search alone is not enough

Dense retrieval is excellent at paraphrase and conceptual similarity. “How do I get my money back?” and “What is the refund process?” share almost no words, and their embeddings sit close together. That is the whole value proposition.

It is correspondingly weak on exact tokens — part numbers, error codes, statute references, surnames, API method names. Embeddings blur precisely the strings you needed matched exactly. In document Q&A over technical or legal material, that is not an edge case; it is a large fraction of real queries.

Hybrid: dense plus lexical

Run BM25 alongside vector search and fuse the two ranked lists. Reciprocal Rank Fusion is the standard method because it needs no score normalisation between two incomparable scales (Cormack, Clarke & Buettcher, 2009):

RRF assigns each document a score of 1 / (k + rank) summed across result lists, with k around 60 damping the influence of top positions. It is a handful of lines of code, it is implemented natively in Elasticsearch and several vector stores, and it reliably improves recall because dense and lexical retrieval fail on different queries.

Hybrid search is close to a free win. If you take one thing from this article and you are currently running pure vector search, take this.

Reranking is the largest single improvement available

The reason is architectural. Vector search uses a bi-encoder: query and document are embedded independently, so scoring is a distance between two fixed vectors. That is what makes it fast enough to search millions of documents — and it also means the query and the document never actually interact.

A cross-encoder processes query and document together, attending across both. Far more accurate, and far too slow to run over a whole corpus.

So use both:

query → hybrid retrieve (top 50-100) → cross-encoder rerank → top 5-10 → generate

Typical reranking cost is on the order of a few hundred milliseconds. On most pipelines it is the highest-value latency you will spend.

A note on k

Raising k does not monotonically improve answers. Recall rises, precision falls, and each extra marginal passage gives the model another opportunity to ground its answer in the wrong place. This is why reranking a wide candidate set beats simply widening k — you get the recall of a wide net with the precision of a narrow one.

Filtering by time and permission

For regulated content, the current version of a document is often the only permissible source. Store valid_from / valid_to on each chunk and filter at query time. The same mechanism handles tenant isolation and document-level ACLs — and as noted above, it must run before the similarity search.

---

6. Generation: grounding, citations, refusal

By this point the hard work is done. The generation prompt’s job is narrow: use the supplied context, cite it, and decline when it does not contain the answer.

Answer the question using ONLY the numbered sources below.
Cite the source number after each claim, like [2].
If the sources do not contain the answer, reply exactly:
"I don't have information about that in the available documents."

[1] (handbook.pdf, "Parental Leave", 2026-01-14)
{chunk}

[2] (policy.pdf, "Entitlements", 2025-11-02)
{chunk}

Question: {question}

Four deliberate choices in that template:

“ONLY” is stated positively. Describing the target outperforms prohibiting the failure. “Answer only from these sources” beats “do not use outside knowledge.”

Citations are required inline. This makes verification cheap for the user and measurably reduces unsupported claims, because the model has to attach each assertion to something concrete.

The refusal has exact wording. A refusal you cannot detect is a refusal you cannot measure. Fixed phrasing makes it greppable in logs and testable in your eval suite.

Context ordering is not arbitrary. Put the strongest passages at the beginning and end. Recall on long contexts is measurably weakest in the middle — the “lost in the middle” effect (Liu et al., 2023) — and it does not disappear just because the context window got bigger.

Guardrails

Validate output before it reaches a user: schema conformance where you expect structure, PII detection, and a check that cited source numbers actually exist in what you supplied. That last one catches a specific and common failure — the model inventing a citation — and it is a few lines of code.

For content safety classification, Llama Guard and equivalents are reasonable off-the-shelf components. Do not rely on the generation prompt alone to enforce policy.

On model choice

I have deliberately not included a model comparison table with prices. Any such table is wrong within a quarter — the article you are reading originally contained one recommending models that were already superseded. Read the provider’s current pricing page.

What does not rot is the decision framework:

• Output tokens cost several times more than input tokens across every major provider. Constraining answer length is a cost control, not just a style choice.
• A large, unchanging prompt prefix should be cached. Providers charge a fraction for cached prefix reads. Order your prompt stable-content-first so the cache can work, then verify your cache-hit rate — a timestamp near the top of a system prompt silently defeats it on every request.
• Route by difficulty. Extraction and classification do not need your most capable model; synthesis across contradictory sources might.
• Self-hosting is an operations decision, not a cost decision. It wins on privacy, predictable capacity, and per-token price at sustained high volume. It costs you a GPU fleet and the people to run it.

---

7. Evaluation: the part that gets skipped

If you skip one section of this article, skip a different one. Without evaluation you are tuning by anecdote, and RAG has too many moving parts for that to work.

Build the dataset first

50–100 questions, each paired with the passage that should answer it. Draw them from real user queries where you can. That labelled passage is what makes everything else measurable.

Composition matters more than size:

Two of those rows carry disproportionate weight. Refusal cases are the most commonly missing and the most valuable: systems regress in both directions, becoming over-cautious after a safety-flavoured prompt edit or confidently fabricating after a helpfulness edit, and neither shows up in accuracy measured only over answerable questions. Regression cases are how the set compounds — every production bug becomes a permanent test, and after a year your suite encodes everything the system has ever got wrong.

Measure the two halves separately

Retrieval:

• Recall@k — was the correct passage anywhere in the top k? This is the primary metric. The model can only use what arrives.
• Precision@k — what fraction of returned passages were relevant? Diagnoses dilution as k grows.
• MRR / nDCG@k — how highly was the relevant material ranked?

Generation, assuming retrieval succeeded:

• Faithfulness — is every claim supported by the retrieved context?
• Answer relevance — does it address the question actually asked?
• Citation accuracy — do the cited sources exist and support the claim?
• Refusal accuracy — does it decline when the corpus genuinely lacks the answer?

Ragas implements several of these and is a reasonable starting harness.

The diagnostic that saves months

Was the correct passage retrieved?
├── No  → retrieval problem: chunking, embeddings, hybrid, reranking, k
└── Yes → generation problem: prompt, context ordering, model

Log the retrieved chunk IDs on every request. When an answer is wrong, check retrieval first. Tuning the prompt while the retriever is broken is the single most common way teams lose a quarter.

Two practical cautions

Know your noise floor. Sampling means the same input yields different outputs. Run an identical configuration twice and record the difference — that number is the minimum movement worth interpreting. Most teams have never measured it and consequently over-read every result.

If you use a model as a judge, calibrate it. Model graders have documented biases: they favour longer answers, more assertive answers, and answers stylistically like their own (Zheng et al., 2023). Grade thirty examples yourself, compare with the judge, and confirm they broadly agree before trusting it on three hundred.

---

8. Serving: latency, streaming, async

Where the time goes

A single request typically spends: embedding the query (tens of ms), retrieval (tens of ms), reranking (hundreds of ms), then generation — which dominates everything else and scales with output length.

Two consequences:

Stream, always, in anything user-facing. Streaming does not reduce cost by a single token, and it transforms perceived latency. The user sees progress in a few hundred milliseconds instead of staring at a spinner for eight seconds.

Optimise the right thing. Shaving 20ms off vector search is irrelevant when generation takes four seconds. Reranking is worth its few hundred milliseconds because it changes the answer; micro-optimising retrieval usually is not.

Synchronous or async?

Interactive Q&A should be synchronous and streamed. Reach for queues and workers when the work genuinely does not fit a request cycle:

• Ingestion and reindexing — always async. These are long, bursty, and retry-heavy.
• Bulk or scheduled question answering — async, and often a good fit for batch inference pricing.
• Interactive Q&A — synchronous. A queue here adds latency and complexity to solve a problem you do not have.

If you do add a queue, remember that an LLM call is not idempotent. If a request failed after a side effect committed, a naive retry repeats it. Use an idempotency key and record what has already been actioned.

---

9. Observability and cost

You cannot debug what you did not record, and aggregate metrics are not enough.

Log per request: request ID, the question, retrieved chunk IDs and their scores, the prompt version, the model ID, token counts (input, cached, output), latency split by stage, and the final answer.

Prompt version and model ID are the two that get omitted and the two you always need. Without them, “quality got worse last Tuesday” is unanswerable.

Track in aggregate: latency percentiles per stage (p50/p95/p99 — the mean hides everything), retrieval recall on your eval set over time, refusal rate, citation-validation failure rate, cache-hit rate, and cost per query.

Two alerts that catch real problems and are usually missing: cache-hit rate falling (someone put something volatile in the prompt prefix) and refusal rate moving sharply in either direction (a prompt or model change altered behaviour).

Sample for quality. Metrics tell you the system is up, not that it is good. Grade a small percentage of real traffic daily and track the trend. Slow quality drift will never appear in an infrastructure alert.

On cost, the levers in the order you should pull them: cache the stable prefix, trim redundant prompt text, constrain output length, batch anything not latency-sensitive, and only then consider a smaller model. The first four are free; the last one trades quality.

---

10. Security, tenancy, and compliance

Document Q&A systems tend to sit on exactly the material an organisation cares most about.

Access control belongs in the retrieval filter, in your code. Every chunk carries the permission key of its source document, and every query filters on the requesting user’s entitlements before the similarity search runs. Never ask the model to respect permissions — instructions are not an authorisation boundary.

Prompt injection is not solved. Retrieved content and your instructions arrive in the same token stream, with no cryptographic separation between them. A poisoned document can carry instructions. Mitigations reduce probability; they are not a boundary:

• The only structural control is capability separation — if untrusted content is in the context, dangerous capabilities must not be. A component that reads arbitrary documents should not also hold a send-email or write-database tool.
• Fence retrieved content clearly and label it as data.
• Validate output before acting on it.

Data residency and retention are contractual questions with real answers. Know where inference runs, what the provider retains, and for how long. If that matters to your buyers, it will appear in their security review.

---

11. Failure modes and what to build first

The failure table

Where standard RAG stops

Top-k retrieval answers local questions, where the answer sits in one or a few passages. It structurally cannot answer global ones — “how has this policy changed over five years?”, “what themes recur across these 400 incident reports?” No amount of reranking fixes this, because the answer is not in any k passages.

That needs a different shape: hierarchical summarisation over the corpus, or a graph index built by extracting entities and relationships and querying over community summaries (GraphRAG). Real additional machinery — build it only once you have confirmed your users actually ask global questions.

Recognising “this is a global question and my architecture is local” is the senior skill here. It saves months.

Build order

1. Evaluation set: 50-100 questions with expected passages
2. Baseline: structural chunking + vector search on pgvector. Record recall@k
3. Add hybrid retrieval (BM25 + RRF). Re-measure
4. Add cross-encoder reranking. Re-measure
5. Tune chunk size against recall@k
6. Only now tune the generation prompt
7. Add refusal handling; measure refusal accuracy
8. Add observability; sample production traffic for quality

Steps 2 through 5 are all retrieval. The generation prompt comes sixth on purpose — it is what teams tune first, and it is what matters least while recall is poor.

---

Ready to build?

The short version, if you skim nothing else:

• Retrieval is the system. Generation is the easy half.
• Hybrid search plus a reranker is the highest-leverage pair of changes available to most existing pipelines.
• Build the eval set before you build the pipeline. Fifty labelled questions beat any amount of intuition.
• Start on Postgres. Add infrastructure when measurement says you need it, not before.
• Log the retrieved chunk IDs. When something is wrong, that log tells you which half to fix.

I build Django and Python backends with LLM and retrieval components for clients who need this working rather than demoed. If you are somewhere in the middle of this and the answers are not good enough yet, get in touch — the first thing I will ask is what your recall@k is, and if the answer is “we haven’t measured it”, that is where we start.

---

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., the original RAG paper
• Retrieval-Augmented Generation for Large Language Models: A Survey — a broad map of the technique space
• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., on why context position matters
• Reciprocal Rank Fusion — Cormack et al., the standard hybrid-fusion method
• Efficient and robust approximate nearest neighbor search using HNSW — Malkov & Yashunin
• Judging LLM-as-a-Judge — Zheng et al., measured biases in model graders
• Ragas documentation — evaluation framework for RAG pipelines