--- title: Seven Places RAG Can Lie slug: rag-has-seven-places-to-lie canonical_url: https://blog.openfactoryai.com/rag-has-seven-places-to-lie published_at: 2026-07-14T20:50:00.289665+00:00 author: OpenFactoryAI tags: RAG, retrieval augmented generation, vector search, groundedness, citations tldr: Retrieval-augmented generation fails before, during, and after vector search. A source may be missing, stale, unauthorized, badly chunked, not retrieved, removed by reranking, buried in context, ignored by generation, or cited without supporting the claim. Treat RAG as a traced control system with stage-specific metrics and a no-answer path. Optimize supported correct outcomes, not retrieval similarity or answer fluency alone. key_takeaways: - Corpus coverage is upstream of retrieval quality. - Retrieval recall and answer correctness need separate labels. - Reranking and context assembly can delete good evidence after retrieval finds it. - A citation is correct only when the cited span supports the attached claim. - Stage-level traces turn aggregate RAG failure into a repairable control problem. --- A RAG answer cites a real document and is still wrong. The document may not contain the claim. The cited section may contradict it. The index may hold an old revision. A correct passage may have been retrieved and then dropped. The model may have answered from memory while attaching a nearby citation. Calling this a “vector database problem” is like blaming a warehouse shelf for a wrong shipment. Retrieval-augmented generation is a chain: ```text source -> ingest -> index -> retrieve -> rerank -> assemble -> generate -> verify citation -> observe outcome -> update ``` Every arrow can lose information or authority. The system is useful only when it can say where. ::figure{template=pipeline caption="RAG is an evidence control loop, not a vector lookup" items="Sources :: versioned truth | Ingest :: parse and authorize | Retrieve :: candidate recall | Rerank :: evidence priority | Context :: bounded assembly | Generate :: use evidence | Verify :: claims and citations | Feedback :: repair the failing stage"} ## The original idea was retrieval plus generation [Lewis et al.](https://papers.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html) introduced retrieval-augmented generation models that combine parametric memory in a generator with non-parametric memory in a dense Wikipedia index. The paper frames provenance and knowledge updating as important motivations. Production teams generalized “RAG” to many architectures: keyword or dense search, hybrid retrieval, metadata filters, rerankers, prompt assembly, hosted generators, and citation formatting. That flexibility is useful. It also means a RAG label conveys almost no reliability contract by itself. Define the shipped system: ```text corpus and version ingestion and chunking retrieval methods and k filters and authorization reranker and selection context construction generator and instructions claim/citation verification no-answer and escalation policy ``` Only then can an evaluation result be reproduced. ## Failure 1: the source was never eligible evidence Before retrieval, ask whether the current, authoritative answer exists in the corpus. Common failures: - the relevant system was never connected; - a parser dropped a table, footnote, image, or code block; - the latest revision failed ingestion; - deletion did not propagate; - access-control metadata is missing; - the source conflicts with a more authoritative source; - a timestamped fact has expired. Measure corpus coverage on real questions. For each answerable item, identify the source and exact supporting span before evaluating a retriever. If annotators cannot locate evidence, a retrieval miss cannot be diagnosed honestly. Store source identity through every transformation: ```text document_id, revision, content_hash, authority, valid_from, valid_to, tenant, ACL, parser_version, chunk_id, source_offsets ``` An embedding without provenance is not evidence. ## Failure 2: chunking broke the fact Chunking makes documents searchable but can separate a rule from its exception, a table header from its row, or a function signature from its contract. Fixed token windows are simple. Structure-aware chunks preserve headings, paragraphs, tables, and code units. Parent-child retrieval can index small passages but return a larger source neighborhood. Overlap protects boundaries while increasing duplication and index cost. Test chunking with boundary cases: - answer and unit in adjacent cells; - prohibition followed by exception; - definition in a heading and condition below; - entity name in one paragraph and pronoun in the next; - code symbol and implementation separated; - multi-hop answer across two documents. Record the original offsets so a reranker or citation viewer can recover context without inventing it. ## Failure 3: retrieval optimized similarity, not evidence recall [Dense Passage Retrieval](https://aclanthology.org/2020.emnlp-main.550/) uses a dual encoder to place questions and passages in a shared vector space. Dense retrieval can capture semantic relationships beyond exact token overlap. Sparse retrieval remains valuable for identifiers, names, numbers, error codes, and rare terms. Hybrid retrieval is not automatically superior. It adds weights and merging policy that need evaluation. The relevant metric at this stage is usually evidence recall at `k`: ```text recall@k = questions where at least one supporting passage is in top k ------------------------------------------------------------ answerable questions with annotated supporting evidence ``` Also report authorization precision. A retriever that finds the right private passage for the wrong user is not accurate. Build slices for: - semantic paraphrase; - exact entity or identifier; - rare terminology; - multilingual query/source pairs; - temporal and version-sensitive questions; - multi-hop evidence; - no-answer questions. One aggregate recall hides the retrieval family you need. ## Multi-hop questions turn retrieval into a workflow Some questions cannot be supported by one passage. “Which policy applies to the account that owns invoice 17?” may require an invoice-to-account lookup followed by an account-to-policy lookup. A single embedding query can retrieve both by luck, but the system needs an explicit plan when the second query depends on the first result. A multi-hop trace should show: ```text question -> hop 1 query and candidate set -> selected entity or relation -> hop 2 query derived from verified hop 1 state -> combined evidence graph -> answer claims and citations ``` Do not let an unverified first-hop model guess become a trusted second-hop filter. If hop one extracts an account ID, validate its format and source span before using it to fetch private policy. The second retrieval inherits the authorization scope of the task, not authority invented by the generated intermediate text. Recall compounds. If each of two required hops finds its evidence 90 percent of the time under the relevant conditional workload, the complete path is at most 81 percent before reranking and generation: ```text .90 * .90 = .81 ``` The multiplication is an illustrative conditional model, not a claim that hop failures are independent. In production, estimate the traced transition directly. A shared entity-resolution error can correlate both failures. Evaluate: - hop-level supporting evidence recall; - correctness of the entity or relation passed forward; - whether both sources survive final context selection; - whether the answer joins them correctly; - citations attached to each claim, not only the final paragraph; - stops when a required hop lacks evidence. Parallel retrieval can reduce latency when hops are independent. Sequential retrieval is necessary when the next query genuinely depends on a verified result. An agent that launches every conceivable search in parallel may improve recall while flooding context with distractors and multiplying tool cost. Apply a bounded search budget and stop when the evidence contract is satisfied. ::figure{template=branch caption="Multi-hop RAG must verify each edge before deriving the next query" items="Question :: needs invoice owner policy | Hop 1 :: retrieve invoice 17 | Verify :: account ID and source | Hop 2 :: retrieve current account policy | Verify :: authority and date | Join :: construct supported answer | Stop :: cite both spans"} ## Freshness is a control loop, not a nightly re-index A retriever can achieve perfect recall against an obsolete index. The answer remains wrong. Define freshness by source type. A product manual may tolerate hours. Inventory, permissions, incidents, and balances may require a live tool call or transaction snapshot. “Most recent chunk in the vector store” is not a freshness guarantee unless ingestion delay and revision lineage are measured. Track four timestamps: ```text source effective time source observed or committed time index available time answer generated time ``` The difference between source commit and index availability is ingestion lag. The difference between source effective time and answer time is evidence age. Both need SLOs where staleness has consequence. Use event-driven invalidation when source systems emit reliable changes. Tombstone deleted or revoked documents so old vector replicas cannot continue serving them. Keep an index version in every answer trace. During a rolling rebuild, route a request to one coherent snapshot rather than combining candidates from incompatible revisions without declaring it. For live facts, retrieve stable explanatory policy from the corpus and fetch the changing value through an authorized tool. The response can cite the policy document and attach a receipt or timestamp for the live value. This separates knowledge from state. Test freshness failures deliberately: - a document is revised while its old chunk remains searchable; - an access grant is revoked during a conversation; - two replicas expose different index versions; - a source is deleted but cached output survives; - a question asks “today” while the latest evidence is yesterday; - a retry crosses a revision boundary. The correct response may be to wait, fetch live, or state that the system cannot establish a current answer. Fresh-looking prose is not freshness. ::figure{template=matrix caption="Retrieval methods expose different evidence strengths and misses" items="Sparse :: exact terms / paraphrase miss | Dense :: semantic match / entity confusion | Hybrid :: broader recall / merge tuning | Metadata :: authority filter / missing tags | Multi-hop :: linked evidence / compounding recall | No retrieval :: parametric memory / no provenance"} ## Failure 4: reranking threw away the answer Retrievers favor speed across a large corpus. A reranker applies a more expensive relevance function to a smaller candidate set. [ColBERT](https://doi.org/10.1145/3397271.3401075) represents query and document tokens contextually and uses late interaction, occupying a design point between single-vector retrieval and full cross-encoding. Whatever method is used, evaluate two transitions: ```text retrieval candidate recall@k context evidence recall after reranking and truncation ``` If recall@50 is 92 percent but the final context contains support for only 74 percent, improving the vector index cannot recover the 18 points lost downstream. Reranking can overvalue passages that repeat the question, underweight terse authoritative tables, or select redundant chunks from one document. Add diversity and source-authority constraints where the task requires them. ## Failure 5: context assembly buried or corrupted evidence The selected passages must fit beside policy, query, tools, and output reserve. Assembly decides order, labels, separators, truncation, and conflict presentation. The [Window Is Not the Memory](/context-is-a-budget) shows why more context can reduce value. For RAG, use explicit blocks: ```text [SOURCE id=policy-17 rev=4 authority=official] verbatim excerpt with offsets [/SOURCE] ``` Do not silently concatenate passages into a synthetic document. Preserve disagreement. If two sources conflict, expose authority and date so the answer can abstain or explain the conflict. Test relevant evidence at different positions and under distractors. A good reranker cannot help if the final ordering makes the model ignore the evidence. ## Failure 6: generation saw evidence and did not use it Retrieval recall is not grounded answer quality. The model can: - answer from parametric memory instead of the source; - merge facts from incompatible passages; - invert a condition; - overgeneralize a narrow rule; - produce a correct claim for an unsupported reason; - refuse despite sufficient evidence; - answer when evidence is insufficient. Annotate claims, not only whole answers. For each atomic claim classify: ```text entailed by cited evidence contradicted by evidence not supported not requiring external support ``` Whole-answer exact match is useful for short QA but inadequate for long operational responses. [Self-RAG](https://openreview.net/forum?id=hSyW5go0v8) trains retrieval and critique behavior through reflection tokens. It is evidence that retrieval policy can be learned as part of generation. The model's self-critique is still a model output, not independent ground truth. Pair it with deterministic or human validation appropriate to consequence. ## Failure 7: the citation is real but does not support the claim A citation can fail in four distinct ways: - **invalid:** source or span does not exist; - **irrelevant:** source exists but concerns another subject; - **non-entailing:** related passage does not establish the claim; - **mis-scoped:** passage supports a narrower claim than the answer states. Citation verification should operate on atomic claims and exact source spans. URL validity alone catches only the first class. For high-consequence answers, use a verifier that has the claim and cited span, not the entire persuasive answer. Deterministic checks can validate identifiers, dates, quoted values, and source existence. Human review remains appropriate when entailment is nuanced and failure loss is high. Require a no-answer path: ```text if no authorized source supports the answer: state what is missing do not fill the gap from model memory retrieve again, ask a question, or escalate ``` Abstention is a successful control outcome when evidence is absent. ::figure{template=branch caption="A real citation can still fail support verification" items="Claim :: atomic statement | Source exists? :: validity | Same subject? :: relevance | Entails claim? :: support | Scope matches? :: qualification | Current and authorized? :: policy | Accept or abstain :: outcome"} ## Error rates multiply across the path Use an illustrative sequence of conditional pass rates: ```text current source exists .95 retriever finds support .85 reranker retains support .90 generator uses support correctly .88 citation verifier accepts true support .97 ``` The fully supported path is: ```text .95 * .85 * .90 * .88 * .97 = .6204 ``` About 620 of 1,000 illustrative questions traverse every gate. This is not a production forecast or independence assumption. Real stages correlate. Estimate conditional transitions directly from traced examples. The point is structural: five metrics that each look respectable can still yield a weak end-to-end outcome. ::figure{template=waterfall caption="Illustrative conditional gates reduce 1,000 questions to 620 supported outcomes" items="Questions :: 1000 | Corpus gap :: -50 | Retrieval miss :: -142.5 | Rerank loss :: -72.75 | Generation misuse :: -87.21 | Verification loss :: -12.79 | Supported path :: 620.35"} Do not improve the easiest metric. Improve the stage contributing the most consequence-weighted loss. ## One score hides the repair Create a metric-to-stage map: | Stage | Primary metric | Important countermetric | |---|---|---| | corpus | answerable coverage, freshness | unauthorized inclusion | | parse/chunk | supporting-span preservation | duplication/index size | | retrieve | evidence recall@k | authorization precision, latency | | rerank/select | final-context evidence recall | redundancy, cost | | generate | claim correctness, evidence use | abstention on answerable items | | citations | claim-level entailment | verifier false rejection | | outcome | verified resolution | latency and cost | Aggregate answer accuracy belongs at the outcome layer. It cannot replace the intermediate measures needed to locate failure. ## Latency and cost are also a waterfall An illustrative trace might spend: ```text query policy/rewrite 24 ms hybrid retrieval 46 ms reranking 71 ms context assembly 9 ms model queue 180 ms prefill 220 ms decode 640 ms citation verification 85 ms total 1,275 ms ``` The values are invented. The trace shows why replacing the vector store may save little when decode and queue dominate. Conversely, a cross-encoder reranker can be material at high request volume even when generation is expensive. Track per-stage calls and bytes. Query rewriting, multi-hop retrieval, corrective retrieval, and citation repair can multiply work. Feed the full trace into cost per verified outcome, not merely the final model call. ## A builder playbook ### 1. Build a source-of-truth evaluation set Start with production-shaped questions, answerability, exact supporting spans, authority, revision, and authorization. Include unanswerable, stale, conflicting, and adversarial sources. ### 2. Freeze and name every stage Version corpus snapshot, parser, chunker, embedding model, index, query transformation, filters, reranker, context builder, generator, and verifier. A score without this lineage cannot be compared. ### 3. Emit an evidence trace For each answer, retain retrieved candidates and scores, filter decisions, reranked order, final context ranges, atomic claims, citations, verifier results, latency, cost, and outcome. ### 4. Evaluate transitions conditionally Measure retriever only on covered questions, reranker only where retrieval found evidence, and generation both with gold evidence and pipeline evidence. Gold-context generation isolates the reader ceiling. ### 5. Add a no-answer contract Calibrate abstention separately from answer correctness. Test whether unsupported confident answers fall as abstention rises, and price false refusals versus false claims. ### 6. Fix the largest failure contribution If corpus coverage is low, connect or refresh sources. If candidate recall is low, change retrieval. If final-context recall is low, fix reranking or budget. If gold-context generation fails, change prompt/model/constraints. If citations fail, verify claims. ### 7. Re-run outcome economics Include ingestion operations, indexes, reranking, model calls, corrective loops, verification, review, and failure loss. Optimize supported correct outcomes per dollar and deadline. ## The decision Buy or build a vector database when storage, filtering, and nearest-neighbor search are the bottleneck. Do not expect it to repair absent sources, broken chunks, bad authority, discarded evidence, unsupported generation, or decorative citations. RAG becomes dependable when every answer carries a trace from current authorized source to atomic claim, and every missing link has a controlled stop. ## References - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://papers.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html), Lewis et al., NeurIPS 2020. Establishes the parametric plus retrieved-memory formulation. - [Dense Passage Retrieval for Open-Domain Question Answering](https://aclanthology.org/2020.emnlp-main.550/), Karpukhin et al., EMNLP 2020. A foundational dual-encoder dense retrieval system and recall evaluation. - [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://doi.org/10.1145/3397271.3401075), Khattab and Zaharia, SIGIR 2020. Introduces token-level late interaction for retrieval. - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://openreview.net/forum?id=hSyW5go0v8), Asai et al., ICLR 2024. Integrates learned retrieval and critique behavior into generation. All stage rates and timings in the worked examples are illustrative. The complete arithmetic and limitations are retained in the claim ledger. ## FAQ ### Is RAG just a vector database? No. A vector database can store and retrieve candidates. RAG also needs current authorized sources, parsing, chunking, filters, reranking, context assembly, generation, claim-level citation verification, abstention, tracing, and feedback. ### What is the most important RAG metric? The outcome metric is supported correct resolution under the product's latency and cost constraints. To improve it, retain stage metrics such as corpus coverage, evidence recall, final-context recall, claim correctness, citation entailment, and abstention. ### Why can RAG cite a real source and still hallucinate? The cited passage may be irrelevant, non-entailing, stale, unauthorized, or narrower than the claim. The model may answer from parametric memory and attach a nearby citation. Verify atomic claims against exact spans. ### How should RAG retrieval be evaluated? On questions whose supporting source is confirmed in the indexed corpus, measure whether authorized supporting evidence appears in top k. Slice exact entities, paraphrases, rare terms, time-sensitive queries, multi-hop needs, and no-answer cases. ### What is a RAG no-answer path? When no authorized source supports an answer, the system explicitly states the evidence gap and retrieves again, asks for clarification, or escalates. It does not silently fill the gap from model memory. ### How do you debug a wrong RAG answer? Inspect the evidence trace: corpus revision, parsed spans, retrieved candidates, filter and rerank decisions, final context, atomic claims, cited spans, verifier outcomes, latency, and eventual resolution. The first broken transition identifies the repair layer. ## Test yourself ### 1. What should be established before measuring retriever recall? - [ ] The answer is fluent - [x] The current supporting source exists in the evaluated corpus - [ ] The vector dimension is large - [ ] The model has a long window **Explanation:** A retriever cannot find evidence that was never present or correctly ingested. ### 2. If retrieval recall@50 is high but final-context recall is low, which stage is the likely loss point? - [ ] Source creation only - [x] Reranking, selection, or context budgeting - [ ] GPU quantization - [ ] DNS **Explanation:** The evidence was found but removed before generation. ### 3. What makes a citation supported? - [ ] The URL exists - [x] The exact authorized current span entails the attached atomic claim at the stated scope - [ ] The model repeats it - [ ] The document is long **Explanation:** Validity and topical relevance alone do not establish entailment or scope. ### 4. Why are five 90%-plus component metrics not necessarily enough? - [x] Errors across sequential stages can compound - [ ] Percentages cannot be multiplied - [ ] Retrieval has no errors - [ ] Generation is deterministic **Explanation:** Conditional losses at each transition reduce the fully supported end-to-end path. ### 5. What is the best use of model self-critique in RAG? - [ ] Treat it as unquestionable ground truth - [x] Use it as one learned signal alongside independent verification - [ ] Remove citations - [ ] Skip retrieval **Explanation:** A model's critique can help but is not independent proof of its own support.