Retrieval-Augmented Generation

Retrieval-augmented generation is a pattern that retrieves relevant information from sources outside a model's learned parameters and supplies it as context before the model generates an answer [1].

RAG is a way to make an AI answer use externally retrieved information at generation time, whether or not similar material appeared in the model's training data. Instead of asking the model to answer from its learned parameters and prompt alone, the system first searches a document store, pulls back relevant passages, adds those passages to the prompt or context, and asks the model to answer from them.

This can make AI systems more useful for company policies, product documentation, support knowledge bases, research notes, and internal procedures. It can also reduce some hallucination risk by grounding answers in supplied material. It does not guarantee correctness. The system can retrieve the wrong passage, miss the right document, use stale content, expose data the user should not see, or be manipulated by instructions hidden inside retrieved documents.

Many useful AI tasks depend on information that was not in the model's training data or that changes often. A support policy changes. A school handbook is updated. A product has a new setup step. A contract template has local exceptions. RAG gives builders a practical way to connect a model to selected knowledge without retraining the model each time.

The original RAG paper by Lewis and colleagues combined retrieval with generation for knowledge-intensive NLP tasks, showing the architecture as a research pattern rather than a guarantee of truth [1]. Vendor guidance from AWS is useful for practical implementation choices, such as retrieval options and architecture tradeoffs, but it should be treated as implementation guidance, not as the sole neutral authority for the concept [2].

For organizations, RAG is attractive because the knowledge source can be managed separately from the model. A team can update documents, remove stale pages, improve chunking, and change permissions without training a new foundation model. That makes RAG practical, but it also means the quality of the answer depends on ordinary information-management work: clean documents, clear ownership, current policies, and tested retrieval.

A basic RAG system has three parts. First is retrieval. The system takes the user question and searches documents or structured records, often using keyword search, vector embeddings, metadata filters, structured queries, or a combination of methods. Second is augmentation, often better described as context assembly. The system adds the retrieved material to the model prompt, usually with instructions about how to use it. Third is generation. The language model writes an answer using the user question, the instructions, and the retrieved context.

A RAG answer may include citations or source labels, but provenance is only useful when it is accurate. A citation can point to a document that was retrieved but does not support the exact sentence. A passage can be relevant to the topic but not to the claim. A model can also blend retrieved text with unsupported language. That is why source verification remains part of the workflow.

RAG also creates security and governance questions. NIST treats generative AI systems as risk-managed systems across their lifecycle, not as isolated model calls [3]. OWASP's 2025 Top 10 for LLM and generative AI applications highlights risks including sensitive information disclosure, excessive agency, vector and embedding weaknesses, and misinformation [4]. Its prompt injection guidance also describes indirect attacks where malicious instructions are hidden in external content that an AI system processes [5]. Retrieved documents are one place where that risk can appear.

A builder should test each part separately. Retrieval tests ask whether the right passages are found. Grounding tests ask whether the answer stays inside those passages. Citation tests ask whether each source marker supports the sentence it appears beside. Permission tests ask whether the user was allowed to retrieve the material in the first place. End-to-end tests ask whether the whole system handles real user questions and failure cases.

Imagine a customer support assistant for a software company. A customer asks, "Can I get a refund after 45 days if the product never activated?" The RAG system turns the question into a search, retrieves chunks from the refund policy and activation troubleshooting guide, and adds those passages to the model context.

The model then generates a response: "The policy says refunds are normally available within 30 days, but failed activation can be escalated for review. Ask the customer for the activation error and order ID." A good interface would show the policy passages used for that answer. A human support worker should still check that the retrieved policy is current, that the customer account has the right purchase date, and that the answer does not expose internal-only notes.

The biggest misconception is that RAG makes hallucinations disappear. RAG can reduce some risk because the model has relevant context to work from. It can also make failures easier to inspect because users can look at the retrieved passages. But the system can still fail before generation, during generation, or after generation.

Another misconception is that a source link means the answer is proven. The retrieved source must actually support the claim. If the answer says "refunds are allowed after 45 days," but the source says "refunds are normally allowed within 30 days," the citation is not enough.

Retrieval failure is the first major risk. The system may retrieve a popular but irrelevant page, miss a rare exception, or choose a chunk that lacks the needed surrounding context. Stale or missing documents are another risk. A perfect search over an outdated policy still produces outdated grounding.

Access control is also central. A RAG system should not retrieve salary documents, customer records, legal notes, or internal security procedures for users who are not allowed to see them. Prompt injection in retrieved documents is a separate risk: a malicious or compromised document can contain instructions that try to override the system task, leak data, or change the answer. RAG needs document hygiene, permissions, monitoring, and human review for high-stakes uses.

Another limitation is evaluation. A demo can look strong with a few hand-picked questions, while the real system fails on edge cases, synonyms, long documents, conflicting policies, or missing information. A useful RAG system needs examples of good answers, bad retrievals, no-answer cases, stale documents, and permission boundaries. Teams should also decide how the product behaves when retrieval results are weak, conflicting, or absent: answer with uncertainty, ask a follow-up question, route to a person, or refuse to answer from weak evidence.

Before trusting a RAG answer, ask: Were the right documents retrieved? Are they current? Is the answer limited to what the sources support? Are citations attached to the exact claims they support? Could any retrieved document contain hostile instructions? Does the user have permission to see every retrieved source? What should happen if retrieval finds nothing?

For low-stakes drafting, a RAG answer can be a strong starting point. For policy, security, customer, legal, financial, medical, or employment decisions, require source review and a clear approval owner. RAG reduces some risks by bringing evidence closer to the model, but it does not replace verification. Good systems make uncertainty visible instead of hiding it during real use and review.

Continue with Embedding for retrieval, Large Language Model and Context Window for generation, and Source Verification for checking whether retrieved passages support the answer.