What Is Retrieval-Augmented Generation (RAG)?

How RAG Works

RAG pairs two components: a retriever that locates relevant passages and a generator (the LLM) that reads those passages and produces a response. Patrick Lewis and co-authors introduced the architecture at NeurIPS 2020, pairing a pre-trained seq2seq model (parametric memory) with a dense vector index of Wikipedia accessed through a neural retriever (non-parametric memory). Their paper reported that RAG produces "more specific, diverse and factual" output than a parametric-only baseline (Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020), arXiv 2005.11401, 2020).

At runtime the pipeline works in three steps. First, the user's query is encoded into a vector and compared against a pre-indexed document store to surface the most relevant passages. Second, those passages are appended to the model's context window as additional input. Third, the LLM generates a response that blends its parametric knowledge with the retrieved evidence.

The document store is usually a vector database: a system that stores high-dimensional embeddings of text chunks and supports fast approximate-nearest-neighbor search. Vector databases backing RAG applications grew 377% year over year, the fastest growth of any LLM-related technology category tracked (Databricks, State of Data + AI (Enterprise Adoption & Growth Trends), 2024).

Why RAG Became the Standard Grounding Approach

LLMs freeze their knowledge at the end of pre-training. Any fact that changes after that cutoff, a price, a regulation, a product spec, requires either a new training run or a retrieval layer. RAG handles this cheaply. Instead of retraining, you update the document store and the model automatically cites current information the next time a relevant query arrives.

Enterprise adoption reflects this practicality. RAG became the dominant approach for grounding LLMs in proprietary or current data, with adoption rising to 51% of enterprises in 2024 from 31% the prior year (Menlo Ventures, 2024: The State of Generative AI in the Enterprise, 2024).

A second advantage is auditability. Because the retrieved documents are part of the context window, a developer can inspect which passages the model used and trace claims back to a source. This is difficult or impossible with a purely parametric model.

Enterprise knowledge bases. Internal documentation, legal filings, and support content change frequently. A RAG pipeline indexes these documents and lets employees or customers query them in natural language without waiting for the next model fine-tune.

Real-time web data for AI agents. AI agents need current information, product pages, news, search results, that no static corpus can supply. Feeding live web content into the retrieval layer gives the agent accurate, up-to-date context. Massive's Web Render API can serve as the fresh-web-data source layer here: the Search endpoint returns rendered search results (including AI Overviews via awaiting=ai), and the Browsing endpoint returns clean HTML or markdown from any public URL, both of which are clean formats for chunking and embedding into a RAG pipeline.

Code and API documentation. Library APIs change with every release. RAG over a versioned doc corpus lets a coding assistant cite the correct method signature for the version in use rather than producing an outdated one.

Customer support automation. Support bots grounded in a live product catalog, a return policy document, and a known-issues feed answer accurately and cite the relevant policy, reducing escalations.

Chunk size matters. Chunks that are too small lose context; chunks that are too large dilute the relevant signal. Most practitioners start with 256-512 token chunks with a 10-20% overlap to avoid splitting mid-thought.

Clean your source documents first. Boilerplate, navigation text, and ads in a retrieved page produce noisy embeddings and degrade answer quality. If you pull live web content, parse it to clean markdown or plain text before indexing.

Evaluate retrieval and generation separately. A poor answer can come from bad retrieval (the right document was not returned) or from bad generation (the model ignored a good document). Keeping metrics separate for each stage helps you find and fix the right component.

Refresh the index on a schedule that matches source change rates. A product catalog that updates daily needs daily re-indexing. A legal corpus that updates quarterly can be re-indexed less often. Stale documents in the retrieval store are a common source of RAG errors.

Watch context-window limits. Every retrieved chunk consumes tokens. With many passages, you may exceed the model's context window or dilute the signal. Re-ranking retrieved chunks by relevance and keeping only the top three to five is a practical fix.

RAG addresses the most common production failure mode for LLMs: answers that are plausible but stale or wrong because the model cannot access information beyond its training cutoff. By separating retrieval from generation, the architecture keeps the generative model focused on reasoning while the retrieval component handles knowing the current facts. Enterprise adoption crossed the 51% mark in 2024, and the supporting infrastructure, vector databases, retrieval APIs, and live web access layers, is now mature enough that most teams can build a working RAG pipeline without specialized ML expertise. The primary engineering challenge is no longer whether to use RAG, but which data sources to retrieve from and how to keep those sources fresh and clean.