Building a RAG Pipeline on Live Web Data (Without Stale Indexes)
All Posts

Building a RAG Pipeline on Live Web Data (Without Stale Indexes)

Ryan Turner
Ryan Turner · Head of Growth

A live-web RAG pipeline retrieves from the open web at query time instead of reading from a pre-crawled vector index. That keeps answers fresh, because the data is fetched when the user asks, not weeks earlier when you ran your crawl. The trade-off is direct: live fetch adds latency and per-query cost, while a cached index is fast but goes stale. Most production systems we see land in the hybrid middle, fetching live for time-sensitive queries and reusing cached chunks under a freshness TTL.

Key Takeaways
  • Classic RAG answers from a static index, so its freshness ceiling is the date of your last crawl.
  • Live-web RAG discovers sources with a search API, fetches and cleans pages at query time, then grounds the answer with citations.
  • The hard part is not retrieval. It is deciding when to fetch live versus reuse a cached chunk, governed by a per-topic freshness TTL.
  • In 2025, Gartner projected 40% of enterprise apps would feature task-specific AI agents by end of 2026, up from under 5%, and those agents need current data.
  • Clean markdown beats raw HTML at the ingestion step because it cuts token cost and removes nav, ads, and boilerplate before chunking.

Classic RAG made sense when your corpus was a slow-moving knowledge base: docs, policies, tickets. Point it at the open web, however, and the model breaks down. Prices change, news breaks, rankings shift, and a vector index built last Tuesday confidently returns last Tuesday's reality. The fix is not a bigger index or a faster re-crawl schedule. Instead, it is moving the fetch to query time for the data that actually moves. RAG is retrieval-augmented generation: a model answers from documents you fetch and feed it, not from its training weights alone. This post walks the architecture stage by stage, then covers the freshness logic that separates live-web RAG from the classic version. For the wider context on serving agents current data, start with the pillar on how to give AI agents live web access.

Why does classic RAG go stale on web data?

Classic RAG goes stale because it answers from a snapshot. You crawl, chunk, embed, and store, then every query reads that frozen copy until the next crawl. For a stable corpus that is fine. For the open web, however, it is a liability, and demand for current-data agents is climbing. In 2025, Gartner projected that 40% of enterprise apps would feature task-specific AI agents by end of 2026, up from under 5% in 2025. Agents answering real questions cannot run on a stale snapshot.

The staleness problem has two parts. First, coverage: the web you indexed last month is missing pages that did not exist yet, so no amount of clever retrieval recovers them. Second, drift: pages you did index have changed underneath you, and your embeddings still point at the old text. Re-crawling on a tighter schedule narrows the window but never closes it, and meanwhile it burns compute on pages nobody will query.

Live-web RAG inverts the order. Instead of pre-fetching everything and hoping the right page is in the index, you discover and fetch sources at the moment of the query. As a result, the cost moves from "crawl the whole web continuously" to "fetch the handful of pages this query needs." For background on why grounding matters and how it reduces hallucination, see our guide on grounding LLMs with live web data.

What does a live-web RAG architecture look like?

A live-web RAG pipeline runs seven stages: query understanding, live source discovery, fetch and clean, chunk and embed, retrieve top-k, ground the generation with citations, then cache with a freshness TTL. The first six produce the answer. The seventh decides what you keep, so the next similar query can skip the live fetch. Each stage is concrete, and in practice most failures trace back to a weak source-discovery or fetch step.

Here is the flow as a step list:

1. query understanding -> rewrite the user question into search intent
2. source discovery -> search API returns candidate URLs
3. fetch + clean -> render each URL to clean markdown
4. chunk + embed -> split markdown, embed chunks at query time
5. retrieve top-k -> rank chunks against the query embedding
6. ground + cite -> LLM answers using only retrieved chunks, with source links
7. cache + TTL -> store chunks with a freshness deadline for reuse

The stages below describe each step. None of them require a giant pre-built index. The "vector store" here is small and short-lived, often scoped to a single query or session.

Stage 1: query understanding

Turn the raw user question into search intent before you touch the web. Strip conversational filler, expand abbreviations, and extract the entities and the time sensitivity. For example, "What's the latest on the X acquisition" implies recency; a definitional question does not. This stage decides how aggressively the rest of the pipeline favors fresh data over cached chunks. Cheap to run, with a big payoff on quality.

Stage 2: live source discovery

Discovery is where most pipelines quietly fail, because the model cannot ground on pages it never found. Source discovery is the step that converts query intent into a candidate URL set, typically through a search API rather than guessing domains. A geotargetable SERP endpoint matters here: results for "best X near me" or a price query differ by country and city, and you want the sources your user would actually see. For a comparison of the options, see web search APIs for agents.

This is the first stage where Massive's Web Render API does the work. The Search endpoint (/search) retrieves SERP results from major engines and is geotargetable by country, subdivision, or city. For queries that turn on what an AI summary says, awaiting=ai waits up to a minute for an AI Overview, and awaiting=answers pulls People-Also-Ask. You get the candidate URL set, ranked the way a real user in that location would see it.

Stage 3: fetch and clean

Fetching the candidate pages is where live RAG meets the modern web's defenses, and the modern web is hostile to bots. In 2025, Imperva reported that automated bots were 51% of all web traffic in 2024, the first time bots passed humans in a decade, with bad bots at 37%. Sites respond by blocking aggressively, so naive datacenter fetches get challenged or fed decoy content.

There are two requirements at this stage. First, your fetch has to survive the page's anti-bot layer, or you ground on an error page. Residential proxies route requests through real consumer devices, so traffic originates from residential IPs rather than a flagged datacenter range. Massive's Web Render API runs fetches over a real consumer-device network spanning 195+ countries with roughly 1.3M daily active devices. In our testing, residential-IP success on protected sites typically lands far higher than datacenter (rough ranges, residential ~85-99% versus datacenter ~20-40%); treat that as a vendor benchmark, not independent research.

Second, you want clean text, not raw HTML. The Browsing endpoint (/browser) supports format=markdown as a first-class output, returning LLM-ready markdown with nav, ads, and boilerplate stripped. That matters before chunking: markdown cuts token counts substantially versus raw HTML, which lowers embedding and generation cost and keeps your chunks meaningful instead of full of menu links. Practitioners have documented the same effect (dev.to, Browser Tools for AI Agents Part 4: Skip the Browser, 2026).

Stage 4: chunk and embed

Split the cleaned markdown into chunks and embed them at query time. Because the corpus is just the handful of pages this query pulled, this is fast and cheap; you are embedding kilobytes, not a crawl of the web. Keep chunks aligned to markdown structure, by heading and paragraph, so each chunk stays self-contained. Markdown's headings give you natural boundaries that raw HTML does not.

Stage 5: retrieve top-k

Rank the freshly embedded chunks against the query embedding and keep the top-k. With a small per-query corpus, retrieval is simple and you can afford a higher k, then let the generation model filter. The discipline here is to keep only chunks that clear a relevance threshold, so a weak source does not dilute the context window.

Stage 6: ground the generation with citations

Hand the model only the retrieved chunks and instruct it to answer from them, with a source link per claim. Grounding is the practice of constraining a model's answer to the retrieved evidence rather than its parametric memory, so this is the grounding contract: no chunk, no claim. Because each chunk carries its source URL from Stage 2, citations come for free, and a reader (or a downstream check) can verify the answer against the live page. Grounding on fetched-this-second text is the whole point of going live.

Stage 7: cache with a freshness TTL

Store the chunks you fetched with a freshness deadline so the next similar query can reuse them instead of re-fetching. This is what makes live RAG affordable at scale. The cache turns the second identical query from a full live fetch into a lookup, and the TTL is what keeps that lookup honest. The next section covers how to set it.

How do you avoid stale indexes with freshness TTLs?

You avoid stale indexes by attaching a freshness TTL to every cached chunk and fetching live again once it expires. A freshness TTL is a per-chunk time-to-live that marks how long a cached fetch stays trustworthy before it must be refreshed. The TTL is per topic, not global: a stock price might be valid for seconds, a product spec for days, an encyclopedia definition for weeks. When a query arrives, you check the cache first, serve chunks that are still inside their TTL, and trigger a live fetch for anything expired or missing. That is the hybrid middle, fast when you can be, fresh when you must be.

Set the TTL from the query understanding stage. If Stage 1 flagged the question as recency-sensitive, shorten or bypass the TTL and force a live fetch. If it is a stable definitional question, by comparison, a long TTL is fine and you serve from cache. This is the lever that controls your latency and cost: more live fetches mean fresher answers and higher per-query cost, more cache hits mean the reverse.

Invalidation matters as much as expiry. A TTL handles time-based staleness, but some events demand immediate invalidation: a page you cited 404s, a source you trust publishes a correction, or a known-volatile entity (a live score, a breaking story) appears in the query. Build an explicit invalidation path for those rather than waiting out the clock. In short, the combination of per-topic TTL plus event-driven invalidation is what separates a live-web pipeline from a classic index that simply re-crawls on a cron.

One more reason live tends to beat a static index in 2025: the open web is actively closing to bulk crawlers. Cloudflare reported that on July 1, 2025 it began blocking AI crawlers by default across roughly 20% of the web and launched a pay-per-crawl marketplace. As a result, a pre-built index of the open web gets harder and more expensive to maintain every quarter. Query-time fetch over a real-device network sidesteps the bulk-crawl problem, because you fetch a few pages a real user could reach, not the whole web on a schedule. If you want to expose this pipeline to agents as a callable tool, see how to build an MCP server for web data extraction.

When should you fetch live versus reuse a cached chunk?

Fetch live when the query is recency-sensitive or the matching cache entry is past its TTL; reuse a cached chunk when it is still fresh and the question is stable. The decision runs per query, informed by Stage 1's time-sensitivity signal and the chunk's remaining TTL. Getting this rule right is where you spend your latency and cost budget, so tune it against real traffic, not a guess.

A practical default: treat the cache as the fast path and live fetch as the correctness backstop. Serve from cache when you have an in-TTL chunk that clears your relevance threshold. Fall through to a live fetch, however, when the cache misses, the chunk has expired, the query carries recency intent, or the cached source has been invalidated. This keeps the common, repeated queries cheap while guaranteeing the volatile ones are current.

Tune the thresholds by watching two failure modes. Stale answers (a cache TTL set too long for that topic) push you toward shorter TTLs and more live fetches. Cost and latency spikes (too many live fetches on stable queries) push the other way. From what we observe across agent workloads, there is no single correct setting; the right balance depends on your traffic mix and how fast your sources actually change.

Sources

Frequently Asked Questions

Does live-web RAG replace the vector database?

No, it changes its role. Instead of a giant persistent index of the whole web, you keep a small, short-lived store scoped to a query or session, often just the chunks from the pages you fetched. You may still keep a persistent store for stable internal content. The live layer, meanwhile, handles the parts of the answer that move.

Isn't fetching at query time too slow for production?

It adds latency, but the freshness TTL is the mitigation. Repeated and stable queries hit the cache and return fast, while only recency-sensitive or cache-missed queries pay the live-fetch cost. Using fast speed tiers on the render step and a tight top-k keeps the live path lean enough for interactive use.

Why fetch over a real-device network instead of a plain HTTP client?

Because the modern web blocks bots aggressively. In 2025, Imperva reported automated bots were 51% of web traffic in 2024, and sites respond by challenging datacenter requests. Fetching over a real consumer-device network means requests come from residential origins, so protected pages return real content instead of a block page or decoy.

How do I pick a freshness TTL?

Set it per topic from how fast that data changes, not one global value. Volatile data (prices, scores, breaking news) gets seconds to minutes; stable reference content gets hours to weeks. Let the query-understanding stage shorten or bypass the TTL when it detects recency intent, and add event-driven invalidation for corrections and dead links.