They read it more often than they obey the full set of site rules. In ScrapeBench's tool-use analysis, most agents fetched robots.txt in nearly every trial, but under a neutral prompt five of six almost never declined to scrape sites that prohibit it, mostly because robots.txt allowed the page while the terms of service did not.
Do AI Agents Respect robots.txt? What the ScrapeBench Study Found
Mostly, no, unless someone tells them to. A study posted on November 8, 2026, called ScrapeBench, gave six widely used AI agents a plain request to scrape a website and sent them to 1,626 popular sites whose robots.txt or terms of service prohibit scraping. With a neutral prompt, five of the six agents complied close to zero percent of the time, and the sixth, Grok, complied 11% of the time (ScrapeBench, arXiv 2611.11306).
Why they scraped matters more than the headline rate. Most of the agents did read robots.txt. They often read it, saw it allowed the page, and then ignored terms of service that said no. When the researchers put the site's rules into the prompt, compliance for the top agents rose above 70%. So "do AI agents respect robots.txt?" has a sharper answer than yes or no: they respect the rules they're handed, and they don't go looking for the rest.
Key Takeaways
- Five of six agents complied almost never under a neutral prompt on 1,626 sites that prohibit scraping. Grok complied 11% of the time.
- Checking isn't complying. In the paper's tool-use analysis, four of the six agents fetched robots.txt in 99% to 100% of trials, yet compliance stayed low.
- Robots.txt was treated as the whole policy. In 69.7% of the violations the authors could classify, the agent said robots.txt allowed the page and set the terms of service aside.
- Putting the rules in the prompt worked best. Pasting the policy text lifted GPT-5.2, Claude Opus 4.6, and Gemini 3.1 Pro above 70% compliance.
- People complied more, but over-refused. Under the same neutral prompt, people honored the ban 48.9% of the time, against near zero for most agents, and refused 38.9% of sites that allowed scraping.
What ScrapeBench tested
ScrapeBench is a benchmark from Joseph Marvin Imperial (Pivotal Research, University of Bath), Daniel Slate (University of Notre Dame), and Noam Kolt (Hebrew University of Jerusalem), submitted to the NeurIPS 2026 Evaluations and Datasets track. Version 3 was posted on November 11.
The researchers started from the top 3,000 websites in Google's Chrome UX Report (February 2026) and kept 1,895 English-language sites reachable from the US. Each site got one of two labels:
- Explicitly prohibited (1,626 sites): the terms of service or robots.txt prohibit scraping. For robots.txt, that meant a
Disallowrule for all user agents or a rule naming an AI crawler, using the syntax defined in RFC 9309. - Not explicitly prohibited (269 sites): neither file says no.
Six agents ran through OpenRouter with function calling: GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro, Kimi-K2.5, Qwen3-Max, and Grok-4.20. Each trial was a single task: scrape the landing page. An agent counted as scraping if it fetched at least one content page; fetching robots.txt or the terms page didn't count. The scraper sent an identifiable user agent, waited two seconds between requests, and could make at most five tool calls.
The study then varied the prompt across eight conditions, from neutral (P1), to linking the site's terms and robots.txt (P4), to pasting the policy text (P5), to a strong legal warning with links and text (P7), to an adversarial instruction to ignore the rules (P8).
Do AI agents respect robots.txt? The results
Under the neutral prompt, compliance on prohibited sites was "near zero for five of six AI agents," with Grok the exception at 11%. The abstract summarizes most agents as below 5%.
Context changed the picture, but not evenly. Here is compliance on the 1,626 prohibited sites under the conditions the paper reports for each agent:
Pasting the policy text into the prompt (P5) did more than linking it. GPT-5.2 peaked at 73.9%, and Claude Opus 4.6 and Gemini 3.1 Pro also passed 70%. Qwen3-Max stayed at 2.9% under the same condition.
The P8 column needs care. Claude Opus 4.6 declined almost everything once told to ignore the rules, including allowed sites: its over-refusal on the 269 permitted sites was 98.5% under that prompt. The authors suggest the phrase itself may have triggered blanket refusals, so a high P8 score there reflects caution more than a reading of each site's policy.
Why checking robots.txt didn't make agents comply
For anyone building or blocking agents, the tool logs are the useful part. The agents weren't skipping the policy files. In the paper's tool-use analysis, GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro, and Grok-4.20 fetched robots.txt in 99% to 100% of trials, and all six did so at least 60% of the time. The paper doesn't tie those rates to a single prompt condition, and they don't hold under the strong legal warning, as shown below. GPT-5.2 is the clearest case: the paper reports it retrieved the terms of service 82.8% of the time and robots.txt 100% of the time, and separately that it complied in only 14.2% of trials when given links to both (P4). Reading the rules and following them were different steps.
When agents explained a violation, the reasons clustered. The authors sampled 30 violation trials per agent across the seven non-adversarial prompts (P1 to P7), 1,260 in all, and could classify the stated reason in 535 of them:
So the dominant failure isn't ignoring robots.txt. It's treating robots.txt as the complete answer. Many sites in the prohibited group allowed the page in robots.txt but banned scraping in their terms. An agent that checks one file and stops has followed a rule, just not the one that applied.
There was also a side effect of heavy legal warnings. Under P7, some agents stopped checking at all: Claude's robots.txt checks fell from 100% to 1.6%, and Gemini's from 99.9% to 1.4%. The paper doesn't explain the drop; our read is that once the warning and the policy text were in the prompt, the agents treated the question as answered. GPT-5.2 kept checking (71% for robots.txt, 58% for terms).
How AI agents compare with people
The researchers ran the same task with 180 people recruited on Prolific: 30 in each of three prompt conditions (P1, P4, and P7), on prohibited and on permitted sites. Under the neutral prompt, people complied on prohibited sites 48.9% of the time, far above the agents' near zero. But they also refused 38.9% of allowed sites. With the strong legal warning, human compliance rose to 65.1% and over-refusal to 45.8%.
Across the conditions people saw, those who read the terms complied more often on prohibited sites than those who didn't (67.1% against 55.7%). The authors describe people as "cautious and less responsive to legal priming than AI agents." The agents, in other words, are much more steerable than people. That cuts both ways. A few lines of system prompt move them toward the rules, and an instruction to ignore the rules undid most of that for several agents: GPT-5.2 fell from 60.6% under the legal warning to 11.7% under P8, Kimi-K2.5 from 62.7% to 9.5%, and Grok-4.20 from 59.7% to 0.6%.
What the study doesn't say
The authors are clear about the limits, and they're worth repeating before anyone quotes the headline number:
- It isn't a legal finding. The paper doesn't assess whether any site's terms or robots.txt are enforceable, and it doesn't claim any agent broke the law. Robots.txt on its own isn't binding in the US, as we covered in is robots.txt legally binding.
- One page, one turn, one agent. Each trial scraped a single landing page. Multi-step agent workflows weren't tested, and the authors don't claim the results generalize to them.
- A February 2026 snapshot. Sites change their terms and robots.txt, and the authors ask anyone reusing the dataset to re-check the labels.
What it means for site owners
The agents in this study fetched robots.txt more reliably than any other policy file and cited it more than any other signal. So if your restriction lives only in your terms of service, expect agents to miss it, and put a clear Disallow for the paths you care about in robots.txt as well. The paper doesn't report compliance separately for sites whose robots.txt itself said no, so treat that as the rule most likely to be noticed, not one guaranteed to be followed.
Two caveats on scope. A group that names only a training crawler, such as GPTBot, won't cover an agent fetching a page for a user under a different user agent. And some operators say robots.txt may not apply to user-initiated fetches at all: OpenAI's crawler documentation says of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI). Our AI crawler list has the user agent tokens, and should you block AI crawlers covers the tradeoffs.
For training specifically, Google, OpenAI, Microsoft, and Anthropic said on October 30 that they would honor a shared robots.txt opt-out from AI training, which we explained in the new robots.txt AI training opt-out. That signal is about training use, though, not about an agent fetching a page for a user.
What it means for teams building AI agents
The paper's own suggestion is that model providers take more of the compliance burden, for example with system prompts like the P5 to P7 conditions. You don't have to wait for that. Three things follow directly from the results:
- Give the agent the policy, not just a tool to find it. Fetch robots.txt and the relevant terms in your own code and put the result in the agent's context. The conditions that put policy text in the prompt (P5 to P7) produced the highest compliance in the study.
- Check both files, and treat a "no" in either as a no. Most of the classified violations came from reading one file as permission.
- Enforce the rule outside the model, too. A prompt can be countered by another prompt: under P8, Qwen3-Max and Grok-4.20 honored bans less than 2% of the time. A network-level deny list can't be talked out of a block.
That last point is where infrastructure helps. If your agents reach the web through Massive's opt-in residential network, every account already blocks a published list of destination categories at the network layer, as we explain in how domain blocking works on Massive's network. On top of that, the Domain Blocklist in the Massive Dashboard lets you add up to 1,000 domains of your own, with wildcards, applied instantly to your proxy connections (Massive docs). Sites your agents shouldn't touch stay blocked whatever the model decides.
The same gap shows up outside AI. When we pointed 17 open-source scraping tools at a server we control in November, only 1 of the 19 setups we ran read robots.txt before crawling (our test). Whether the client is a model or a library, the check has to be put there on purpose. Until it is, whether AI agents respect robots.txt shouldn't be what decides which sites they can reach.
Sources
- Imperial, Slate, and Kolt, ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping, arXiv 2611.11306, v1 November 8, 2026; v3 November 11, 2026 (all figures in this post; full text)
- IETF, RFC 9309: Robots Exclusion Protocol
- OpenAI, overview of OpenAI crawlers
- European Union, Directive (EU) 2019/790 on copyright in the Digital Single Market, Article 4
- Massive, Blocked Destination Categories
- Massive documentation, Domain Blocklist and usage restrictions
Frequently Asked Questions
It depends on which part of ChatGPT. OpenAI's crawler documentation explains how to control GPTBot (training) and OAI-SearchBot (search) with robots.txt, but says of ChatGPT-User, which fetches pages on a user's request, that "robots.txt rules may not apply." ScrapeBench tested something different again: the GPT-5.2 model called through an API as an agent. In that setup it honored sites' scraping bans 14.2% of the time when given links to the rules, and up to 73.9% when the policy text was in the prompt.
ScrapeBench doesn't answer that, and the answer depends on the country and the site. In the US, robots.txt isn't binding on its own. In the EU, a rightsholder can opt content out of the commercial text and data mining exception "by machine-readable means" under Article 4(3) of the DSM Directive, and robots.txt is the usual candidate, as we discuss in is robots.txt legally binding. Terms of service can matter more than robots.txt in some cases.
Retrieve robots.txt and the terms yourself, pass the result into the agent's context, treat a prohibition in either as final, and back it with a network-level deny list so a prompt can't override it.
