Should You Block AI Crawlers? What Each Bot Does Before You Disallow It

Should you block AI crawlers? It depends on which AI use you want to stop, because each major AI company now runs separate bots for training, for search, and for fetches a user asks for. The common myth is that one Disallow keeps a site out of "AI." In practice, blocking a training crawler such as GPTBot does not remove a page from ChatGPT's search answers, and blocking Google-Extended does not remove it from Google's AI Overviews.

The reverse also holds. On November 1, 2026, we checked the robots.txt files of 49 news sites: 35 block PerplexityBot, a bot Perplexity says "is not used to crawl content for AI foundation models." Publishers can have licensing or traffic reasons to block a search bot. Either way, those sites are opting out of a search bot, not a training crawler. This post explains what each bot does, according to the companies that run them, and how to choose.

Key Takeaways

• OpenAI, Anthropic, Perplexity, Amazon, Apple, and Google each publish separate robots.txt names by purpose: training, search, and in most cases user-triggered access. Perplexity documents no training crawler.
• Blocking GPTBot opts out of OpenAI model training. ChatGPT search uses OAI-SearchBot, and sites that block it "will not be shown in ChatGPT search answers."
• Google AI Overviews are controlled by Googlebot and snippet controls, not Google-Extended. 30 of 49 news sites we checked block Google-Extended without blocking Googlebot site-wide.
• User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, by their operators' own description.
• A shared training opt-out, __usage_ai-training, is coming in 2027. Our news post on the new robots.txt AI training opt-out covers it.

Where does the "block AI crawlers" myth come from?

The early advice was simple: add User-agent: GPTBot and Disallow: /, and you were out of ChatGPT. Since then, the companies have documented separate crawlers by purpose. OpenAI's documentation now says "each setting is independent of the others," so a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" (OpenAI crawler docs).

Content management plugins and CDN toggles that "block AI bots" also tend to block whole lists at once. Cloudflare, for example, announced on July 1, 2025 that new domains would block AI crawlers by default, with "every new domain" starting "with the default of control" (Cloudflare), which we covered in Cloudflare's AI crawler defaults. One switch can cover training bots and search bots alike.

What do training, search, and user bots each do?

The table below uses each company's own crawler documentation, as of November 1, 2026.

Three details in the docs change the decision:

1. Google's AI features run on Googlebot. Google says "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search," AI features included, and that nosnippet, data-nosnippet, max-snippet, or noindex limit what is shown (Google Search Central). Google-Extended governs Gemini model training and grounding in Gemini Apps and Vertex AI, and "does not impact a site's inclusion in Google Search" (Google crawler docs).
2. Blocking a search bot has a visible cost. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" (Anthropic).
3. User fetchers act for a person. OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion" and that robots.txt "may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules" (Perplexity crawler docs). Anthropic is the exception among the three: disabling Claude-User "prevents our system from retrieving your content in response to a user query."

What 49 news sites actually block

On November 1, 2026, we requested robots.txt from 50 major news sites with a browser user agent; 49 returned a valid file. We applied RFC 9309 group matching for each company's bot names. A bot counts as blocked when its group, or the * group if it has none, disallows the whole site.

• Training bots are the most blocked. ClaudeBot is disallowed on 39 of 49 sites, Applebot-Extended on 35, Google-Extended on 30, and GPTBot and Amazonbot on 28 each.
• Many sites already split training from search. 12 sites block GPTBot site-wide but not OAI-SearchBot (8 leave it unrestricted, 4 limit some paths), and 13 block ClaudeBot but not Claude-SearchBot (11 unrestricted, 2 limited). They stay at least partly eligible for ChatGPT and Claude search answers while opting out of training.
• 16 sites block OAI-SearchBot, and 26 block Claude-SearchBot, site-wide. Per OpenAI and Anthropic, that keeps them out of those search answers, or reduces their visibility there.
• 35 sites block PerplexityBot, which Perplexity says it does not use for model training. Perplexity recommends allowing PerplexityBot for a site to appear in its search results, so this block affects search, not training.
• 30 sites block Google-Extended but don't block Googlebot site-wide (21 leave it unrestricted, 9 limit some paths). Those pages can still appear in AI Overviews, because Googlebot controls Search, AI features included.

One caveat about method: we fetched each file with a browser user agent, and a site may serve crawlers a different file. One site's file, as served to us, disallows Googlebot itself, so we treat single-site results with care.

How to decide what to block

Start from the outcome you want, then pick the bots:

1. "Don't train on my content, but cite me in AI answers." Disallow the training controls: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, and Common Crawl's CCBot, which builds "an open repository of web crawl data that is universally accessible" (Common Crawl), so anyone, AI developers included, can download what it collects. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Note that Google-Extended also covers Gemini Apps grounding, so you may lose citations there.
2. "Keep me out of AI answers entirely." Disallow the search bots as well. For Google, that means snippet controls such as nosnippet, which also change how your page looks in regular results. User-triggered fetchers may still reach public pages, so a firewall rule is the only firm block. Before writing one, verify bots by the IP lists their operators publish, such as OpenAI's gptbot.json and Perplexity's perplexitybot.json, because a user agent string can be faked.
3. "I want maximum AI visibility." Allow everything, and make sure your CDN's bot settings don't block search bots you meant to allow.

Write the rules per bot name, and keep them separate from your * group. For outcome 1, the training part looks like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

RFC 9309 lets one group list several user agents, and search bots you don't name fall through to your * group (Amazon says Amzn-SearchBot follows the rules you give other search bots). These are the six operators we checked; other companies run crawlers of their own. A shared option is coming: the shared __usage_ai-training token lets one group express a training preference to companies that adopt it. It doesn't stop crawling, and wildcard rules don't count toward it.

For data teams on the other side of the file, the same split applies. Read robots.txt for the user agent you actually send, and keep records of what it said when you fetched. Our posts on whether robots.txt is legally binding and ethical web scraping cover why that record matters. Massive's own network is built on devices whose owners opted in through the Massive SDK, and the teams using it set their own collection rules.

The bottom line

• "AI crawler" now means three jobs: training, search, and user-triggered fetching. Each has its own robots.txt name.
• Blocking a training bot does not remove you from AI search, and blocking a search bot does not stop training.
• Decide the outcome first, then write a rule per bot name.

Sources

• OpenAI, Overview of OpenAI Crawlers
• Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
• Google Search Central, AI features and your website
• Google for Developers, Google's common crawlers
• Perplexity, Perplexity Crawlers
• Apple, About Applebot
• Amazon, Amazonbot
• Common Crawl, CCBot
• Cloudflare, "Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large", July 1, 2025
• IETF, RFC 9309: Robots Exclusion Protocol, September 2022
• Massive, robots.txt check of 50 news sites, November 1, 2026 (first-party test)