# Should You Block AI Crawlers? What Each Bot Does Before You Disallow It Should you block AI crawlers? It depends on which AI use you want to stop, because each major AI company now runs separate bots for training, for search, and for fetches a user asks for. The common myth is that one `Disallow` keeps a site out of "AI." In practice, blocking a training crawler such as GPTBot does not remove a page from ChatGPT's search answers, and blocking Google-Extended does not remove it from Google's AI Overviews. The reverse also holds. On November 1, 2026, we checked the robots.txt files of 49 news sites: 35 block PerplexityBot, a bot Perplexity says "is not used to crawl content for AI foundation models." Publishers can have licensing or traffic reasons to block a search bot. Either way, those sites are opting out of a search bot, not a training crawler. This post explains what each bot does, according to the companies that run them, and how to choose. > **Key Takeaways** > > - OpenAI, Anthropic, Perplexity, Amazon, Apple, and Google each publish separate robots.txt names by purpose: **training**, **search**, and in most cases **user-triggered** access. Perplexity documents no training crawler. > - Blocking **GPTBot** opts out of OpenAI model training. ChatGPT search uses **OAI-SearchBot**, and sites that block it "will not be shown in ChatGPT search answers." > - **Google AI Overviews** are controlled by Googlebot and snippet controls, not Google-Extended. 30 of 49 news sites we checked block Google-Extended without blocking Googlebot site-wide. > - User-triggered fetchers such as **ChatGPT-User** and **Perplexity-User** may not follow robots.txt, by their operators' own description. > - A shared training opt-out, `__usage_ai-training`, is coming in 2027. Our [news post on the new robots.txt AI training opt-out](https://www.joinmassive.com/blog/robots-txt-ai-training-opt-out) covers it. ## Where does the "block AI crawlers" myth come from? The early advice was simple: add `User-agent: GPTBot` and `Disallow: /`, and you were out of ChatGPT. Since then, the companies have documented separate crawlers by purpose. OpenAI's documentation now says "each setting is independent of the others," so a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" ([OpenAI crawler docs](https://developers.openai.com/api/docs/bots)). Content management plugins and CDN toggles that "block AI bots" also tend to block whole lists at once. Cloudflare, for example, announced on July 1, 2025 that new domains would block AI crawlers by default, with "every new domain" starting "with the default of control" ([Cloudflare](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/)), which we covered in [Cloudflare's AI crawler defaults](https://www.joinmassive.com/blog/cloudflare-ai-crawler-defaults). One switch can cover training bots and search bots alike. ## What do training, search, and user bots each do? The table below uses each company's own crawler documentation, as of November 1, 2026. | Company | Training | Search and citations | User-triggered fetches | |---|---|---|---| | OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User ("robots.txt rules may not apply") | | Anthropic | ClaudeBot | Claude-SearchBot | Claude-User (lets sites limit user requests) | | Google | Google-Extended (also Gemini Apps grounding) | Googlebot, including AI Overviews | Not covered here | | Perplexity | None ("not used to crawl content for AI foundation models") | PerplexityBot | Perplexity-User ("generally ignores robots.txt rules") | | Apple | Applebot-Extended (a control token; it "does not crawl webpages") | Applebot | Not covered here | | Amazon | Amazonbot ("may be used to train Amazon AI models") | Amzn-SearchBot | Amzn-User ("may not follow all robots.txt directives") | Three details in the docs change the decision: 1. **Google's AI features run on Googlebot.** Google says "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search," AI features included, and that `nosnippet`, `data-nosnippet`, `max-snippet`, or `noindex` limit what is shown ([Google Search Central](https://developers.google.com/search/docs/appearance/ai-features)). Google-Extended governs Gemini model training and grounding in Gemini Apps and Vertex AI, and "does not impact a site's inclusion in Google Search" ([Google crawler docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)). 2. **Blocking a search bot has a visible cost.** OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). 3. **User fetchers act for a person.** OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion" and that robots.txt "may not apply." Perplexity says Perplexity-User "generally ignores robots.txt rules" ([Perplexity crawler docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)). Anthropic is the exception among the three: disabling Claude-User "prevents our system from retrieving your content in response to a user query." ## What 49 news sites actually block On November 1, 2026, we requested robots.txt from 50 major news sites with a browser user agent; 49 returned a valid file. We applied [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) group matching for each company's bot names. A bot counts as blocked when its group, or the `*` group if it has none, disallows the whole site. ![Bar chart of how many of 49 news sites block each AI bot site-wide on November 1, 2026. Training bots: ClaudeBot 39, Applebot-Extended 35, Google-Extended 30, GPTBot 28, Amazonbot 28. Search bots: PerplexityBot 35, Claude-SearchBot 26, OAI-SearchBot 16, Amzn-SearchBot 11, Applebot 4, Googlebot 1. User-triggered fetchers: Claude-User 26, Perplexity-User 24, ChatGPT-User 23, Amzn-User 8.](assets/ai-bots-blocked-by-purpose.png "How 49 news sites treat AI bots by purpose, checked November 1, 2026.") - **Training bots are the most blocked.** ClaudeBot is disallowed on 39 of 49 sites, Applebot-Extended on 35, Google-Extended on 30, and GPTBot and Amazonbot on 28 each. - **Many sites already split training from search.** 12 sites block GPTBot site-wide but not OAI-SearchBot (8 leave it unrestricted, 4 limit some paths), and 13 block ClaudeBot but not Claude-SearchBot (11 unrestricted, 2 limited). They stay at least partly eligible for ChatGPT and Claude search answers while opting out of training. - **16 sites block OAI-SearchBot,** and **26 block Claude-SearchBot,** site-wide. Per OpenAI and Anthropic, that keeps them out of those search answers, or reduces their visibility there. - **35 sites block PerplexityBot,** which Perplexity says it does not use for model training. Perplexity recommends allowing PerplexityBot for a site to appear in its search results, so this block affects search, not training. - **30 sites block Google-Extended but don't block Googlebot site-wide** (21 leave it unrestricted, 9 limit some paths). Those pages can still appear in AI Overviews, because Googlebot controls Search, AI features included. One caveat about method: we fetched each file with a browser user agent, and a site may serve crawlers a different file. One site's file, as served to us, disallows Googlebot itself, so we treat single-site results with care. ## How to decide what to block Start from the outcome you want, then pick the bots: 1. **"Don't train on my content, but cite me in AI answers."** Disallow the training controls: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, and Common Crawl's CCBot, which builds "an open repository of web crawl data that is universally accessible" ([Common Crawl](https://commoncrawl.org/ccbot)), so anyone, AI developers included, can download what it collects. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot. Note that Google-Extended also covers Gemini Apps grounding, so you may lose citations there. 2. **"Keep me out of AI answers entirely."** Disallow the search bots as well. For Google, that means snippet controls such as `nosnippet`, which also change how your page looks in regular results. User-triggered fetchers may still reach public pages, so a firewall rule is the only firm block. Before writing one, verify bots by the IP lists their operators publish, such as OpenAI's [gptbot.json](https://openai.com/gptbot.json) and Perplexity's [perplexitybot.json](https://www.perplexity.ai/perplexitybot.json), because a user agent string can be faked. 3. **"I want maximum AI visibility."** Allow everything, and make sure your CDN's bot settings don't block search bots you meant to allow. Write the rules per bot name, and keep them separate from your `*` group. For outcome 1, the training part looks like this:
ROBOTS.TXT
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: CCBot
Disallow: /
RFC 9309 lets one group list several user agents, and search bots you don't name fall through to your `*` group (Amazon says Amzn-SearchBot follows the rules you give other search bots). These are the six operators we checked; other companies run crawlers of their own. A shared option is coming: the shared `__usage_ai-training` token lets one group express a training preference to companies that adopt it. It doesn't stop crawling, and wildcard rules don't count toward it. For data teams on the other side of the file, the same split applies. Read robots.txt for the user agent you actually send, and keep records of what it said when you fetched. Our posts on [whether robots.txt is legally binding](https://www.joinmassive.com/blog/is-robots-txt-legally-binding) and [ethical web scraping](https://www.joinmassive.com/blog/ethical-web-scraping) cover why that record matters. Massive's own network is built on devices whose owners opted in through the Massive SDK, and the teams using it set their own collection rules. ## Frequently Asked Questions ### Should I block AI crawlers? Block the training crawlers if you don't want your content used to train models, and allow the search crawlers if you want to be cited in AI answers. Blocking everything also costs you visibility in AI search results, and some user-triggered fetchers may not follow robots.txt anyway. ### Does blocking GPTBot remove my site from ChatGPT? No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." ### Does blocking Google-Extended remove my site from AI Overviews? No. Google says AI features in Search, including AI Overviews, are controlled through Googlebot and snippet controls. Google-Extended covers Gemini model training and Gemini Apps grounding, and does not affect Google Search. ### Does blocking AI crawlers hurt SEO? Blocking training-only controls such as Google-Extended or Applebot-Extended does not affect Google or Apple search, per both companies. Blocking AI search bots has a cost: OpenAI says sites that block OAI-SearchBot are left out of ChatGPT search answers (navigational links can still appear), and Perplexity recommends allowing PerplexityBot for a site to appear in its results. ### How do I block AI crawlers in robots.txt? Add a group per bot name, for example `User-agent: GPTBot` followed by `Disallow: /`. Each company documents its bot names on its own crawler page, linked in the sources below. ## The bottom line - "AI crawler" now means three jobs: training, search, and user-triggered fetching. Each has its own robots.txt name. - Blocking a training bot does not remove you from AI search, and blocking a search bot does not stop training. - Decide the outcome first, then write a rule per bot name. ## Sources - OpenAI, [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) - Anthropic, ["Does Anthropic crawl data from the web, and how can site owners block the crawler?"](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - Google Search Central, [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) - Google for Developers, [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) - Perplexity, [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) - Apple, [About Applebot](https://support.apple.com/en-us/119829) - Amazon, [Amazonbot](https://developer.amazon.com/amazonbot) - Common Crawl, [CCBot](https://commoncrawl.org/ccbot) - Cloudflare, ["Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large"](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), July 1, 2025 - IETF, [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), September 2022 - Massive, robots.txt check of 50 news sites, November 1, 2026 (first-party test)