# AI Crawler List 2026: The AI Bots of 9 Major Operators, What They Do, and How to Verify Them

This [AI crawler](https://www.joinmassive.com/glossary/what-is-an-ai-crawler) list covers the user agents that nine organizations document for training, search, and user-triggered fetching: OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta, Microsoft, and Common Crawl. For each one we read the operator's own crawler page on November 1, 2026, and downloaded the published IP address list for each bot in our table that has one, so you can check whether a request that claims to be a given bot really comes from it.

The verification part matters because a user agent is just a text header. Common Crawl says it is "aware of crawlers falsely identifying themselves as CCBot" ([Common Crawl](https://commoncrawl.org/ccbot)). Matching the source IP against a published list, or running a reverse DNS check, is how you tell the real bot from an imitation. This is our transparency report on how open each operator is about that.

> **Key Takeaways**
>
> - The nine operators document **23 AI-related user agents and control tokens** (two of them for ads). Most split them into training, search, and user-triggered fetching.
> - **Eight of the nine publish IP lists** for their bots (seven as JSON files). We downloaded 14 lists covering about **94,000 IPv4 addresses**.
> - **User-triggered fetchers can be the largest.** OpenAI's ChatGPT-User list covers 69,280 IPv4 addresses, against 2,704 for GPTBot.
> - **Google, Apple, and Common Crawl document reverse DNS checks** on the pages we read. OpenAI, Anthropic, Perplexity, and Amazon point to IP lists, Microsoft publishes `bingbot.json`, and Meta documents user agents and robots.txt rules but links no IP list.
> - Two names on the list never crawl: **Google-Extended** and **Applebot-Extended** are robots.txt control tokens read by the operators' main crawlers.

## The AI crawler list

Purposes below are quoted or paraphrased from each operator's crawler documentation, linked in the sources. "Respects robots.txt" reflects what the operator says, not our own testing.

| Operator | User agent or token | Purpose | robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Crawls content "that may be used in training our generative AI foundation models" | Yes |
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Yes |
| OpenAI | ChatGPT-User | Fetches pages for user actions in ChatGPT | "May not apply" |
| OpenAI | OAI-AdsBot | Checks landing pages submitted as ChatGPT ads | Visits submitted pages only |
| Anthropic | ClaudeBot | Collects content that "could potentially contribute" to training | Yes |
| Anthropic | Claude-SearchBot | Indexes content for search results | Yes |
| Anthropic | Claude-User | Fetches pages when users ask Claude questions | Yes, per Anthropic |
| Google | Googlebot | Search, including AI features in Search | Yes |
| Google | Google-Extended | Token for Gemini training and Gemini Apps grounding; no separate crawler | Yes (control only) |
| Google | Google-CloudVertexBot | Crawls "requested by the site owners" for building Vertex AI Agents; rules for it have "no effect on Google Search" | Yes |
| Perplexity | PerplexityBot | Search results; "not used to crawl content for AI foundation models" | Yes |
| Perplexity | Perplexity-User | Fetches pages for user questions | "Generally ignores" |
| Apple | Applebot | Search in Spotlight, Siri, and Safari; data "may also be used" to train Apple foundation models | Yes |
| Apple | Applebot-Extended | Token to opt out of training Apple's models; "does not crawl webpages" | Yes (control only) |
| Amazon | Amazonbot | Improves products and services; "may be used to train Amazon AI models" | Yes |
| Amazon | Amzn-SearchBot | Search experiences such as Alexa; not for model training | Yes |
| Amazon | Amzn-User | Live fetches for user requests | "May not follow all" |
| Meta | Meta-ExternalAgent | Use cases "such as training foundation AI models or improving products by indexing content directly" | Yes |
| Meta | Meta-WebIndexer | Improves Meta AI search results and citations | Yes |
| Meta | Meta-ExternalFetcher | Fetches links at a user's request | "May bypass" |
| Meta | Meta-ExternalAds | Advertising and business products | Yes |
| Microsoft | Bingbot | Bing search; from February 1, 2027, Microsoft will read training and grounding preferences as it crawls | Yes, per Microsoft's [announcement](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) |
| Common Crawl | CCBot | Builds "an open repository of web crawl data" anyone can use | Yes |

Two more names are worth knowing. **`__usage_ai-training`** is the shared training opt-out token that Google, OpenAI, Microsoft, and Anthropic have committed to honor (Anthropic [has started implementing it](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/), and Microsoft starts on February 1, 2027, per its [announcement](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)); our [news post on the new robots.txt AI training opt-out](https://www.joinmassive.com/blog/robots-txt-ai-training-opt-out) explains it. And Google runs other crawlers, such as its ads-quality crawlers, that are not AI-specific and are left off this list, along with their IP file. OAI-AdsBot and Meta-ExternalAds are included because their operators list them alongside their AI bots. This report covers these nine operators; other AI bots are outside it.

## Which AI crawlers publish their IP addresses?

On November 1, 2026, we downloaded the IP list for each bot in the table above that has one and counted the IPv4 and IPv6 prefixes in each. Google's common crawlers file covers Googlebot and Google-CloudVertexBot. Amazon publishes single addresses on web pages; the others publish JSON files, each with a `creationTime` field that shows when the file was generated.

![Chart of the published IP lists for AI crawlers on November 1, 2026, by IPv4 addresses covered: OpenAI ChatGPT-User 69,280; Google common crawlers 5,088 plus 147 IPv6 prefixes; Applebot 4,752; Bingbot 4,736; GPTBot 2,704; OAI-SearchBot 2,560; Amazonbot 1,292; Anthropic 1,037; Amzn-User 1,023; Amzn-SearchBot 816; OAI-AdsBot 256; CCBot 28 plus 1 IPv6 prefix; PerplexityBot 13; Perplexity-User 10. Meta's crawler page links no IP list.](assets/ai-crawler-ip-lists.png "Published IP lists for AI crawlers, downloaded November 1, 2026.")

| File | IPv4 prefixes | IPv4 addresses | IPv6 prefixes | File date |
|---|---:|---:|---:|---|
| OpenAI GPTBot | 18 | 2,704 | 0 | 2026-09-22 |
| OpenAI OAI-SearchBot | 39 | 2,560 | 0 | 2026-01-02 |
| OpenAI ChatGPT-User | 235 | 69,280 | 0 | 2026-10-27 |
| OpenAI OAI-AdsBot | 2 | 256 | 0 | 2026-05-12 |
| Anthropic (all three bots) | 14 | 1,037 | 0 | 2026-10-20 |
| Google common crawlers | 170 | 5,088 | 147 | 2026-10-30 |
| Perplexity PerplexityBot | 13 | 13 | 0 | 2026-10-28 |
| Perplexity Perplexity-User | 10 | 10 | 0 | 2026-10-28 |
| Apple Applebot | 24 | 4,752 | 0 | 2026-09-15 |
| Amazon Amazonbot | 1,292 | 1,292 | 0 | Not dated |
| Amazon Amzn-SearchBot | 816 | 816 | 0 | Not dated |
| Amazon Amzn-User | 1,023 | 1,023 | 0 | Not dated |
| Microsoft Bingbot | 28 | 4,736 | 0 | 2024-01-03 |
| Common Crawl CCBot | 4 | 28 | 1 | 2026-08-11 |

What the numbers show:

- **Eight of nine operators publish a list.** Meta's crawler page lists user agent strings and robots.txt guidance but does not link an IP list. Google-Extended and Applebot-Extended need no list because they don't crawl.
- **Per-purpose lists make per-purpose rules possible.** OpenAI, Perplexity, and Amazon publish one list per bot, so a firewall can treat a search bot differently from a training bot. Anthropic publishes one file for its three bots, so the IP alone can't tell them apart; the user agent has to.
- **User fetchers can dwarf crawlers.** ChatGPT-User covers more than 25 times as many IPv4 addresses as GPTBot. If you allowlist AI traffic by IP, the user fetcher list is the one most likely to change your numbers.
- **Most lists are maintained.** Five of the files were generated in October 2026. The oldest JSON date is Bingbot's, January 3, 2024. The date shows when a file was generated, not whether its contents changed.

## How to verify an AI crawler

Use the method each operator documents:

1. **Match the IP against the published list.** Perplexity's docs advise matching "each User-Agent against its own IP ranges," and show a Cloudflare rule that combines the two ([Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)). Refresh the lists on a schedule, because they change.
2. **Use reverse DNS where it's documented.** Google says to run a reverse DNS lookup on the IP, check that the name ends in `googlebot.com`, `google.com`, or `googleusercontent.com` depending on the crawler, and then run a forward lookup to confirm it maps back ([Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)). Apple uses `*.applebot.apple.com`. Common Crawl runs CCBot "on dedicated IP address ranges with reverse DNS," except over IPv6.
3. **Treat the user agent as a claim, not proof.** A request that says GPTBot but comes from an address outside a current copy of `gptbot.json` is not OpenAI's crawler.

A worked example: your log shows a request with `GPTBot/1.4` in the user agent ([what GPTBot is](https://www.joinmassive.com/glossary/what-is-gptbot)). Look up its source IP in OpenAI's `gptbot.json`, which on November 1, 2026 listed 18 IPv4 prefixes. If the IP falls inside one of them, the request came from OpenAI's crawler. If not, someone borrowed the name, and it is not OpenAI's traffic.

Anthropic adds a caution for site owners who block by IP instead of robots.txt: doing so "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file" ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). The robots.txt rule is the opt-out; the IP list is for verification.

If you're deciding which of these bots to block, our myth-busting post [Should you block AI crawlers?](https://www.joinmassive.com/blog/should-i-block-ai-crawlers) looks at what 49 news sites block and what each choice costs.

## Where Massive fits

Massive is not on this list, and it isn't a crawler with a bot of its own. It is a device-access network plus a rendering stack: customers send their own requests, or have the Web Render API fetch pages for them, through consumer devices whose owners opted in through the Massive SDK. Those customers choose what they collect and which user agent they send. Our transparency posts on [how domain blocking works on Massive's network](https://www.joinmassive.com/blog/domain-blocking-transparency) and [what Massive blocks by default](https://www.joinmassive.com/blog/ethical-residential-proxy-blocked-destinations) cover the controls on our side. The habits above apply to anyone collecting web data: identify your client honestly, read robots.txt for the user agent you send, and keep a record of what it said.

## Frequently Asked Questions

### What are the main AI crawlers?

The main ones on this list are GPTBot, OAI-SearchBot, and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot, and Claude-User from Anthropic; Googlebot, Google-CloudVertexBot, and the Google-Extended token; PerplexityBot and Perplexity-User; Applebot plus Applebot-Extended; Amazonbot, Amzn-SearchBot, and Amzn-User; Meta's Meta-ExternalAgent family; Bingbot; and Common Crawl's CCBot.

### What is the user agent for ChatGPT?

ChatGPT uses OAI-SearchBot for search and ChatGPT-User for fetches users ask for. OpenAI's training crawler is GPTBot. Each has its own user agent and published IP list, and robots.txt rules may not apply to ChatGPT-User, because a person triggered the fetch.

### How do I verify that a request really comes from an AI crawler?

Check the source IP against the operator's published IP list for that bot. Google, Apple, and Common Crawl also document reverse DNS checks on their crawler pages. A user agent alone can be faked.

### Do AI crawlers respect robots.txt?

The training and search crawlers on this list say they do. Several user-triggered fetchers, including ChatGPT-User, Perplexity-User, Amzn-User, and Meta-ExternalFetcher, say they may not, because a person requested the page.

### Which AI crawlers don't publish IP addresses?

Of the nine operators here, Meta's crawler page does not link an IP list. Google-Extended and Applebot-Extended don't crawl, so they have no addresses of their own.

## The bottom line

- Nine operators document 23 AI user agents and tokens, mostly split by training, search, and user-triggered fetching.
- Eight publish IP lists, and those lists, plus reverse DNS where documented, are how you verify a bot.
- robots.txt is the opt-out; the IP lists are for checking who is actually knocking.

## Sources

- OpenAI, [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- Anthropic, ["Does Anthropic crawl data from the web, and how can site owners block the crawler?"](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- Google for Developers, [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) and [Verifying Googlebot and other Google crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- Perplexity, [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- Apple, [About Applebot](https://support.apple.com/en-us/119829)
- Amazon, [Amazonbot](https://developer.amazon.com/amazonbot)
- Meta for Developers, [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Microsoft, [bingbot.json](https://www.bing.com/toolbox/bingbot.json)
- Google Search Central Blog, ["Supporting a shared AI training control for websites"](https://developers.google.com/search/blog/2026/10/ai-training-control), October 30, 2026
- OpenAI, ["Making it easier for websites to express AI training preferences"](https://openai.com/index/helping-websites-express-ai-training-preferences/), October 30, 2026
- IETF ai-control mailing list, [Kevin Kelley, "New draft: draft-kelley-aipref-training-preference-signal-00"](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/), October 30, 2026
- Krishna Madhavan, Microsoft Bing Blogs, ["Your content, your choice: clearer preferences for training and grounding"](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), October 30, 2026
- Common Crawl, [CCBot](https://commoncrawl.org/ccbot)
- Massive, download and count of 14 published crawler IP lists, November 1, 2026 (first-party test)