AI Crawler List 2026: The AI Bots of 9 Major Operators, What They Do, and How to Verify Them

This AI crawler list covers the user agents that nine organizations document for training, search, and user-triggered fetching: OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Meta, Microsoft, and Common Crawl. For each one we read the operator's own crawler page on November 1, 2026, and downloaded the published IP address list for each bot in our table that has one, so you can check whether a request that claims to be a given bot really comes from it.

The verification part matters because a user agent is just a text header. Common Crawl says it is "aware of crawlers falsely identifying themselves as CCBot" (Common Crawl). Matching the source IP against a published list, or running a reverse DNS check, is how you tell the real bot from an imitation. This is our transparency report on how open each operator is about that.

Key Takeaways

• The nine operators document 23 AI-related user agents and control tokens (two of them for ads). Most split them into training, search, and user-triggered fetching.
• Eight of the nine publish IP lists for their bots (seven as JSON files). We downloaded 14 lists covering about 94,000 IPv4 addresses.
• User-triggered fetchers can be the largest. OpenAI's ChatGPT-User list covers 69,280 IPv4 addresses, against 2,704 for GPTBot.
• Google, Apple, and Common Crawl document reverse DNS checks on the pages we read. OpenAI, Anthropic, Perplexity, and Amazon point to IP lists, Microsoft publishes bingbot.json, and Meta documents user agents and robots.txt rules but links no IP list.
• Two names on the list never crawl: Google-Extended and Applebot-Extended are robots.txt control tokens read by the operators' main crawlers.

The AI crawler list

Purposes below are quoted or paraphrased from each operator's crawler documentation, linked in the sources. "Respects robots.txt" reflects what the operator says, not our own testing.

Two more names are worth knowing. __usage_ai-training is the shared training opt-out token that Google, OpenAI, Microsoft, and Anthropic have committed to honor (Anthropic has started implementing it, and Microsoft starts on February 1, 2027, per its announcement); our news post on the new robots.txt AI training opt-out explains it. And Google runs other crawlers, such as its ads-quality crawlers, that are not AI-specific and are left off this list, along with their IP file. OAI-AdsBot and Meta-ExternalAds are included because their operators list them alongside their AI bots. This report covers these nine operators; other AI bots are outside it.

Which AI crawlers publish their IP addresses?

On November 1, 2026, we downloaded the IP list for each bot in the table above that has one and counted the IPv4 and IPv6 prefixes in each. Google's common crawlers file covers Googlebot and Google-CloudVertexBot. Amazon publishes single addresses on web pages; the others publish JSON files, each with a creationTime field that shows when the file was generated.

What the numbers show:

• Eight of nine operators publish a list. Meta's crawler page lists user agent strings and robots.txt guidance but does not link an IP list. Google-Extended and Applebot-Extended need no list because they don't crawl.
• Per-purpose lists make per-purpose rules possible. OpenAI, Perplexity, and Amazon publish one list per bot, so a firewall can treat a search bot differently from a training bot. Anthropic publishes one file for its three bots, so the IP alone can't tell them apart; the user agent has to.
• User fetchers can dwarf crawlers. ChatGPT-User covers more than 25 times as many IPv4 addresses as GPTBot. If you allowlist AI traffic by IP, the user fetcher list is the one most likely to change your numbers.
• Most lists are maintained. Five of the files were generated in October 2026. The oldest JSON date is Bingbot's, January 3, 2024. The date shows when a file was generated, not whether its contents changed.

How to verify an AI crawler

Use the method each operator documents:

1. Match the IP against the published list. Perplexity's docs advise matching "each User-Agent against its own IP ranges," and show a Cloudflare rule that combines the two (Perplexity). Refresh the lists on a schedule, because they change.
2. Use reverse DNS where it's documented. Google says to run a reverse DNS lookup on the IP, check that the name ends in googlebot.com, google.com, or googleusercontent.com depending on the crawler, and then run a forward lookup to confirm it maps back (Google). Apple uses *.applebot.apple.com. Common Crawl runs CCBot "on dedicated IP address ranges with reverse DNS," except over IPv6.
3. Treat the user agent as a claim, not proof. A request that says GPTBot but comes from an address outside a current copy of gptbot.json is not OpenAI's crawler.

A worked example: your log shows a request with GPTBot/1.4 in the user agent (what GPTBot is). Look up its source IP in OpenAI's gptbot.json, which on November 1, 2026 listed 18 IPv4 prefixes. If the IP falls inside one of them, the request came from OpenAI's crawler. If not, someone borrowed the name, and it is not OpenAI's traffic.

Anthropic adds a caution for site owners who block by IP instead of robots.txt: doing so "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file" (Anthropic). The robots.txt rule is the opt-out; the IP list is for verification.

If you're deciding which of these bots to block, our myth-busting post Should you block AI crawlers? looks at what 49 news sites block and what each choice costs.

Where Massive fits

Massive is not on this list, and it isn't a crawler with a bot of its own. It is a device-access network plus a rendering stack: customers send their own requests, or have the Web Render API fetch pages for them, through consumer devices whose owners opted in through the Massive SDK. Those customers choose what they collect and which user agent they send. Our transparency posts on how domain blocking works on Massive's network and what Massive blocks by default cover the controls on our side. The habits above apply to anyone collecting web data: identify your client honestly, read robots.txt for the user agent you send, and keep a record of what it said.

The bottom line

• Nine operators document 23 AI user agents and tokens, mostly split by training, search, and user-triggered fetching.
• Eight publish IP lists, and those lists, plus reverse DNS where documented, are how you verify a bot.
• robots.txt is the opt-out; the IP lists are for checking who is actually knocking.

Sources

• OpenAI, Overview of OpenAI Crawlers
• Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
• Google for Developers, Google's common crawlers and Verifying Googlebot and other Google crawlers
• Perplexity, Perplexity Crawlers
• Apple, About Applebot
• Amazon, Amazonbot
• Meta for Developers, Meta Web Crawlers
• Microsoft, bingbot.json
• Google Search Central Blog, "Supporting a shared AI training control for websites", October 30, 2026
• OpenAI, "Making it easier for websites to express AI training preferences", October 30, 2026
• IETF ai-control mailing list, Kevin Kelley, "New draft: draft-kelley-aipref-training-preference-signal-00", October 30, 2026
• Krishna Madhavan, Microsoft Bing Blogs, "Your content, your choice: clearer preferences for training and grounding", October 30, 2026
• Common Crawl, CCBot
• Massive, download and count of 14 published crawler IP lists, November 1, 2026 (first-party test)