# The New robots.txt AI Training Opt-Out: What Google, OpenAI, Microsoft and Anthropic Agreed To
On October 30, 2026, Google, OpenAI, Microsoft and Anthropic each said they will honor one shared AI training opt-out in robots.txt. A site adds a group for the policy token `__usage_ai-training` and lists the paths it does not want used to train generative AI models. One group can stand in for a separate training opt-out per company, at least for the companies that adopt it.
The token comes from an Internet-Draft written by Anthropic's Kevin Kelley, "Machine-Readable Training Preference Signal" ([IETF Datatracker](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)). It is a draft, not a standard, and none of the four companies honors it yet: Microsoft starts on February 1, 2027, Google in early 2027, OpenAI "in the coming months," and Anthropic says it has started implementing it. This post covers how the token works, what it does not do, how each company will treat it, and what we found when we checked 49 news sites two days after the announcement.
> **Key Takeaways**
>
> - The opt-out is an ordinary robots.txt group: `User-Agent: __usage_ai-training` followed by `Disallow` and `Allow` rules. It adds no new syntax.
> - It is a **preference about use, not a crawl block.** The draft says the token "is not a directive for crawling behavior."
> - A `User-Agent: *` rule **does not count.** A file with no `__usage_ai-training` group "expresses no preference at all," in the draft author's words.
> - Microsoft will honor it from **February 1, 2027** and stop reading NoArchive and NoCache as training or grounding preferences after **March 1, 2027**.
> - On November 1, 2026, **0 of 49** news sites we checked used the token. **10 of the 33** homepages we could load still carry NoArchive or NoCache tags.
## What did Google, OpenAI, Microsoft and Anthropic announce?
Each company published its own statement on October 30, 2026, pointing to the same Internet-Draft, an individual submission written for the discussion in the IETF AI Preferences (AIPREF) working group:
- **Google** said it will "support a shared AI training control in robots.txt" and aims to do so "along with other AI providers in 2027" ([Google Search Central Blog](https://developers.google.com/search/blog/2026/10/ai-training-control)).
- **OpenAI** said it "will honor a common signal" and aims "to implement the signal in the coming months." Its existing crawler controls "will continue to apply" ([OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/)).
- **Microsoft** said it will honor the shared preference "starting February 1, 2027," and added a separate preference for grounding ([Bing Blogs](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)).
- **Anthropic**, whose engineer wrote the draft, said on the IETF mailing list that it "has already started implementing support," adding that "it'll take us some time to add support across our services" ([IETF ai-control list](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/)).
The draft itself is an individual submission dated October 30, 2026, which expires on May 3, 2027 unless it is revised. Internet-Drafts "may be updated, replaced, or obsoleted by other documents at any time," as the draft's own boilerplate says. Microsoft put it plainly: "the efforts of the IETF AI Preferences Working Group continue toward a final standard."
## How do you block AI training bots with robots.txt now?
You write a normal group whose user-agent line is the policy token. This is the draft's own example:
It reads as a preference against AI training for every path except `/press-kit/`, where it expresses a preference in favor. Path matching works exactly as in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the robots.txt standard, including longest-match precedence. Parsers that don't know the token ignore the group, the same way they ignore any user agent they don't recognize.
The token's meaning is borrowed from the working group's vocabulary draft. "AI Training" means "using an asset to modify the learned parameters of a generative AI model." Training a model used only for non-generative tasks, such as classification or ranking, is not included ([draft section 3](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)).
Until the companies ship support, the per-crawler rules still do the work. For three of the four companies, those are GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google). Microsoft has no separate training crawler: its 2023 controls were the NoArchive and NoCache tags read by Bingbot.
## What the shared opt-out does not do
The draft is narrow on purpose. Five limits matter more than the syntax:
1. **It does not stop crawling.** "This policy token is not a directive for crawling behavior," the draft says (section 4.4). A crawler MAY use the preference to skip pages it could not use. If you want fewer requests, you still need rules for each crawler's own name.
2. **Wildcard rules don't count.** `User-Agent: *` groups "MUST NOT be interpreted as matching the policy token" (section 4.2). A site that blocks every crawler with `Disallow: /` under `*` has expressed no training preference under this scheme.
3. **No group means no preference.** If the file has no `__usage_ai-training` line, "no preference has been expressed." Microsoft's version is explicit: both training and grounding "default to Allow when no applicable preference is published."
4. **Private deals win.** "Private agreements MAY override preferences expressed via this mechanism" (section 4.5). A licensing deal can permit what the file says no to.
5. **It looks forward only.** Microsoft says the preferences cover "future AI generative model training," and that they "do not establish licensing or payment arrangements." Nothing in the draft deals with data already collected.
One rule in the draft affects anyone who keeps data: the preferences that apply to a page "are those expressed in the robots.txt file in effect when the asset was retrieved." A crawler that wants to show it followed the signal needs to record what robots.txt said on the day it fetched each page.
## How will each company apply the new opt-out?
The announcements agree on the token and differ on the details.
| Company | Start date | How the token maps to existing controls | Separate grounding control |
|---|---|---|---|
| Google | Early 2027 | A token opt-out, with Google-Extended not specified, counts as disallowing Google-Extended | Google-Extended already covers Gemini Apps grounding |
| Microsoft | February 1, 2027 | Replaces NoArchive and NoCache for training and grounding; page-level override | Yes, page-level `ai-grounding` with Disallow or DisallowExtended |
| OpenAI | "Coming months" | Complements GPTBot, which "will continue to apply" | Existing controls, including OAI-SearchBot, unchanged |
| Anthropic | Started implementing | "Complements our existing crawler controls rather than replacing them" | Existing controls, including Claude-SearchBot, unchanged |
Google's mapping has a side effect worth knowing. Google's crawler documentation, which puts its support for the shared token in "early 2027," says Google-Extended covers both training of future Gemini models and "grounding" in Gemini Apps and Vertex AI ([Google crawler docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)). Google's announcement adds that "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So a site that uses only the shared token may also leave Gemini Apps grounding. Google says Google-Extended "does not impact a site's inclusion in Google Search."
Microsoft's grounding control is the most detailed. `ai-grounding:disallow` keeps a page out of generated answers, and `ai-grounding:disallowextended` allows "Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365," but excludes grounding services Microsoft sells to other companies. Microsoft warns that either setting will "significantly reduce your content's visibility in AI-generated experiences," without affecting Bing search eligibility. Page-level preferences use a meta tag or header:
## What we found on 49 news sites
On November 1, 2026, we requested the robots.txt files and homepages of the same 50 news sites from our [EDPB web scraping guidelines](https://www.joinmassive.com/blog/edpb-web-scraping-guidelines) survey. 49 returned a valid robots.txt (Politico refused the request). We applied RFC 9309 group matching to each file and searched each homepage for Microsoft's legacy tags.
- **0 of 49** files contain the `__usage_ai-training` token or any other `__usage_` token. That is expected two days after a draft, but it sets the baseline.
- **17 of 49** block all four training crawlers we checked by name (GPTBot, ClaudeBot, Google-Extended, and Common Crawl's CCBot), and **40** block at least one by name. CCBot is not covered by any of the announcements. These sites already say no to training crawler by crawler. The token would let them say it once, for the companies that adopt it.
- **3 of 49** (Reuters, The Times, and The Wall Street Journal) block every crawler they don't list through a site-wide `User-Agent: *` rule. Under the draft, that wildcard says nothing about training, although it still blocks compliant crawlers from fetching pages.
- **Bingbot** has no group of its own and no site-wide wildcard block in 43 of the 49 files, as you'd expect for a search crawler.
- **10 of the 33 homepages** that loaded (16 refused the request or failed to respond) carry a `noarchive` or `nocache` directive. Five address Bingbot by name: the Guardian and the BBC in an `X-Robots-Tag` header, and The Verge, Vox, and HuffPost in a meta tag. The Chicago Tribune, NPR, Le Monde, Der Spiegel, and the Daily Mail use a general `noarchive`.
Those last 10 sites are the ones with a deadline. Microsoft says that after March 1, 2027, "NoArchive and NoCache will no longer be read as a preference for training or grounding." A site that relied on them must publish the new training preference, and set grounding preferences page by page, to keep the same choice with Microsoft. To keep grounding withheld, Microsoft says those sites must also record a temporary grounding preference in Bing Webmaster Tools before March 1, 2027 (it opens December 15, 2026, for sites using NoArchive or NoCache on that date). Otherwise those pages "default to Allow for grounding" until Microsoft recrawls them. The temporary preference lasts until a recrawl or September 30, 2027.
There is a catch for anyone who adds the new token early. Microsoft says: "If we detect new AI Training or AI Grounding preferences on a site, we will disregard any NoArchive and NoCache preferences that may remain on that site." A site that adds a `__usage_ai-training` group and relied on NoArchive to keep pages out of Copilot answers should publish page-level `ai-grounding` preferences at the same time.
## What should publishers and data teams do now?
For **publishers**, the token costs one group and changes nothing for crawlers that ignore it:
1. Add a `__usage_ai-training` group that matches your actual preference, including any paths you want used, such as a press kit.
2. Keep your named crawler rules. They control fetching, and the companies say they will keep honoring them.
3. If you use NoArchive or NoCache, put March 1, 2027 on the calendar, and set your Microsoft grounding preference when you add the training group, not after, because a new preference makes Microsoft ignore the old tags.
For **teams that collect web data** for models, the draft points to three habits worth adopting before it is final:
1. **Parse the token.** Treat a disallowed path as a training preference, separate from crawl rules.
2. **Log robots.txt per fetch.** Preferences attach to the file "in effect when the asset was retrieved," so store it with the date.
3. **Keep uses apart.** Data collected for search, monitoring, or price tracking is a different use from training, and the signals now say so.
Massive is a device-access network, not a crawler with its own policy token: the teams using it decide what to collect and what they train on. We covered the legal side of these signals in [Is robots.txt Legally Binding?](https://www.joinmassive.com/blog/is-robots-txt-legally-binding) and the market for licensed training data in [AI Data Licensing Deals in 2026](https://www.joinmassive.com/blog/ai-data-licensing-deals).
## Frequently Asked Questions
### What is __usage_ai-training in robots.txt?
It is a policy token defined in an IETF Internet-Draft from October 30, 2026. A robots.txt group that names it in its user-agent line states a preference for or against using the listed paths to train generative AI models. Google, OpenAI, Microsoft, and Anthropic have said they will honor it.
### Does the new opt-out stop AI crawlers from visiting my site?
No. The draft says the token "is not a directive for crawling behavior." To limit fetching, keep rules for each crawler's own user agent, such as GPTBot or ClaudeBot.
### Does User-agent: * Disallow: / opt me out of AI training?
Not under this mechanism. The draft says wildcard groups must not be read as matching the policy token, so a file without a `__usage_ai-training` group expresses no training preference.
### When does the shared AI training opt-out take effect?
Microsoft starts honoring it on February 1, 2027. Google aims to support it in early 2027, OpenAI "in the coming months," and Anthropic says it has started implementing it. Until then, use each company's own controls.
### Will opting out of AI training hurt my search rankings?
Microsoft says its training and grounding preferences don't change eligibility for Bing search results, and Google says Google-Extended doesn't affect inclusion in Google Search. Opting out of grounding can reduce how often your pages appear in AI-generated answers.
## The bottom line
- One robots.txt group, `User-Agent: __usage_ai-training`, will express an AI training preference to four major AI companies, starting in 2027.
- It governs use, not access, and only explicit groups count. Wildcard rules and silence express no preference.
- Sites using NoArchive or NoCache for Microsoft have until March 1, 2027 to switch.
## Sources
- IETF Datatracker, [draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal"](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/), October 30, 2026
- IETF ai-control mailing list, [Kevin Kelley, "New draft: draft-kelley-aipref-training-preference-signal-00"](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/), October 30, 2026
- Google Search Central Blog, ["Supporting a shared AI training control for websites"](https://developers.google.com/search/blog/2026/10/ai-training-control), October 30, 2026
- OpenAI, ["Making it easier for websites to express AI training preferences"](https://openai.com/index/helping-websites-express-ai-training-preferences/), October 30, 2026
- Krishna Madhavan, Microsoft Bing Blogs, ["Your content, your choice: clearer preferences for training and grounding"](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), October 30, 2026
- Google for Developers, [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), accessed November 1, 2026
- IETF, [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), September 2022
- Massive, robots.txt and homepage check of 50 news sites, November 1, 2026 (first-party test)