Website owners are getting a single robots.txt AI training opt-out that is meant to work across several providers, instead of having to block every AI crawler one by one. On October 30, 2026, OpenAI, Google and Microsoft announced that they will honor a shared signal called __usage_ai-training. It is based on an Internet-Draft submitted to the IETF the same day by an Anthropic engineer. Nothing is in effect yet: the providers plan to implement the signal over the coming months and into 2027.
Here is what the new rule looks like, what it does and does not do, which dates matter and what operators of blogs, shops and company sites should do now.
Key Takeaways
- The new policy token
__usage_ai-traininggoes into a robots.txt User-Agent line like a crawler name.Disallow: /then expresses a preference against AI training for the whole site. - The rule governs use for training, not crawling. To stop pages from being fetched, operators still need to block individual crawlers.
- OpenAI aims to implement the signal “in the coming months”, Google in 2027, Microsoft starting February 1, 2027.
- The IETF document is an individual draft with no formal standing as a standard. Compliance is voluntary.
- Microsoft is also adding a page-level preference for “grounding”, the use of content to inform AI answers.
How the robots.txt AI training rule works
The draft “Machine-Readable Training Preference Signal” by Kevin Kelley (Anthropic) adds no new directives and no new grammar. It defines a so-called policy token that sits in a User-Agent group just like an ordinary crawler name. The draft’s own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This expresses a preference against AI training for every path except /press-kit/, where training is explicitly allowed. Path matching and longest-match precedence work exactly as in RFC 9309, the robots.txt standard. Existing parsers that do not know the token simply ignore the group, like any unknown crawler.
Three rules in the draft are worth knowing:
- Wildcards do not count. A
User-Agent: *group must not be interpreted as matching the policy token. A site that blocks everything with a wildcard today has said nothing about AI training. - No group, no preference. If there is no
User-Agent: __usage_ai-trainingline, no preference has been expressed. - Not a crawling directive. The draft states plainly that the token is not a directive for crawling behavior. A provider may still fetch a page, for example for search, but should not train on it.
Precedence rules apply as well: specific expressions should take precedence over general ones, providers may treat their own tokens as more specific than the shared signal, and private agreements such as licensing deals may override preferences expressed in robots.txt.
The definition of “AI training” comes from the IETF AI Preferences working group vocabulary: using an asset to modify the learned parameters of a generative AI model. Models used exclusively for non-generative tasks such as classification or ranking are not covered.
What OpenAI, Google and Microsoft have committed to
OpenAI says it will honor the shared signal for foundational generative AI training and aims to implement it “in the coming months”, with details to follow. Its existing crawler controls, such as GPTBot rules, explicitly remain in force.
Google says on its Search Central Blog that it aims to support the signal along with other providers in 2027. If a site opts out via the policy token and does not explicitly specify Google-Extended, Google plans to treat that as if Google-Extended had been disallowed. Google-Extended has been Google’s own AI training control since 2023. Notably, Google says sites that explicitly allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. Operators who set the shared token but still want links in Gemini answers should therefore allow Google-Extended explicitly.
Microsoft is the most specific. Starting February 1, 2027, Bing will honor the new preferences as it crawls. Beyond robots.txt, training can be controlled per page via a meta tag or HTTP header, for example <meta name="Bingbot" content="ai-training:disallow">. The page-level setting overrides robots.txt.
According to The Verge, Meta also signaled support without saying whether it will honor the token. We did not find a separate announcement on Anthropic’s own newsroom. Reports quote Kelley as saying Anthropic has already started implementing support; we could not verify that independently.
Microsoft separates training from grounding
Microsoft goes beyond the shared signal with a separate, page-level-only preference for “AI grounding”, meaning the use of content to inform AI-generated responses in Bing, Copilot or Microsoft 365. The options are Allow (the default), Disallow and a new DisallowExtended, which limits grounding to Microsoft-operated experiences and excludes grounding services Microsoft provides to third parties, such as grounding through API access in Foundry.
The new settings replace the NoArchive and NoCache tags that publishers have used since 2023 to restrict training and grounding at Microsoft. After March 1, 2027, Microsoft will no longer read those tags as training or grounding preferences. Sites using them should publish the new settings by then and, for grounding, also record a temporary preference in Bing Webmaster Tools, which is set to open on December 15, 2026. Microsoft stresses that none of the settings affect eligibility for traditional Bing search results and states plainly that neither choice guarantees inclusion, traffic or payment.
The limits of the signal
The simplification is useful, but operators should understand its limits:
- It is voluntary: robots.txt is a coordination mechanism, not an access control. Crawlers that ignore it are not stopped by the token.
- It is not retroactive: a new entry does not remove content already in datasets or trained models.
- It is a draft: the IETF datatracker notes that an individual Internet-Draft is not endorsed by the IETF and has no formal standing. Details and even the token name could still change.
- Timelines differ: as things stand, months will pass before all major providers actually evaluate the signal.
Google points out that the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary method for machine-readable preferences. Whether the token counts as a valid rights reservation under European copyright law is not settled by that, and publishers should seek legal advice if in doubt.
What website operators should do now
If you do not want your content used for AI training, you can add the group today. Parsers that do not know the token will ignore it, so there is no downside. Keep existing rules for individual crawlers such as GPTBot, ClaudeBot or Google-Extended in place until providers actually evaluate the shared signal. One possible combination:
User-Agent: __usage_ai-training
Disallow: /
User-Agent: GPTBot
Disallow: /WordPress users usually manage robots.txt through an SEO plugin or a physical file in the web root. After every change, check the file actually served at /robots.txt, because caching plugins and CDNs sometimes deliver stale versions. If you need a refresher on how meta tags in the HTML head work, which Microsoft uses for its page-level settings, see our HTML basics guide.
If you want to stay visible in AI answers, weigh the options carefully: at Google, links in Gemini answers depend on Google-Extended, at Microsoft on the grounding setting. A blanket block can cost visibility. How quickly AI agents reach into web content, and the conflicts that creates, was illustrated recently by the Wikidata incident involving OpenAI agents.
Conclusion
The shared token makes opting out of AI training via robots.txt much simpler, and the commitments from OpenAI, Google and Microsoft give it weight. It remains a voluntary preference without technical enforcement, and its rollout stretches into 2027. Details are in the IETF draft, on the Google Search Central Blog and on the Bing blog.
