OpenAI, Google, Microsoft and Anthropic back one robots.txt token to opt out of AI training

Website owners who do not want their pages used to train AI models have so far had to block crawlers one by one, from GPTBot to Google-Extended. On October 30, OpenAI, Google, Microsoft and Anthropic each said they would respect a single shared signal instead, written into the robots.txt file that sites already use to talk to crawlers.

How the token works

The mechanism is described in an Internet-Draft titled "Machine-Readable Training Preference Signal", written by Kevin Kelley of Anthropic and published on the IETF Datatracker the same day. It adds no new directives. A site creates an ordinary robots.txt group whose User-Agent line is the policy token __usage_ai-training, then uses the familiar Disallow and Allow rules: Disallow: / expresses a preference against training on the whole site, and an Allow line can carve out exceptions such as a press kit.

The draft ties the token to the AI Preferences (AIPREF) working group's definition of AI training, meaning the use of content to change the learned parameters of a generative model. A wildcard User-Agent: * group does not count, so a file without the token expresses no preference. The token is also not a crawling rule: whether a bot may fetch a page is still governed by the groups addressed to that bot.

Different timelines

Commitments vary. Kelley wrote that Anthropic has already begun implementing support, according to TokenPost. OpenAI says it will add support in the coming months, as reported by Unite.AI. Google plans to support the signal in 2027 and will treat an opt-out as a block on Google-Extended unless a site addresses Google-Extended directly, TokenPost reports. Microsoft will start honoring the new preferences on February 1, 2027, according to Search Engine Roundtable, and is adding a separate control for "grounding", the use of content to build AI answers, which leaves regular Bing rankings untouched.

What it does not settle

The draft is an individual submission: the IETF notes it is not endorsed and has no formal standing. It does not address search, retrieval or content collected before a site publishes the token, and private agreements may override it. The working group's vocabulary draft states that such preferences are not a security mechanism and that recipients choose whether to follow them. Compliance therefore rests on the companies' own commitments, with no independent verification described so far.