Google, OpenAI, Microsoft, Anthropic back AI opt-out rule

Four of the largest AI developers will honor a single robots.txt signal that lets websites refuse the use of their content for generative AI training. Publishers will no longer need to block each company's crawler one by one. Google Alphabet Inc. ↘ 1.94%, OpenAI, Anthropic and Microsoft ↘ 3.58% committed on Friday, October 30, to support a shared "policy token" proposed at the Internet Engineering Task Force, The Verge reported, citing earlier reporting by Search Engine Roundtable.theverge.com

The companies will switch it on at different times. Anthropic has already started supporting the directive, Microsoft will begin on February 1, 2027, Google plans to follow in early 2027 and OpenAI has promised to implement it in the coming months, according to Engadget.engadget.com

How the signal works

The proposal comes from Kevin Kelley, Anthropic's technical lead for standards. Site owners add a robots.txt group with "__usage_ai-training" as the user agent, followed by Disallow rules, and can use Allow lines to open specific folders, such as a press kit, for training. The token covers training only. It is not a crawling directive, and it does not change search indexing.engadget.com

In a post on OpenAI's website, the company called it a "voluntary effort" and said the new signal "will not replace the controls we already provide through robots.txt." Google's John Mueller said websites will be able to use a shared token "to set broad preferences across participating AI providers." He added that if a site opts out with the token and does not name Google-Extended, Google will treat that as a Google-Extended block, according to Search Engine Land.openai.com+1

Microsoft adds grounding controls

Microsoft went further than the other three. Bing will offer separate settings for AI training and AI grounding, which covers the use of pages to generate AI answers. A "Limited Grounding" option will let sites support Microsoft's own AI products while refusing its third-party grounding services. Microsoft also set a timeline that runs from temporary controls on December 15, 2026, to a March 1, 2027, deadline for sites still using older NoArchive or NoCache tags. According to Bing's blog, none of these choices affects eligibility for regular search results.seroundtable.com+1

Limits of the pledge

The directive cannot undo past collection, because content already gathered stays in the companies' hands. It also does not stop crawlers from fetching pages, and compliance remains voluntary. Meta ↘ 3.96% said it supports the idea but has not committed to implementing it. The draft itself remains a work-in-progress proposal from the IETF's AI Preferences working group rather than a finished standard.zglg.work+3