Google, Microsoft and OpenAI have announced plans to support a common mechanism that would allow website owners and publishers to restrict the use of their content for training generative artificial intelligence models through a single instruction in their robots.txt files.
The proposed system would enable publishers to communicate their AI training preferences to multiple participating companies without having to configure separate restrictions for each provider.
The mechanism is based on an Internet Draft submitted to the Internet Engineering Task Force's AI Preferences working group by Anthropic's Kevin Kelley. The proposal introduces a shared identifier, "__usage_ai-training", which websites can use alongside existing Allow and Disallow instructions.
Under the proposed framework, publishers could restrict AI training across an entire website while permitting access to selected sections. However, the instruction would not prevent web crawlers from accessing content, and it would not automatically control how material is used to generate AI responses.
The proposal is still under development and has not been adopted as an official internet standard. Google, Microsoft and OpenAI have outlined different implementation timelines, with none of the three supporting the shared instruction at the time of the announcement.
Google plans to introduce support in early 2027. The company said it would interpret the shared opt-out as a restriction under Google-Extended when a website has not specified separate Google-Extended instructions.
Google-Extended governs the use of website content for training future Gemini models and grounding responses in Gemini applications and Vertex AI. The company clarified that the control does not affect a website's inclusion in Google Search.
Microsoft, meanwhile, plans to introduce separate controls for AI training and grounding. Its approach would allow publishers to distinguish between using their content to train models and using it to support AI-generated responses.
The company expects to begin honouring the new preferences on February 1, 2027. It will stop recognising its existing NoArchive and NoCache tags as AI training or grounding restrictions after March 1, 2027.
OpenAI has indicated that it intends to support the shared instruction within the next few months, while continuing to recognise its existing robots.txt controls.
Anthropic has also begun development work related to the proposal, although a timeline for full implementation has not been confirmed.
The initiative comes as publishers and digital businesses seek greater control over how their online content is used by AI companies.
For website operators, a common instruction could simplify the management of AI training permissions. However, differences in how companies interpret training and grounding restrictions mean publishers may still need provider-specific controls.
Further technical guidance is expected as participating companies prepare to implement the proposed system.
Disclaimer: This article may include information derived from interviews, press releases, public statements, research, company communications and other publicly available or third-party sources. Such material may be summarised, paraphrased or contextualised for journalistic and editorial purposes. All rights in third-party content remain with their respective owners.