Google announced on October 30 that it plans to support a shared control for AI training in robots.txt, letting website owners state their training preferences once instead of writing separate instructions for every AI provider. The approach is based on work from the AIPREF working group at the Internet Engineering Task Force, and Google aims to support the new control alongside other AI providers in 2027. The company said existing controls, including Google-Extended, will keep working beside the shared signal.
One token instead of a list of instructions
Today, a publisher that wants to limit AI training must manage instructions developer by developer, learning each provider’s control and keeping them current. The shared approach replaces that patchwork with a single policy token in robots.txt, setting broad training preferences across participating providers in one step.
The key detail is how Google will read an opt-out. When a website uses the policy token to opt out of training, and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended. Other providers can handle the token similarly, so a site can opt out across providers, or pick and choose.
What does not change
Company-specific controls will continue to function alongside the new joint signal, so the shared token adds a broad option rather than replacing fine-grained choices. A website owner can express a general preference through the industry signal while still tailoring instructions for one developer, such as Google, through Google-Extended.
The distinction also affects visibility in AI answers. Websites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks through grounding in Gemini Apps, Google said. In other words, the training choice and the grounding choice are related but not identical, and publishers should check which permission they are actually setting before they publish the token.
Why publishers asked for this
Google said the draft proposal reflects feedback from publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training. The new mechanism gives website owners the IETF’s AI training definition now, while multistakeholder discussions continue on the rest of the AIPREF vocabulary.
The timing is tied to how far AI answers have spread. With AI Overviews now appearing on more than 80% of searches, publishers face a different question than when AI training controls were a niche concern. A single, widely recognised signal is easier to adopt than a growing list of provider-specific directives, especially for smaller sites.
Google also framed the proposal as consistent with the European Union’s Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary method for expressing machine-readable preferences. Support across major AI providers, Google argued, could turn it into a cross-industry standard that is robust and durable, rather than overlapping company policies.
Why It Matters
For publishers, the practical gain is less busywork and fewer mistakes. One token that several providers honour is harder to get wrong than a dozen separate settings, and a mistake in either direction is costly: too open, and content trains models the publisher wanted to exclude; too closed, and the site may lose visibility it wanted to keep.
Publishers are weighing stronger responses to AI search. Reddit is moving toward a logged-in experience as AI search traffic falls, a far more drastic step than a robots.txt token. A shared opt-out gives publishers a middle path, but it is not a complete answer: the control is still a draft, and its value depends on how many providers honour it.
Background
Google has offered Google-Extended as its AI training control since 2023, and it also provides a Search Console method for opting out of Search generative AI features. Bing, OpenAI and Anthropic announced adoption of the same shared proposal on the same day, and Google said it will provide updates on its progress as the work moves through the IETF.
Source: Google announcement and Roundtable coverage.
0 Comments