AI Training Opt-Out: What Website Publishers Need to Know

For years, publishers have faced a fragmented choice: allow or restrict each artificial-intelligence company’s crawler separately, often without a clear way to distinguish training from other uses of their content. That may soon become easier. Google, Microsoft, OpenAI and Anthropic have all signalled support for a shared website preference that could let publishers express an AI-training choice through one machine-readable rule in their robots.txt file.

The proposal is important for independent newsrooms, specialist publishers, bloggers and businesses that invest in original reporting and expertise. But it is not a magic switch, and it is not yet a fully deployed global standard. The timing, technical scope and differences between providers matter.

What you need to know: A shared token called __usage_ai-training is being proposed for robots.txt. It is designed to communicate whether site content may be used to train generative AI models. Provider support is planned on different timelines, so publishers should not assume the rule is already being enforced everywhere.

What has changed?

On 30 October 2026, Google announced plans to support a shared AI-training control in robots.txt, building on work by the Internet Engineering Task Force’s AI Preferences (AIPREF) working group. Microsoft and OpenAI also announced support for a common signal. Anthropic’s standards lead said the company had begun implementing the proposal, while noting that support across its services would take time.

The proposed policy token is __usage_ai-training. Instead of maintaining a separate training instruction for every participating AI provider, a website could publish a single policy group that participating systems recognise.

This is a meaningful change in direction. Until now, a publisher seeking to express preferences has generally had to learn the crawler names and controls used by individual companies. A shared signal could make the process easier to manage and more consistent.

How the shared robots.txt rule is expected to work

A robots.txt file sits at the root of a website and communicates instructions to automated crawlers. The proposed signal uses a dedicated policy group to describe a website’s preference about AI training. A simplified example from the proposal is:

User-Agent: __usage_ai-training
Disallow: /

This expresses a site-wide preference against using the covered content for AI training. A publisher could also use path-specific rules to treat selected sections differently. For example, it might disallow training across most of the site while allowing a designated press-kit directory.

There is an important technical distinction: the token is a policy identifier, not the name of a crawler. It is intended to communicate a use preference. It does not, by itself, prevent a bot from visiting or downloading a page.

AI training is not the same as search indexing

One of the most important points for publishers is that AI training, crawling, search indexing and AI-generated answer features are not interchangeable.

Crawling: an automated system visits and retrieves a page.
Search indexing: a search engine processes content so it can potentially appear in search results.
AI training: content is used to modify the learned parameters of a generative AI model.
AI grounding: content is retrieved or referenced to help produce an answer at the time a user asks a question.

The proposed shared token is focused on AI training. It should not be treated as a universal block on crawling, indexing, search visibility or every form of AI use. Some providers have separate controls for other activities. Google, for example, says its existing Google-Extended control continues to matter, and that its handling of the new shared opt-out will interact with that control.

What each major AI company has said

Although the companies are aligned around a shared direction, their implementation plans are not identical.

Company	Position announced	Timing indicated
Google	Plans to support the shared signal; explains how it will relate to Google-Extended.	During 2027.
Microsoft	Plans to honour the shared AI-training preference and is introducing separate controls for AI grounding.	From 1 February 2027; transition from older controls is also planned.
OpenAI	Says it will honour a common robots.txt signal.	In the coming months; no exact date stated.
Anthropic	Its standards lead says implementation has started.	No complete rollout date stated.

These are announced intentions and implementation plans, not proof that every service already recognises the token. Publishers should check each provider’s current crawler documentation before making decisions based on the rule.

Why this matters to independent publishers

Original journalism and specialist publishing require investment: reporters, editors, photographers, researchers, hosting, distribution and fact-checking all cost money. When content is collected at scale and used to develop commercial AI systems, publishers want meaningful choices about how their work is used.

A shared signal could reduce the technical burden on smaller organisations. A newsroom without a dedicated infrastructure team may find it difficult to track every crawler name, policy update and exception. A common mechanism would make it easier to express a preference in one place, provided the major services implement it consistently.

It could also encourage better conversations about licensing and attribution. A machine-readable preference is not a replacement for commercial agreements, copyright law or negotiations over access to valuable archives. It is one technical tool in a much larger debate about the relationship between content creators and AI developers.

What publishers should do now

For publishers, the right response is preparation rather than a rushed change to a live website.

1. Audit your current robots.txt file

Find your existing file, usually at yourdomain.com/robots.txt. Review the rules already in place for search engines and AI-related crawlers. Keep a copy before making changes.

2. Decide what you want to permit

Consider whether your preference should apply to the whole website or only selected areas. News articles, public press releases, subscriber-only material, image libraries and commercial datasets may need different treatment. Make sure the policy reflects your actual rights and business strategy.

3. Keep training and grounding separate

Decide whether you are addressing model training, use of your content to support AI-generated answers, or both. A training opt-out does not necessarily prevent a service from retrieving a page to answer a user’s question. Check the controls offered by each provider.

4. Track rollout dates and documentation

Do not assume a rule is effective simply because it appears in your file. Google, Microsoft, OpenAI and Anthropic have described different timelines. Microsoft has also announced changes to its older NoArchive and NoCache controls, so sites that rely on those settings should review Microsoft’s migration guidance ahead of the stated deadlines.

5. Test and document your changes

When support becomes available, validate the syntax, check that the intended paths are covered and record when the file was changed. Revisit the policy as providers publish implementation details.

What the proposal cannot guarantee

The shared signal is a step towards clearer communication, not a technical enforcement system. The proposal does not automatically remove material that may have been collected previously. It does not itself create a licensing agreement, settle copyright disputes or provide an audit trail proving how every dataset was used.

It also depends on adoption. A company that has not implemented the signal may not respond to it. Other AI developers may choose different approaches, and the proposal remains part of an evolving standards process. The Internet-Draft is not the same as a completed, universally adopted IETF standard.

For publishers, this means a layered approach remains sensible: maintain clear crawler rules, review provider-specific controls, understand your copyright and licensing position, and keep records of permissions and policy decisions.

The bigger question: who gets to set the rules for the open web?

The debate over AI and publisher content is moving from broad arguments into practical internet infrastructure. A common robots.txt preference could give website owners a simpler way to communicate their choices, but its real value will depend on whether companies implement it, respect it and explain how it interacts with other uses of web content.

For independent media, the issue is not simply whether AI companies can access a website. It is whether publishers can make informed decisions about the use of their work while preserving the ability to be discovered by readers. The shared signal is a promising development—but transparency, consistent implementation and fair commercial arrangements will determine whether it delivers meaningful control.

Frequently asked questions

Does the new robots.txt token block AI companies from crawling my website?

No. The proposed __usage_ai-training token communicates a preference about AI training. It is not a general instruction to block crawling, and it should not be confused with rules that restrict crawler access.

Will adding the rule remove my website from Google Search?

Google says the shared training opt-out is intended to control AI training use and that its existing controls remain relevant. Do not assume that a training preference is the same as a search-indexing block. Review Google’s current documentation for the exact effect of each control.

Should every website add the new rule immediately?

Not necessarily. Provider support is being introduced on different timelines, and the draft is not a fully deployed universal standard. Website owners should first decide their content-use policy, review existing rules and wait for the relevant providers’ implementation guidance before relying on the new signal.

Sources

Editorial note: This article explains a developing technical proposal and company announcements available as of 1 November 2026. Implementation details and standards status may change. Website owners should consult current provider documentation before changing crawler policies.