One robots.txt Line to Opt Out of AI Training: What Businesses Must Do

On 30 October 2026, OpenAI, Google, Microsoft and Anthropic announced support for a single shared robots.txt signal for AI training preferences: add a group for User-Agent: __usage_ai-training followed by Disallow: /. That tells participating AI companies not to use your site for training. Add it now, but do not treat it as a magic switch: it is not yet a formal standard, not every company will honour it on the same date, and it does not remove old training data, block search or stop live answers.

What changed on 30 October 2026

Until now, opting out of AI training meant juggling separate controls for separate companies. The 30 October announcement gives publishers one shared token to express a training preference, instead of a different instruction for every crawler.

The signal is set out in an individual IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, “Machine-Readable Training Preference Signal”, by Kevin Kelley of Anthropic, published on 30 October 2026. It is an individual draft, not yet a formal standard, and it expires on 3 May 2027 unless revised or adopted.

The code is deliberately simple:

User-agent: __usage_ai-training
Disallow: /

Coverage of the announcement is summarised in SaveDelete's report on the shared AI-training robots.txt token and Emergent's report on the big AI labs backing the shared token.

What the signal does — and what it does not do

What it is designed to do

• Express a training preference in one place, covering the companies that commit to honouring it.
• Give crawlers a machine-readable instruction, in a file they already read, rather than a notice buried in terms and conditions.
• Reduce future training use by companies that implement it.

What it does not do

• It does not remove content already in training datasets, or un-train an existing model.
• It does not block search crawling or ranking. It is not an SEO block.
• It does not stop live web grounding for answers. A model can still be given live web content to ground an answer.
• It does not bind companies that have not committed to it. Support and implementation are not the same thing.

Training, search and live answers are three different decisions. If your business depends on being found or cited by AI answers, manage those channels separately — see our generative engine optimisation guide on getting cited by ChatGPT.

Who honours it, and when

Do not read “announced support” as “live everywhere today”. Timelines differ:

• Anthropic has started implementing. It is the company most likely to be acting on the signal first.
• OpenAI aims to implement in the coming months.
• Microsoft will honour it from 1 February 2027 as it crawls. Until then, do not assume its crawlers are applying it.
• Google aims for 2027, with its crawler documentation pointing to early 2027.
• Meta has expressed support for a common signal but has NOT committed to honouring this token. Do not tell a client Meta is covered. It is not, on this announcement.

Microsoft separately announced an AI Grounding control, dealing with live answers versus training. Its transition settings open on 15 December 2026, and temporary settings expire on 30 September 2027. It is about grounding, not this robots.txt token.

What to do now: a five-step action plan

1. Audit what you actually publish

List your own articles and product pages, user-generated content, licensed images and copy, and syndicated content. A training preference only makes sense once you know which content is yours to restrict, which is licensed under terms that may already limit reuse, and which you actually want widely quoted.

2. Add the shared line — correctly

Add the group to robots.txt exactly as specified, and test that the file still parses. A malformed robots.txt file can cause more harm than no change, because search crawlers read the same file. Add the new group alongside your existing groups; do not touch your search rules unless you have separately decided to change them.

3. Keep per-company tokens until support lands

Because implementation dates differ and Meta has not committed to this token, your existing per-company training controls are still doing real work. Keep them until each company's implementation is confirmed, then review — do not delete them on announcement day.

4. Separate search, training and live answers in your policy

Write the distinction down for anyone who touches the site. Search is one decision, training is a second, live grounding is a third. Conflating them causes the two classic errors: blocking search by accident while trying to block training, or assuming a training opt-out stops a model quoting your live pages. For teams building AI agents or AI automation, the same separation should govern your own data pipeline: what you may train on, retrieve at answer time, and index are three different permissions.

5. Review content licensing alongside the technical signal

Robots.txt is a technical preference, not a contract. If licensing matters to your business, review licences, terms and syndication agreements in parallel, including whether licensed content on your site is covered by the licensor's terms. Teams in custom software development, SaaS development and CRM development should also check how training data is sourced for their own products: an opt-out on your marketing site does not govern a dataset bought, scraped or licensed elsewhere.

What this means for different kinds of business

• Publishers and content brands: your archive is the asset. Decide whether exclusion is a house policy or a per-section decision, and keep the implementation log above.
• E-commerce and service businesses: product copy should usually be found, while original guides and tools are worth controlling. Our e-commerce development work regularly draws exactly that line.
• Agencies: you need a per-client policy, not a house default. Record each client's choice and which legacy controls remain live.
• App and software teams: a robots.txt decision affects web copy, not the app itself. Teams in Flutter app development should check web, app-store and in-app content separately.

For how these tools work in practice, see our AI chatbot versus AI agent guide and our AI automation for logistics companies guide, which show retrieval, training and live data kept apart in a working system.

What NOT to assume

• Do not assume it is already a standard. It is an individual draft, expiring 3 May 2027 unless revised or adopted.
• Do not assume all four companies honour it today. Only Anthropic has started implementing.
• Do not assume Meta is covered. Meta supports a common signal in principle but has not committed to this token.
• Do not assume old content is withdrawn. Content already in training datasets is not removed by this signal.
• Do not assume search or live answers are blocked. Neither is — those need separate decisions, including Microsoft's grounding settings from 15 December 2026.

Getting help in your market

Coheser builds AI agent systems and AI automation workflows for businesses that need to control how their content is used, across the United Kingdom, the UAE, Saudi Arabia, the USA and Pakistan. Smaller teams can start with our AI agents for small business automation guide.

Frequently asked questions

Is one robots.txt line enough to opt out of AI training?

It is the right shared line to add, but not enough on its own today. Keep existing per-company controls until each company confirms implementation, because timelines differ and Meta has not committed to this token.

Will this remove my content from models that already trained on it?

No. It addresses future training use by companies that honour it. It does not remove content already in training datasets or un-train an existing model.

Will adding this line hurt my search ranking?

Not by itself: this group is about training preference, not search crawling or ranking.

Is this now a formal web standard?

Not yet. It is an individual IETF Internet-Draft published on 30 October 2026, expiring on 3 May 2027 unless revised or adopted. Treat it as a jointly backed proposal to implement and monitor.

What should I do about Microsoft's AI Grounding control?

Treat it separately from the robots.txt token. Its transition settings open on 15 December 2026 and temporary settings expire on 30 September 2027, so diarise a review when the settings open.