Four of the largest artificial intelligence companies now back a single line in a website's robots.txt file that lets publishers opt out of having their content used to train AI models. Google, Microsoft, OpenAI and Anthropic have all said they will honour the shared signal, which was proposed on 30 October 2026.

The idea comes from Kevin Kelley, a technical lead for standards at Anthropic. He submitted it to the Internet Engineering Task Force (IETF) as an Internet-Draft titled "Machine-Readable Training Preference Signal". The Verge first reported the agreement, and TechRadar covered the story this week as publishers began asking what the signal actually does.

How the opt-out works

A website owner who wants to opt out adds two lines to the robots.txt file at the root of the site:

User-Agent: __usage_ai-training

Disallow: /

The token, __usage_ai-training, sits where a crawler's name would normally go. A site that wants to keep part of its content open to training can add an Allow line for that path, for example for a press kit. Until now, publishers had to write a separate rule for each company's crawler, and sometimes for several crawlers run by the same company.

When each company will honour it

The four backers are moving at different speeds. Anthropic says it has already started implementing support for the token across its services. OpenAI says it will honour the signal for foundational generative AI training and aims to implement it in the coming months.

Microsoft has given the firmest date. It will honour the shared training preference as its Bingbot crawler revisits pages from 1 February 2027, and it will also offer a separate page-level AI Grounding preference. From 1 March 2027, Bing will stop reading its older NoArchive and NoCache tags as training or grounding preferences.

Google aims to support the token in early 2027. Its Search Central post by search advocate John Mueller says a site that opts out through the shared line, without naming Google-Extended separately, will be treated as if it had disallowed Google-Extended. That also takes the site out of grounded answers in Gemini Apps, so publishers need to weigh that trade-off.

What the signal does not do

The limits matter as much as the promise. The draft states plainly that the token "is not a directive for crawling behaviour". A crawler that respects it still requests the page, and search keeps working. The signal covers generative AI training only, not the use of content for search results or for AI answers that cite a page.

It also does nothing about content already collected. A site that adds the line today cannot claw back material that AI developers have already gathered, and there is no way for outsiders to check whether a crawler is actually obeying the preference.

The document itself is an individual Internet-Draft with no formal IETF standing. It expires on 3 May 2027 unless it is revised or adopted. Calling it a standard is premature, TechRadar noted.

Meta has not committed

Meta is the notable absence. Its communications head, Andy Stone, told The Verge that creating a common signal "is an important step", but the company has not committed to honouring it.

Uptake so far is minimal. A check of 50 news sites on 1 November found that none of them carried the new line in their robots.txt files. Publishers who want the protection will need to add it themselves and keep their existing per-company rules in place until each company ships support.