Google, OpenAI, Anthropic and Microsoft Back a Shared robots.txt Signal Against AI Training Scraping

Google, OpenAI, Anthropic and Microsoft have pledged to support a new common signal in robots.txt that lets sites block scraping for AI training. Until now, each company ran its own crawler with its own opt-out name, so publishers had to manage a growing list of rules. A shared signal could replace that list with a single instruction, although the technical details have not yet been disclosed.

The patchwork this is meant to replace

robots.txt is a plain-text file that tells automated crawlers which parts of a site they may fetch. The convention dates to the 1990s and was formalized by the IETF as RFC 9309 in 2022. It was designed for search indexing, not for the question of whether content may be used to train a generative model.

When AI training scraping became a flashpoint in 2023, companies responded one by one. OpenAI published GPTBot and told site owners how to block it. Google introduced a Google-Extended token that controls whether content is used for its Gemini models and related AI products, without changing how a site appears in Search. Anthropic and others followed with their own user-agent names.

The result is that a site owner who wants to opt out of AI training has to name each crawler individually and keep updating the file as new ones appear. A crawler that is not on the list is simply not blocked.

What the new pledge changes

The four companies have committed to support one shared signal for blocking scraping for AI training. If it works as described, a publisher states its preference once and every participating crawler is expected to honor it, regardless of whose it is.

What has not been disclosed matters just as much: the exact name and syntax of the signal, the date participating crawlers will start respecting it, and how compliance would be checked. Treat any specific implementation claim as unconfirmed until a specification is published.

Why this is more than housekeeping

The scraping fight has mostly been argued in courts and legislatures. This pledge moves part of it into a technical standard that sites can act on today without waiting for a ruling.

It also fits a pattern we covered earlier. In Why Congress Should Investigate OpenAI and Anthropic's Research Practices, the concern was that leading labs shape the rules around them. This announcement is a different kind of rule-making: voluntary and industry-led rather than legislated, which makes it faster to adopt but harder to enforce.

Limits worth keeping in mind

robots.txt has always relied on good behavior. Nothing technically stops a crawler from ignoring it, and companies outside this group are not bound by the pledge. Content already collected and used in training is not affected by a new opt-out.

Scope is the other open question. A signal covering only training is a narrower promise than one that also covers content fetched to generate AI answers at query time. Google's existing split, where Google-Extended governs AI training while Googlebot governs Search, suggests it may keep treating those uses separately. That is an inference from past behavior, not something announced, but it is where publishers will probably push hardest.

What it means for different groups

Publishers and site owners could eventually maintain one rule instead of many, but should keep their existing per-crawler rules until a spec exists. Developers building crawlers or AI products will likely face pressure to support the signal, since the largest players have publicly committed to it. Smaller AI companies that do not adopt it may find that their data collection draws more scrutiny by comparison.

Takeaway

A common opt-out signal backed by four of the biggest AI players would simplify a messy corner of the web. Its real value depends on three things: a published specification, verifiable compliance, and a clear answer on whether it covers more than model training.

Reference: https://www.engadget.com/2303225/google-openai-anthropic-ai-crawling-controls/