Google and OpenAI back new robots.txt standard to block AI training

Google, OpenAI, Anthropic, and Microsoft have committed to supporting a new common robots.txt directive that allows publishers to specify their preferences regarding content scraping for generative AI training. The proposal is currently under consideration at IETF.

The initiative aims to simplify how website owners manage and protect their intellectual property from being automatically harvested by various AI model creators without explicit consent.

• A new unified robots.txt signal is introduced to block AI training scrapers.
• The proposal was written by Anthropic standards lead Kevin Kelley via IETF.
• Major tech firms will roll out support in the coming months and early 2027.
• The directive stops AI training data collection without affecting search indexing.

A unified approach to web scraping controls

Previously, website owners had to identify individual crawlers and configure separate rules for each AI provider, making content management cumbersome. Kelley's proposal at IETF changes this by getting multiple artificial intelligence companies to recognize and honor a single common signal.

Publishers will be able to set up a dedicated user agent within their robots.txt file specifically tailored for AI training usage. This allows operators to easily disallow general scraping while selectively permitting access to specific directories, such as press kits or public announcements.

Impact on search engines and past data

It is important to note that this new directive does not prevent regular crawling or block content from appearing in standard search engine results. For instance, Google will interpret the signal specifically as a stop sign for its Gemini-training crawler, Google-Extended.

Furthermore, the directive cannot undo any data harvesting that companies have already performed in the past. It serves strictly as a preventative measure against future scraping of websites or specific web pages.

Timelines and industry rollout

The participating tech companies have outlined their implementation schedules. Anthropic has already started supporting the directive, and OpenAI plans to implement the signal in the coming months, while Microsoft and Google will follow by early 2027.

For digital publishers, businesses, and content creators globally, this standardized signal represents a transparent and efficient mechanism to protect proprietary content while maintaining full visibility in search engines.