Google and OpenAI agree on a unified robots.txt standard for AI

Google, alongside major AI developers such as OpenAI, Microsoft, and Anthropic, has announced plans to implement a unified mechanism in the robots.txt file to manage content used for machine learning. This will allow website owners to set their machine learning preferences for multiple developers at once with a single configuration.

• A unified policy token in robots.txt will replace separate instructions for different AI providers.
• The decision builds on work by the AIPREF working group within the IETF.
• Existing individual control tools like Google-Extended will continue to function.
• The initiative aligns with the EU AI Act's Code of Practice for general-purpose AI models.

A New Standard for Managing AI Training Data

The new solution is based on the developments of the AIPREF working group within the Internet Engineering Task Force (IETF). Instead of creating separate instructions for each AI developer, website owners will be able to use a special policy token in the robots.txt file that defines general rules for all participating providers.

If a site uses this token to prohibit the use of its content for AI training without a separate Google-Extended directive, Google will interpret this as a ban equivalent to Google-Extended. Other AI developers can act similarly, simplifying access management for website administrators.

Compatibility with Existing Tools and EU Regulations

The developers emphasize that existing tools designed for individual companies will continue to work. Website owners can set general rules using the industry standard while simultaneously refining instructions for specific developers, such as using Google-Extended for Google.

Sites that explicitly allow content usage through Google-Extended may appear in answers and receive user traffic through grounding mechanisms in Gemini applications. This ensures transparency and mutual benefit for creators and tech platforms alike.

Significance for the Industry and Regulators

According to Google, this proposal takes into account the wishes of publishers, public organizations, and regulators who advocate for unified industry standards. Support from leading AI developers lays the foundation for a reliable and durable cross-industry standard.

The initiative also complies with the provisions of the Code of Practice for general-purpose AI models under the European Union Artificial Intelligence Act (EU AI Act). The EU AI Act recognizes the robots.txt file as the primary way for website owners to express their preferences in a machine-readable format.