OpenAI, Google and Microsoft back new standard for AI scraping opt-out

OpenAI, Google, Microsoft, and Anthropic have announced their commitment to a new industry standard that allows publishers to opt out of AI scraping more efficiently. The initiative, centered on a shared "policy token," aims to centralize control over how website content is indexed and utilized by artificial intelligence models.

• The standard uses a shared signal within the robots.txt file to govern scraping.
• Publishers can apply AI preferences across multiple platforms simultaneously.
• The system differentiates between search indexing and AI training/grounding.
• Anthropic has already implemented it, with other giants following in the coming months.

Simplifying Web Crawler Management Currently, publishers struggle with the influx of various AI crawlers feeding data into models without granular control. The proposed "policy token," introduced via the Internet Engineering Task Force (IETF), streamlines this process. By placing a specific signal in the robots.txt file, site owners can dictate whether their data can be used for AI training, without needing to negotiate with each company separately. Microsoft has noted that these controls will provide granular options, allowing publishers to distinguish between search indexing (keeping their site visible in Bing) and AI training or grounding, which involves pointing a model to verified sources to improve accuracy.

Adoption Timeline and Industry Support Anthropic is already utilizing the signal, while OpenAI plans to implement it in the coming months. Google and Microsoft have scheduled their support for the upcoming year. While the Meta communications head Andy Stone expressed support for the concept of a common signal, he stopped short of confirming when or if the company will fully honor it in practice. This move represents a significant shift toward standardizing the relationship between publishers and AI companies, following previous individual efforts like Cloudflare's default blocking or licensing models like Really Simple Licensing (RSL).

Why this matters for the ecosystem For developers and business owners, this creates a uniform framework that reduces the technical overhead of managing web scraping. It provides a standardized way to protect proprietary data while maintaining visibility in search engines. For the global digital ecosystem, this is a vital step toward reconciling the rapid growth of AI with the rights and preferences of the content creators who provide the foundational data for these models.