Web giants propose a standard to regulate AI scraping

OpenAI, Google, Microsoft and Anthropic announced an agreement to propose a single standard that allows editors to control the scraping of their content by AI systems, using the robots.txt format as a basis. According to ANSA, the initiative aims to give sites a recognized instruction that will be respected by technology companies that adhere to the protocol.

The announced agreement

• The four companies committed to working on a common standard that indicates if and how content published on web pages can be harvested to train AI models.
• The proposal provides that editors and website owners publish a single technical instruction that AI providers will recognize and follow, reducing the need for individual negotiations.
• According to ANSA, the goal is to create a practical mechanism that benefits both large portals and small content producers.

How the standard will work (scraping, AI, robots.txt)

• The solution leverages the already known robots.txt format, a file that informs search engine robots which pages can be indexed, adapting it to express preferences regarding the use of content for AI training.
• The proposal was drafted by Kevin Kelley, Anthropic's standards lead, and is under review by the Internet Engineering Task Force (IETF), the body responsible for internet standardization, according to ANSA.
• Implementation will not be immediate: companies should begin integrations and testing at different times, and full adoption will depend on approval by the IETF as well as the technical rollout by AI providers.
• In practical terms, editors may signal via a single line in robots.txt whether they authorize or not the collection of their texts, images, and other assets for use in language and multimodal models — the instruction would be interpreted by AI crawlers that follow the standard.

Legal and editorial context

• The announcement comes amid a surge in legal disputes filed by editors against the automatic use of their content by large models. The technical proposal emerges as a partial response to a demand from the press industry for greater control and potentially for compensation.
• ANSA notes that, on regulatory grounds, the European Union recently confirmed the legitimacy of the Italian model for the so-called "equo compenso" for editors, a decision that strengthens positions seeking forms of remuneration for the reuse of journalistic content.
• Experts note that even with a technical standard, copyright and compensation issues may continue to require contractual negotiations and court decisions, since technology does not replace national laws and European directives.

Relevance to editors in Portuguese and descendants of immigrants

• The proposed standard could facilitate protection of content produced in Portuguese, including portals run by Brazilians and descendants of immigrants who publish local and cultural material.
• For small editors and personal sites, the possibility of placing a single instruction in robots.txt represents a significant reduction in technical complexity and cost: instead of multiple signals and policies, a single standard that, if recognized by major AI providers, would be automatically respected.
• The initiative could also simplify editorial strategies related to copyright and neighboring rights negotiations: with a standard signaling, it becomes clearer when content is available for training use or when a license or prior compensation is required.

Limits and next steps

• The proposal still depends on analysis and potential adoption by the IETF; there is no guarantee that all providers will follow it uniformly, nor that legal questions will be resolved by a technical standard alone.
• The technical solution does not eliminate the need for legal oversight: decisions about remuneration, usage exceptions, and enforcement will remain subject to laws and judicial disputes.
• For editors interested, it is advisable to follow the IETF draft's progress and assess, with legal support, how to combine technical signals with contractual agreements and collective actions when appropriate.

Conclusion: OpenAI, Google, Microsoft, and Anthropic's public commitment to a common standard based on robots.txt marks an important step toward a more coordinated approach to controlling the use of content by AI systems. While it does not by itself resolve all legal and compensation issues, the proposal promises to simplify the technical rights-management for many editors, including those publishing in Portuguese.

Source: ANSA