Creative Commons expresses preliminary approval for AI models that require payment for web crawling

Creative Commons Endorses Pay-to-Crawl Solutions for AI Content Use

Earlier this year, Creative Commons introduced a plan to foster an open AI ecosystem. Now, the nonprofit has voiced conditional support for “pay-to-crawl” technologies—systems designed to automate payments to website owners when their content is accessed by automated tools such as AI web crawlers.

Creative Commons, widely recognized for its role in promoting flexible copyright licensing, revealed in July its intention to establish both legal and technical standards for sharing datasets between data holders and AI developers seeking to train their models.

Recently, the organization has expressed “cautious support” for pay-to-crawl mechanisms.

According to a recent Creative Commons blog post, “If implemented thoughtfully, pay-to-crawl could help websites fund the ongoing creation and distribution of their content, and better control how their material is used. This approach might keep information publicly available that would otherwise be hidden behind stricter paywalls or not shared at all.”

Companies such as Cloudflare are leading the charge on pay-to-crawl, proposing that AI bots pay a fee each time they extract content from a website for the purpose of training or updating AI models.

Traditionally, websites have allowed search engine crawlers to index their pages, benefiting from increased visibility and traffic. However, the rise of AI chatbots has changed this landscape. When users receive answers directly from AI, they are less likely to visit the original source, reducing referral traffic for publishers.

This change has already had a significant negative impact on publishers, as search-driven visits decline with no signs of recovery.

Introducing pay-to-crawl could help publishers offset the financial losses caused by AI-driven content consumption. This model may be especially beneficial for smaller sites that lack the leverage to negotiate individual licensing agreements with AI companies. While large-scale deals have been made between organizations like OpenAI and Condé Nast, Axel Springer, Perplexity and Gannett, Amazon and The New York Times, and Meta with various media outlets, smaller publishers are often left out.

Creative Commons did, however, highlight several concerns. They cautioned that pay-to-crawl systems could further concentrate power among a few major players and potentially restrict access for researchers, nonprofit organizations, educators, and others serving the public good.

To address these issues, Creative Commons recommended several guiding principles for responsible pay-to-crawl implementation:

• Pay-to-crawl should not be the default for all websites.
• Broad, one-size-fits-all rules should be avoided.
• Systems should allow for throttling access, not just outright blocking.
• Public interest access must be preserved.
• Solutions should be open, interoperable, and built on standardized components.

Cloudflare is not alone in this space. Microsoft is developing an AI marketplace for publishers, and startups like ProRata.ai and TollBit are also entering the field. Additionally, the RSL Collective has introduced a new standard called Really Simple Licensing (RSL), which specifies which parts of a website can be accessed by crawlers without fully blocking them. This standard has been adopted by Cloudflare, Akamai, and Fastly, and is supported by organizations such as Yahoo, Ziff Davis, and O’Reilly Media.

Creative Commons has also endorsed the RSL standard, alongside its broader initiative, CC signals, which aims to develop new tools and technologies for the AI era.