With web publishers in crisis, a new open standard lets them set the ground rules for AI scrapers. (Or, at least it will try.) The new Really Simple Licensing (RSL) standard creates terms that participants expect AI companies to abide by. Although enforcement is an open question, it can't hurt that some heavy hitters back it. Among others, the list includes Reddit, Yahoo (Engadget's parent company), Medium and People Inc.
RSL adds licensing terms to the robots.txt protocol, the simple file that provides instructions for web crawlers. Supported licensing options include free, attribution, subscription, pay-per-crawl and pay-per-inference. (The latter means AI companies only pay publishers when the content is used to generate a response.)
Launching alongside the standard is a new managing nonprofit, the RSL Collective. It views itself as an equivalent of nonprofits like ASCAP and BMI, which manage music industry royalties. The new group says its standard can "establish fair market prices and strengthen negotiation leverage for all publishers."
Advertisement Advertisement
Advertisement
Participating brands include plenty of internet old-schoolers. Reddit, People Inc., Yahoo, Internet Brands, Ziff Davis, wikiHow, O'Reilly Media, Medium, The Daily Beast, Miso.AI, Raptive, Ranker and Evolve Media are all on board. Former Ask.com CEO Doug Leeds and RSS co-creator Eckart Walther lead the group.
"The RSL Standard gives publishers and platforms a clear, scalable way to set licensing terms in the AI era,” Reddit CEO Steve Huffman wrote in a press release. "The RSL Collective offers a path to do it together. Reddit supports both as important steps toward protecting the open web and the communities that make it thrive." (It's worth noting that Reddit has licensing deals with OpenAI and Google.)
It's unclear whether AI companies will honor the standard. After all, they've been known to simply ignore robots.txt instructions. But the group believes its terms will be legally enforceable.
In an interview with Ars Technica, Leeds pointed to Anthropic's recent $1.5 billion settlement, suggesting "there's real money at stake" for AI companies that don't train "legitimately." (However, that settlement is up in the air after a judge rejected it.) Leeds told The Verge that the standard's collective nature could also help spread legal costs, making challenges to violations more feasible.
Advertisement Advertisement
... continue reading