AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%

The Wikimedia Foundation, the umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects, said on Wednesday that bandwidth consumption for multimedia downloads from Wikimedia Commons has surged by 50% since January 2024. The reason, the outfit wrote in a blog post Tuesday, isn’t due to growing demand from knowledge-thirsty humans, but from automated, data-hungry scrapers looking to train AI models. “Our infrastructure is built to sustain sudden traffic spikes from humans during high-interest events, but the amount of traffic generated by scraper bots is unprecedented and presents growing risks and costs,” the post reads. Wikimedia Commons is a freely accessible repository of images, videos and audio files that are available under open licenses or are otherwise in the public domain. Digging down, Wikimedia says that almost two-thirds (65%) of the most “expensive” traffic — that is, the most resource-intensive in terms of the kind of content consumed — ... Read full article.