Wikipedia is struggling with voracious AI bot crawlers

Wikimedia has seen a 50 percent increase in bandwidth used for downloading multimedia content since January 2024, the foundation said in an update. But it's not because human readers have suddenly developed a voracious appetite for consuming Wikipedia articles and for watching videos or downloading files from Wikimedia Commons. No, the spike in usage came from AI crawlers, or automated programs scraping Wikimedia's openly licensed images, videos, articles and other files to train generative artificial intelligence models. This sudden increase in traffic from bots could slow down access to Wikimedia's pages and assets, especially during high-interest events. When Jimmy Carter died in December, for instance, people's heightened interest in the video of his presidential debate with Ronald Reagan caused slow page load times for some users. Wikimedia is equipped to sustain traffic spikes from human readers during such events, and users watching Carter's video shouldn't have caused any issu ... Read full article.