GoKawiilLinux Foundation reports AI scrapers now dominate its git server traffic
The Linux Foundation's IT infrastructure director revealed that AI crawlers hunting for clean, LLM-free training data are hammering the organization's kernel.org git servers, consuming more CPU power than all legitimate human and tool access combined. Because cgit can generate near-infinite valid URLs for commits, diffs, and patches across hundreds of forks, scrapers are redundantly fetching duplicate data at massive scale.
GoKawiil's interpretation of the reporting above, not reported fact.
This shows how the race for high-quality, human-verified training data is straining core open-source infrastructure that underpins much of modern software. As bots evade detection by mimicking browsers and spreading across IP ranges, maintainers are forced into an escalating arms race just to keep essential developer services running.
- AI scrapers now consume more server resources than all legitimate access to Linux kernel git history combined.
- Kernel commit history is prized as guaranteed pre-AI training data, driving aggressive scraping.
- Bots have evolved past simple user-agent bans to IP spoofing and subnet-wide crawling, complicating mitigation.
Linux Kernel Development by Robert Love — If a story about 1.48 million kernel commits makes you curious what's actually inside all that git history, Robert Love's classic is the friendliest way in. It walks through process scheduling, memory management, and the kernel subsystems that those commits keep reshaping — human-written, no scraper required.
See Linux Kernel Development by Robert Love on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: linux.slashdot.org — Posted, 2026-08-30
Published there as: “AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.