GoKawiil
Tech News
Chris Siebenmann, who runs the Wandering Thoughts blog and CSpace wiki, says he is now attempting to block requests from clients whose HTTP User-Agent header is too generic or suspicious. He attributes the move to a surge in high-volume crawlers, some reportedly gathering data for LLM training, that strain his site's resources.
Hugo Vergnes built a config-driven training framework called little-lm and used it to train a 3.8-billion-parameter language model from scratch on 65 billion tokens, taking 43 hours on eight rented B200 GPUs at a total cost of $998. The model scored 0.384 on the CORE benchmark, outperforming Andrej Karpathy's nanochat d32 model, which cost about the same to train but scored 0.310.
The maintainer of git.kernel.org reports that AI training crawlers are consuming more CPU resources than all legitimate access combined, including actual git clones. Across five geographically distributed nodes, 14 CPU cores are constantly occupied just rendering git commits as HTML pages for these bots, even though the same data is available far more efficiently via direct git clone.