Tech News
← Home  ·  All topics

Common Crawl

4 GoKawiil briefs on this topic · RSS for common crawl

Essay argues no equivalent 'commons' exists for posttraining LLM data

A new essay examines how large language model pretraining relied on an enormous, freely available body of text—Wikipedia, Common Crawl, ArXiv, Stack Exchange, Project Gutenberg and open-source code—that had accumulated over decades. The author notes that Common Crawl alone holds roughly 10 petabytes of archived web data, and that models like GPT-3 and the original Llama were built largely from this material. The piece then argues that no comparable public resource exists for the posttraining stage of building AI models.

Study of 217M Common Crawl pages finds 7,959 sites sending AI-targeted HTTP headers

Researchers scanned 217 million HTTP response headers from a 10% sample of Common Crawl's September 2026 crawl and found 7,959 sites including headers addressed to AI agents. Most, 7,271 sites, were pointers directing agents to resources like llms.txt files or MCP servers, while 30 sites contained attempted prompt injections and 139 sites flagged whether they had identified the visitor as a bot.

Project Timestamper seals 339 billion Common Crawl web snapshots to Bitcoin blockchain

Project Timestamper has cryptographically timestamped the entire Common Crawl archive, covering 339 billion web page snapshots gathered over 18 years and totaling more than 10 petabytes. Using existing hash blocks of up to 3,000 index entries provided by Common Crawl, the team hashed roughly 134 million blocks and committed them via the OpenTimestamps service to the Bitcoin blockchain as of September 26, 2026.

AI firms begin paying for training content, but publishers can't set the price

A market has emerged in which AI companies pay to license publisher content for training large language models, after years of scraping the open web largely for free using resources like Common Crawl. Publishers are now receiving payment, but reportedly have little or no influence over how that price is determined.