GoKawiil
Tech News
Researchers scanned 217 million HTTP response headers from a 10% sample of Common Crawl's September 2026 crawl and found 7,959 sites including headers addressed to AI agents. Most, 7,271 sites, were pointers directing agents to resources like llms.txt files or MCP servers, while 30 sites contained attempted prompt injections and 139 sites flagged whether they had identified the visitor as a bot.
Project Timestamper has cryptographically timestamped the entire Common Crawl archive, covering 339 billion web page snapshots gathered over 18 years and totaling more than 10 petabytes. Using existing hash blocks of up to 3,000 index entries provided by Common Crawl, the team hashed roughly 134 million blocks and committed them via the OpenTimestamps service to the Bitcoin blockchain as of September 26, 2026.
A market has emerged in which AI companies pay to license publisher content for training large language models, after years of scraping the open web largely for free using resources like Common Crawl. Publishers are now receiving payment, but reportedly have little or no influence over how that price is determined.
Paramount has announced release dates for follow-up films to its horror titles 'Heart Eyes' and 'Crawl,' signaling plans to turn both into ongoing franchises. The studio is moving to capitalize on the earlier films' success by locking in theatrical slots for the sequels.