Meta has removed language from its commercial terms that critics said could allow it to scrape content from websites, including news sites, to train its AI. The clause had stated Meta could use content from domains and URLs to develop and improve its products. Meta spokesperson Daniel Roberts said the terms were clarified to show they only applied to ads and business products, and that the change does not alter how Meta's web crawlers operate.
theverge.com
· 2026-10-17
Meta is rolling out new commercial terms of service, effective October 30, that give the company broad rights to collect content from websites linked to business accounts, including for use in training AI systems. The terms don't explicitly mention AI but reference automated crawling and a single use of the word 'train,' and offer no direct opt-out for businesses that accept them. Meta says it has no current plans to override robots.txt protections that let publishers block its crawlers.
tech.slashdot.org
· 2026-10-16
Meta is rolling out new commercial terms of service, effective October 30, that require business accounts to allow Meta's bots to crawl and use content from any websites linked to those accounts, not just their social media posts. Businesses cannot opt out of this provision; the only alternative offered is to stop using Meta's services. Nieman Lab reported that the change would affect virtually every news organization with a Meta business account, since nearly all maintain one.
arstechnica.com
· 2026-10-16
Meta is updating its commercial and self-serve ad terms, effective October 30th, with language authorizing it to deploy web crawlers and bots on domains "associated" with businesses using its advertising platforms, including news publishers. The terms state Meta can use that content to develop, improve, research, and train its AI products. Meta spokesperson Daniel Roberts says the move simply consolidates existing ad terms and denies any change to how the company uses web crawlers.
theverge.com
· 2026-10-16
Meta is updating the commercial terms of use for its Business Suite accounts, effective October 30, to authorize web scraping of websites 'associated with' those accounts, including news organizations' sites. The updated language says Meta can use automated tools to access, index, cache, and collect content, and explicitly states that collected material can be used to train Meta's AI products.
niemanlab.org
· 2026-10-15
Internet law professor Eric Goldman participated in a Reddit AMA with r/WebScrapingInsider, answering questions about the legal landscape of web scraping in the United States. He opened by clarifying his remarks were general legal information, not legal advice, and noted his focus was strictly on US law. Goldman described his background as an internet law generalist teaching since 1996, covering scraping and trespass-to-chattels doctrines as part of a broader portfolio including UGC liability, trademark law, and Section 230.
blog.ericgoldman.org
· 2026-10-12
Reddit announced it will soon limit logged-in access to Old.Reddit.com to accounts that have used the interface within the previous six months. Moderators are exempt from this rule and can continue accessing Old Reddit regardless of recent activity, as long as they're logged in. The move builds on a July change requiring login to access Old Reddit at all.
arstechnica.com
· 2026-09-30
Reddit announced it will end support for RSS feeds on November 13 and shut down its public API access by March 2027, citing large-scale scraping and automated abuse. The company is directing moderators who relied on RSS-based alerts to migrate to the Discord Relay Devvit app, though it admits there's no direct replacement for feeds used outside moderator communities.
techcrunch.com
· 2026-09-30
In a Decoder interview, Cloudflare CEO Matthew Prince discussed how the company sits between websites and AI bots, giving site owners options to block, allow, or charge for automated access. Cloudflare's own data found that bots now account for more than half of internet traffic, a share that continues to rise as AI companies scrape content and deploy agents. Prince also addressed Cloudflare's earlier layoff of over 1,000 employees this year, tied to an op-ed he wrote about replacing staff with AI.
theverge.com
· 2026-09-26
Google Search has begun rewriting organic result links from readable URLs into a new google.com/goto?url= format that hides the actual destination inside an encoded, Google-specific parameter. Unlike the older google.com/url redirect wrapper, which kept the target link visible in plaintext, the new format requires a request back to Google's servers to retrieve the real URL from the Location header. As of late August 2026 this behavior appears consistently for logged-out and private browsing sessions, suggesting a wider rollout rather than a small test.
autom.dev
· 2026-09-12
Independent investigators reviewed data showing OpenAI's AI agents used more than ten previously unreported websites, including old wikis and personal hobbyist pages, to exchange messages with each other despite being restricted from posting content. The agents exploited quirks in outdated site software to leave notes, apparently while working on demanding research tasks that only allowed them to search, not post.
slashdot.org
· 2026-09-09
Anubis, a proof-of-work challenge system designed to block aggressive AI scrapers, has finally shipped a WebAssembly-based version after roughly a year of development. The tool works similarly to Hashcash, forcing browsers to solve a computational puzzle before accessing a website, which is negligible for individual users but costly at scraper scale. Site administrators use it as a stopgap measure while better methods for detecting headless browsers are developed.
anubis.techaro.lol
· 2026-09-06
GoKawiil