Tech News
← Home  ·  All topics

Web Scraping

24 GoKawiil briefs on this topic

AI scraping bots overload university library servers, disrupting public access

Academic libraries across North America, Europe and Oceania report being hit by waves of data-scraping bots seeking content for AI companies' large language models. Staff at institutions including San Jose State University and University of Victoria say the bot traffic has slowed or crashed digital archives, disrupting access for students, researchers and the public. Researchers say the problem emerged in late 2024 and has been worsening since.

Reddit to require login for site access, CEO Steve Huffman confirms

Reddit will end its two-decade practice of allowing anonymous browsing, requiring users to log in to view posts and discussions. CEO Steve Huffman says the change will first be tested in select markets outside the US in the coming months, with no firm timeline for a full rollout. Reddit is also rereleasing an incognito mode for its app.

Meta may delay new business terms after publisher backlash over web-scraping rights

Meta spokesperson Andy Stone said the company may postpone implementation of controversial new business terms beyond their current November 20 start date, pending a full consultation with publishers and industry groups. The terms, already delayed once from an original October 30 date, would let Meta use automated scraping tools on the websites of businesses that use Facebook. Meta has not changed the actual wording of the terms despite the planned consultation.

Media groups' push for Stealth Bot Prohibition Act draws pushback over scraping rules

More than 300 media industry representatives lobbied Congress in late September for the Stealth Bot Prohibition Act, aimed at curbing automated AI data-harvesting that drains referral traffic and ad revenue from news sites. Critics Lila Bailey and Tess Vartanian argue the bill would instead require anyone running an automated web crawler to disclose their identity and purpose, a mandate they say would hinder legitimate activities like investigative journalism, archiving and research.

Meta publishes its own policy governing AI agent behavior amid scraping disputes

Meta has issued its own set of rules defining what its AI agents are permitted to do, stepping into a regulatory vacuum where no industry-wide standard exists for AI agent conduct. The move comes as publishers accuse Meta of scraping their content without permission or compensation, deepening tensions between tech platforms and content creators.

Meta pushes back new commercial terms of use to November 20

Meta has postponed the rollout of its updated Commercial Terms and Self-Serve Ad Terms, originally set for October 30, to November 20. The company says the extra time will let it clarify to partners what is actually changing, after businesses objected to language permitting Meta's crawlers to collect and use content from business accounts' web domains.

Meta delays new business account terms by three weeks after publisher backlash

Meta Platforms has postponed the rollout of new commercial terms of service for business accounts following criticism from news publishers. The terms, announced earlier this month, were said by critics to grant Meta broad rights to crawl and scrape web content linked to those accounts. Meta has pushed the effective date back three weeks but says it still intends to implement the policy.

Meta revises business-account terms but AI scraping concerns persist, lawyers say

Meta edited the business terms for its platform, removing explicit language saying it could 'train' AI models on content from linked news websites, after publishers objected. The revised terms, effective October 30th, still permit Meta to use scraped content to 'develop' business products, a term attorneys say is undefined and broad.

Meta drops 'train' from updated business account terms after publisher backlash

Meta revised commercial terms of service that were set to take effect October 30, removing language that explicitly let it use scraped content to 'train' its AI systems. The new wording instead says scraping of business accounts' associated domains will be used to 'provide, develop, improve, research, and maintain advertising and business products.' Publishers and other businesses had objected after Nieman Lab flagged the original language, which they feared gave Meta broad rights to scrape any website tied to a Facebook or Instagram business account.

More websites block AI agents from browsing on users' behalf

A growing number of websites are blocking autonomous AI agents from accessing their pages, according to reports on web traffic trends. Site operators are reportedly using technical measures to detect and restrict bot-driven agents acting on behalf of users, rather than human visitors.

Meta edits business-account terms but scraping scope remains broad, lawyers say

Meta revised upcoming terms of service for business accounts, removing the word 'train' and narrowing listed uses to 'advertising and business products.' The change followed a Nieman Lab report that the terms would let Meta broadly scrape websites linked to business accounts, potentially to train AI models; the new terms take effect October 30 with no opt-out option.

Meta deletes clause from business terms that alarmed publishers over AI scraping

Meta had updated its commercial terms of use to state it could use content from a business account's domains and URLs to develop, train and maintain its products. Legal experts cited by Nieman Lab warned this broad wording could let Meta scrape publisher websites for AI training, prompting the company to remove the provision after The Verge reported on the controversy.