GoKawiilWelcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!
Residential proxies
As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term "residential proxies" is used to describe systems that are used in this way.
There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.
More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
The second sort of operator works more overtly, pretending to a degree of legitimacy and offering "ethically sourced" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a "free" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.
There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.
Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt . They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.
... continue reading