For those who enjoy Counter-Strike community servers, the situation in Counter-Strike 2 is rather dire.
An avalanche of spam has rendered the server browser unusable. The transition from Global Offensive killed multiple small communities. And large server providers have taken advantage of these problems to monopolise the market.
Trying to find a server either involves capitulating to these big vendors, or trawling through a trench of spam.
Scraping the server browser allows us to have some insight into the state of the market.
Getting the servers
The first task was extracting servers from Valve’s master list.
This is not a trivial matter: Valve provide a REST endpoint for doing so, but the endpoint is restricted to ~10-50k results per query, and does not provide any pagination support. This is because the data is extremely volatile, changing on every single query.
You can provide query parameters to change what servers are returned. By performing multiple segmented queries (i.e. by polling for maps individually, querying by different regions), and polling over a long period of time, we can gather a lot more servers.
The best way to approach this technique would have been to replicate Steam’s server browser. However, this would still require segmentation (due to the volume of servers), and has a messier authentication story. ] After pulling this data, I threw it into a DuckDB database to analyse.
The spam problem
... continue reading