GoKawiilThe “small web” is bigger than you might think
There are currently several initiatives that attempt to reclaim some part of the Internet for non-commercial, personal use; some of these I described in this earlier article. Here I’m using the term “small web” to mean the use of ordinary web browsers and servers, but for private sites, free of advertising and corporate tracking.
I’ve also recently adopted the Gemini protocol, which uses completely different protocols and software from the regular web. The Gemini protocol is so limited that it’s almost incapable of commercial exploitation which is, of course, the main part of its appeal. It’s hard to assess the number of people using Gemini; there seem to be about 6,000 Gemini “capsules” (sites) world-wide, but many of them seem to be defunct. Gemini online forums typically seem to have perhaps a hundred or so active users. It’s a small community, mostly populated by IT professionals at present.
What got me thinking about the size of the small web was my use of feed aggregators in Gemini. Like regular websites, the owners of Gemini capsules often announce updates to their contents using feeds. A feed is a simple file in an agreed format that lists site updates, usually with a time-stamp for each update. As in the regular web, Gemini feeds are usually in ATOM or RSS format – XML documents with a particular layout, described by a specification.
A “feed aggregator” is a service that examines the feeds from a range of sites or capsules, and publishes updates in chronological order, typically in a way that is accessible to the public. By looking at a Gemini feed aggregator, I can see immediately what new content is available, across the whole the Gemini community.
Since there aren’t all that many active Gemini capsules, it’s practicable to list updates from many capsules – perhaps all of them – on a single page, which I find very convenient. There are several active Gemini feed aggregators, differing in how they discover, or are informed about, new content.
I look at aggregated Gemini feeds on most days, and I always find something of interest. So I wondered: could I implement something similar for the small web as a whole? Of course, aggregating updates to the whole web on a single page would be absurd, perhaps even impossible. Still, I reasoned, the small web is, well, small. Maybe there aren’t that many updates every day?
To attempt such a task, I first needed a list of sites on the small web. I wasn’t going to try to construct one myself but, fortunately, I didn’t have to. The Kagi search engine has such a list as part of its small web initiative. This list contains sites nominated by Kagi users; apart from “smallness”, one of the criteria for inclusion is that the site publishes an update feed.
Kagi’s list mostly includes private sites and blogs, although some blogs are hosted on corporate platforms like Blogger. When I last looked at it, some time last year, I think it listed about 6,000 sites. That’s about the same number as known Gemini capsules, and I rather expected many of those sites to be moribund.
Yesterday I looked at the list again, and I saw it had risen to about 32,000 entries. That’s more than I expected but, for the purposes of aggregating feeds, it’s not the number of sites, but how often they’re updated, that matters.
... continue reading