GoKawiilThe core of nixpkgs-multiverse, when you strip away the Nix API and the CLI, is an index. It is a map from (attribute, version) to the revision that shipped it as a JSON file.11There are actually a few other files that drive other features such as the statistics or “fast mode”, but they are all JSON as well.
$ ls -lh index/ -rw-r--r--. 1 fmzakari fmzakari 7.5M Aug 19 13:57 history.json -rw-r--r--. 1 fmzakari fmzakari 5.3M Aug 19 13:57 versions.json
As of 9cc0209, versions.json is 5.3 MiB and history.json is 7.5MiB covering 305,492 package versions across 31,904 packages and 1,534 revisions.
The Nix API loads the JSON files lazily and are all read via builtins.fromJSON :
index = builtins . fromJSON ( builtins . readFile ./index/versions.json );
I would like to enrich the data with even more information however it comes at a cost: mo’data, mo’problems.
The goal of the project is to minimize the number of Nixpkgs that are downloaded. If we merely swap fetching huge Nixpkgs for huge JSON, it’s not a clear win.
For now we have to be judicious about what we store in the JSON files and think of clever encoding schemes to make the data small and compact.
If we were not constrained to the Nix builtins , we would leverage established technologies to efficiently encode our dataset that allow multiple query access patterns: databases!
Let’s say we were not restricted to JSON, do we have any other options?
... continue reading