This is a short note to document storing data in JS’s bigint type. I tried this out in a project recently, and it’s not obviously a terrible idea, as long as you take into account all the caveats that make it only useful for very specific cases.
This is a performance thing. In the project where I was exploring this, we had a large amount of objects representing configuration values. The possible configuration keys were the same, but different objects had different keys set. This approach was causing some issues:
Serialization and deserialization produced very large strings because of all the repeated keys.
Comparison/equality of objects was costly because of the number of keys.
We needed to do a lot of operations like “find the subset of keys and values that are identical between all of these objects”. This was slow.
We often needed to update subsets of the items by adding, updating, or removing keys, so each object needed to be an independent clone.
Originally, I planned on doing some sort of object interning to reduce the number of objects we needed to keep in memory at any one time. We would keep only one copy of each unique configuration item, and just reference it in multiple locations. The (now dead) Records and Tuples proposal would have been really useful here, and its replacement, Composites looks like it might help but I didn’t want to tie myself to an incomplete early-stage proposal. So if I went down this route, I’d be building something myself.
While looking all this up, I found Justin Fagnani’s article Composite Map Keys in JavaScript with Bitsets, which is mostly unrelated to what I’m doing here, but did make me think of using bigints in the first place.
The basic idea is:
Define a bunch of fields, how they should be interpreted, their size, etc. E.g. {name: "size", kind: "uint", bits: 8} to represent a number stored as a u8. You also need to know the offset of each field, this can be calculated from the index of each field and the number of bits in the preceding fields. Add functions getBits(bigint, offset, bits) which does bitshifting to fetch the relevant bits from the bigint backing store, and setBits(bigint, offset, bits, value) which does the opposite For each field, add getters and setters that do the type conversions from e.g. JS number to u8.
... continue reading