GoKawiil← Writing
September 6, 2026
tl;dr: As of version 0.1.4 , the gearhash crate has gained a NEON backend which makes it roughly 2× faster on ARM64 at typical chunk sizes. It is selected automatically on aarch64 and backwards compatible, so consumers of the crate don't need to do more than just update. Read on if you're interested in the details of how this was achieved, or skip straight to the final results.
At the end of 2019, I was building a personal backup system, and as part of this, became interested in a technique called content-defined chunking. The key idea behind it is that instead of chunking files on fixed chunk boundaries, you run a sliding window hash function across the file and trigger a chunk boundary whenever the hash has a particular value. The downside is that this gives you variable length chunks over a distribution, but the upside is that your chunking is now much more resilient to byte sequences being inserted or removed from the middle of files.
Anyway, as part of this I came across the FastCDC paper. Its building block is the GEAR rolling hash. Because I like fast things, I spent quite some time trying to work out how to convert the serial algorithm published in the paper into a SIMD algorithm. I ended up publishing the result of this as gearhash , a small Rust crate with optimizations for SSE4.2 and AVX2.
When I wrote the crate, ARM64 was not really a target worth optimizing for. AWS had offered ARM64 instances for a year, but only the first-generation Graviton A1 family, built on Cortex-A72 cores and marketed for scale-out workloads rather than general compute. Graviton2, the first generation with a competitive core, was announced at re:Invent the same month as my first commit and did not reach general availability until May 2020. Apple announced the M1 in November 2020.
Fast forward to today, a lot has changed. Apple has pushed ARM64 into the mainstream of consumer hardware. AWS has shipped several further Graviton generations and says that for three years running more than half of the new CPU capacity it added has been Graviton. GitHub Actions added free ARM64 runners for public repositories in 2025. On all of those machines, the gearhash crate was falling back to the scalar loop.
On top of this, while gearhash initially had virtually no production users aside from myself, it has since become a core part of the Xet client, Hugging Face's storage protocol for large files on the Hub, which has replaced Git LFS as the default. For gearhash this means we're now doing between 10k and 20k downloads per day. This renewed interest in the crate helped me find the motivation to see where I can push things further.
The gear hash kernel is defined as a serial function over 64-bit unsigned integers:
hash = ( hash << 1 ) . wrapping_add ( table [ byte as usize ]);
... continue reading