ScopeDB team builds Rust port of Apache DataSketches, reverse-engineers CPC sketch logic
Developers working on ScopeDB's distinct-counting feature moved from Cloudflare's cardinality-estimator crate to Apache DataSketches, then collaborated with other Rust engineers to create datasketches-rust, starting with an HllSketch implementation. The team is now working through CpcSketch, a more space-efficient algorithm based on Kevin J. Lang's 2017 cardinality estimation research, originally implemented in OCaml and Java/C++ but lacking a Rust version.
GoKawiil's interpretation of the reporting above, not reported fact.
The effort suggests a gap in Rust's data-sketching ecosystem that practitioners are filling by porting established Java/C++ algorithms rather than building from scratch, which could improve reliability for systems like ScopeDB that need auditable, memory-safe approximate counting. Because CPC sketches rely on intricate bit-matrix logic from older academic code, this kind of reverse-engineering work could also make Lang's algorithm more accessible and maintainable for other Rust-based data systems.
- ScopeDB switched its approx_count_distinct feature to a new Rust implementation of Apache DataSketches.
- The datasketches-rust project began with HllSketch and is now tackling the more complex CpcSketch.
- CpcSketch is based on Kevin J. Lang's 2017 cardinality estimation algorithm, previously only available in OCaml, Java and C++.
Source: tisonkun.io, 2026-11-16
Published there as: “Reverse-Engineering Kevin J. Lang's Lost OCaml Code in Apache DataSketches”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.