We are happy to announce the open-source release of opsqueue , our opinionated queueing system!
Why would you want to use it?
Lightweight: small codebase, written in Rust, minimal dependencies
Optimized for batch processing: we prioritize throughput over latency
Built to scale to billions of operations
Built with reliable building blocks: Rust, SQLite, Object Storage (such as S3 or GCS)
Operationally simple: single binary, embedded database, minimal configuration
Scales horizontally: you can have many consumers processing work in parallel
Very flexible: you have full control over how you produce and consume operations. We use a novel prioritization approach where decisions can be made in the middle of ongoing work.
opsqueue is a good choice if you have a use case where you first generate a few million operations (an "operation" is any task that can be executed within a few seconds) and then later execute those operations. In our case, and we expect many other use-cases as well, the systems which generate the operations are often different from the systems that execute them. We usually want to have specialized consumers that maximize throughput (within certain constraints, like e.g. rate-limits) by processing operations in parallel.
... continue reading