GoKawiilEngineering
Same $16.49/month server, same Traceway binary, two embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards at 100x the row count, and stores a billion metric points in 10.8 GB. Full numbers and methodology inside.
TL;DR
I spent six days working on and running a benchmark of observability data on DuckDB. I've done this with SQLite in the last blog post and I really wanted to see how DuckDB compares on a cheap CCX13 Hetzner instance. The way I've done the measurements is by implementing DuckDB as a valid storage engine for Traceway and then running its benchmarking suite. The benchmarks show that DuckDB's columnar engine is able to query 100x more data points without any major backend changes. The write throughput is also 3x to 15x higher. The result is that you can now self-host the full OTel stack that can handle a large data volume on a pretty small server. Keep reading to find out how I've done the measurements!
This is what came back:
SQLite (post 2) DuckDB (this post) Change Metrics writes 61,712 pts/sec 254,242 pts/sec 4x Spans writes 30,508 spans/sec 95,737 spans/sec 3x Logs writes 4,877 rec/sec 75,225 rec/sec 15x Metrics read cliff 1M rows (3.85 s median) 100M rows (3.0 s median) 100x Spans read cliff 100k rows (2.27 s median) 10M rows (902 ms median) 100x Logs read cliff 100k rows (114 ms median) 10M rows (85 ms median) 100x
A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out. It earns the name because of what sits one 10x step past it: queries don't slow down, they stop coming back.
Each signal's read cliff sits exactly 100x further out on DuckDB than on SQLite, at equal or better latency, on identical hardware. I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in under an hour into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within 100x of. Logs, the signal post 2 said to keep off SQLite entirely, are DuckDB's biggest win.
What's actually being compared
Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn't find anyone who had published numbers for it. So I ran post 2's entire methodology against the DuckDB build and put the results side by side.
... continue reading