My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

My browser WASM’t prepared for this. Using DuckDB, Apache Arrow and Web Workers in real life Introduction At Motif Analytics, we are building a highly-interactive analytics tool, which allows finding insights in relatively large datasets, fully in-browser. We have a more traditional cloud-mode too, but the “local” (fully in-browser) mode is an important way for our users to experiment with Motif and see if it matches their needs, without making any commitments on their end. To make that possible, we employ several techniques, which include running DuckDB WASM, Apache Arrow, and Web Workers. During the past months we learned some upsides and downsides of using these technologies. While this experience was gained on our specific use-cases, we believe that it is still somewhat universal and might be interesting for anyone working on modern, responsive web apps which process non-trivial volumes of data. What does parallelization have to do with web browsers? The idea of speeding up pr ... Read full article.