Polars Cloud: The Distributed Cloud Architecture to Run Polars Anywhere

1. The DataFrame scale gap When I started working on Polars, I was surprised how much DataFrame implementations differed from SQL and databases. SQL could run anywhere 1. It could run embedded, on a client server model, or on full OLAP data warehouses. Whereas for dataframes, the API was different per use case and performance was drastically lacking behind SQL solutions. Locally, pandas was dominant, and remotely/distributed, it was PySpark. For end-users, pandas was very easy to get up and running, but it seems to have ignored what databases have learned over decades, there was no query optimization, poor data type implementation, many needless materializations, it offloaded memory handling to NumPy, and a few other design decisions that led to poor scaling and inconsistent behavior. PySpark was much closer to databases, it follows the relational model, has optimization, a distributed query engine and scaled properly. However PySpark is written in Scala, requires the JVM to run loca ... Read full article.