TLDR (but stay for the animations!): Lance is a successor to Iceberg / Delta Lake, more optimized for random reads, and supports adding ad-hoc columns without needing to copy all the data.
Some big things happened in the big data over object storage world in 2025:
Iceberg V3 spec got released and added cool stuff like VARIANT.
turbopuffer announced a vector search over object storages (similar to Quickwit).
Apache Fluss lets Flink manage real-time streams with tiering to object storage.
Datadog bought Quickwit.
Databricks bought Neon.
But something way bigger flew completely under my radar, most likely as I was pretty busy building at $DAY_JOB (some pretty cool stuff, I must say).
This thing is called Lance. It's a file format (like Apache Parquet), a table format (like Apache Iceberg), and a catalog spec (like Iceberg's REST catalog spec).
Lance file format
... continue reading