The Art of Managing Skunks

Since moving from academic research to industry in 2017, I’ve worked on two software projects. Each one started as a small, clean-slate skunkworks effort involving 2-3 people and gradually expanded to a large, conventional software engineering effort with dozens of engineers. The first of these (from 2017 to 2021) was Delos at Meta, a Chubby/ZooKeeper/etcd-like control plane storage system. The second was a new Kafka engine (from 2022 to 2024) that can run on any disaggregated storage layer (and powers the Confluent Freight product, where S3 is used as that storage layer). Nearly every system at Meta depends in some way on Delos as of 2025 (e.g., this article describes an example dependency chain); Confluent Freight just became generally available and time will tell if it succeeds commercially, though early results are promising. While these systems were technically difficult to build and operate (particularly given their critical roles in the stacks of the respective companies), I fo ... Read full article.