GoKawiilAt Modal, we build sandboxes, among other things. Agents run in sandboxes, and agents are eating software. Today, Modal runs millions of sandboxes per day, supports up to fifty thousand concurrent sandboxes per customer, and supports a variety of use cases at scale, from reinforcement learning to background agents .
Increasingly, our users require more and more sandboxes, created at higher and higher rates. Reinforcement learning can require running millions of sandboxes concurrently, and creating bursts of hundreds of thousands of sandboxes at the beginning of rollouts. Similarly, agents increasingly require massive scale and high concurrent creation rates to deal with traffic bursts.
Our existing sandbox platform is really good, but it wasn’t designed for these scales; nor is any other existing solution. We’re obsessed with scale and performance, and we want our infrastructure to accelerate the growth of agents, not add friction. So we went back to the drawing board.
Over the last few months, we’ve rebuilt our core sandbox platform from the ground up for both scale and reliability. On our new system, users can run millions of sandboxes concurrently and create tens of thousands of sandboxes per second. We’ve removed all central bottlenecks from our control plane so there are no practical scaling limits, and we’ve optimized every part of container scheduling and startup, simplifying the scheduling path to a layer of load balancers which create containers directly on our worker fleet.
As a demonstration of what our platform is capable of, we’ve run a million sandboxes concurrently, creating all 1 million in under a minute.
Evidence that we can run a lot of sandboxes.
Why most solutions don’t scale
Running 1 million sandboxes pushes the limits of any container platform, both because of the sheer number of containers, but also because running this many sandboxes requires many tens of thousands of compute nodes. There will be many operations which are either O(containers), O(nodes), or both, which will cause traditional container platforms to hit scaling limits.
For Kubernetes, as an example:
The scheduling algorithm is O(n x p) for n nodes and p pods in the worst case, and scheduling is serialized by default.
... continue reading