Researchers used Lawrence Livermore National Laboratory’s (LLNL) exascale supercomputer El Capitan to perform the largest fluid dynamics simulation ever — surpassing one quadrillion degrees of freedom in a single computational fluid dynamics (CFD) problem. The team focused the effort on rocket–rocket plume interactions.
El Capitan is funded by the National Nuclear Security Administration’s (NNSA) Advanced Simulation and Computing (ASC) program. The work — in part performed prior to the transition of the world’s most powerful supercomputer to classified operations earlier this year — is led by researchers from Georgia Tech and supported by partners at AMD, NVIDIA, HPE, Oak Ridge National Laboratory (ORNL) and New York University’s (NYU) Courant Institute.
The paper is a finalist for the 2025 ACM Gordon Bell Prize, the highest honor in high-performance computing. This year’s winner — selected from a small handful of finalists — will be announced at the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC25) in St. Louis on Nov. 20.
To tackle the extreme challenge of simulating the turbulent exhaust flow generated by many rocket engines firing simultaneously, the team’s approach combined a newly proposed shock-regularization technique called Information Geometric Regularization (IGR), invented and implemented by professors Spencer Bryngelson of Georgia Tech, Florian Schäfer of NYU Courant and Ruijia Cao (now a Cornell Ph.D. student).
Using all 11,136 nodes and more than 44,500 AMD Instinct MI300A Accelerated Processing Units (APUs) on El Capitan, the team achieved better than 500 trillion grid points, or 500 quadrillion degrees of freedom. They further extended this to ORNL’s Frontier, surpassing one quadrillion degrees of freedom. The simulations were conducted with MFC, a permissively licensed open-source code maintained by Bryngelson’s group. With these simulations, they represented the full exhaust dynamics of a complex configuration inspired by SpaceX’s Super Heavy booster.
The simulation sets a new benchmark for exascale CFD performance and memory efficiency. It also paves the way for computation-driven rocket design, replacing costly and limited physical experiments with predictive modeling at unprecedented resolution, according to the team.
Georgia Tech’s Bryngelson, the project’s lead, said the team used specialized techniques to make efficient use of El Cap’s architecture.
“In my view, this is an intriguing and marked advance in the fluid dynamics field,” Bryngelson said. “The method is faster and simpler, uses less energy on El Capitan, and can simulate much larger problems than prior state-of-the-art — orders of magnitude larger.”
The team accessed El Capitan via prior collaborations with LLNL researchers and worked with LLNL’s El Capitan Center of Excellence and HPE to use the machine on the classified network. LLNL facilitated the effort as part of system-scale stress testing ahead of El Capitan's classified deployment, serving as a public example of the full capabilities of the system before it was turned over for classified use in support of the NNSA’s core mission of stockpile stewardship.
“We supported this work primarily to evaluate El Capitan’s scalability and system readiness,” said Livermore Computing’s Development Environment Group Leader Scott Futral. “The biggest benefit to the ASC program was uncovering system software and hardware issues that only appear when the full machine is exercised. Addressing those challenges was critical to operational readiness.”
... continue reading