Google DeepMind’s work with AlphaFold has been nothing short of a miracle, but it is computationally expensive. With that in mind, Apple researchers set off to develop an alternative method to use AI to predict the 3D structure of proteins, and it shows promise. Here are the details.
If you’re not familiar with AlphaFold, this is Google DeepMind’s groundbreaking AI model that can predict the 3D structure of a protein from its amino acid sequence. This has been especially valuable in helping develop more effective drugs, as well as completely new materials.
Until a few years ago, this used to be an incredibly hard problem. Predicting the three-dimensional atomic structure of a single protein could take months, and even years.
But thanks to AlphaFold, and now AlphaFold2, as well as other state-of-the-art models such as RoseTTAFold, and ESMFold, this prediction process takes as little as a few hours, or even minutes, depending on the hardware.
Each of these models employs its own methods and frameworks to achieve such high accuracy, but in general, they require extremely costly calculations, and their frameworks have a very strict structure.
As Apple’s researchers put it:
“Established protein folding models like AlphaFold2 and RoseTTAFold have achieved groundbreaking accuracy by relying on carefully engineered architectures that integrate computationally heavy domain-specific designs for protein folding tasks such as multiple sequence alignments (MSAs) of AA sequences, pair representations, and triangle updates. These design choices (MSA, pair representations, triangular updates, etc.) are an attempt to hard-code our current understanding of the underlying structure generation process into these models, instead of opting to let models to learn this directly from data, which could be beneficial for a variety of reasons.”
Enter Apple’s SimpleFold
In its proposed model, rather than relying on “MSA, pairwise interaction maps, triangular updates or any other equivariant geometric modules,” Apple relies on so-called flow matching models, which were introduced in 2023 and have proven very popular for text-to-image and text-to-3D models.
In a nutshell, flow matching models are an evolution of the diffusion models that we covered in this post. But instead of simply iteratively removing noise from an initial image, they learn a smoother path that turns random noise straight into a finished image in one go.
And because this method skips many of the denoising steps, it is less computationally expensive, and generates results faster.
Apple’s researchers trained SimpleFold at multiple different sizes, including 100M, 360M, 700M, 1.1B, 1.6B, and 3B parameters, and evaluated them on “two widely adopted protein structure prediction benchmarks: CAMEO22 and CASP14, which are rigorous tests for generalization, robustness, and atomic-level accuracy in folding models.”
The results were very promising:
“Despite its simplicity, SimpleFold achieves competitive performance compared with these baselines. In both benchmarks, SimpleFold shows consistently better performance than ESMFlow which is also a flow-matching model built with ESM embeddings. On CAMEO22, SimpleFold demonstrates comparable results to the best folding models (e.g., ESMFold, RoseTTAFold2, and AlphaFold2). In particular, SimpleFold achieves over 95% performance of RoseTTAFold2/AlphaFold2 on most metrics without applying expensive and heuristic triangle attention and MSA.”
And
“For completeness, we report results of SimpleFold using different model sizes. The smallest model SimpleFold-100M shows competitive performance given its advantage of efficiency in both training and inference. In particular, SimpleFold achieves more than 90% of the performance ESMFold on CAMEO22, which demonstrates the effectiveness of building a folding model using general purpose architectural blocks.”
They also saw performance improvements aligned with scaling, which means that bigger models with more training data reliably deliver better folding performance, especially on the most challenging benchmarks.
Finally, they note that SimpleFold is just a first step, and say that they “hope [it] serves as an initiative for the community to build efficient and powerful protein generative models.”
You can read the full study on arXiv.
Accessory deals on Amazon