GoKawiilLastly, we describe the harness, tools and graders used in this benchmark to measure model performance. All models are tasked with finding materials where κ ≥ $kappa , ε 0 ≤ $epsilon , Y ≥ $youngs and G ≥ $shear . The pinned values are chosen to define the appropriate desired window, and the objective below is what the model is given.
Propose dynamically stable, novel, BEOL-compatible crystalline materials that meet all of the following targets: thermal conductivity $kappa, STATIC dielectric constant $epsilon, Young's modulus $youngs, and shear modulus $shear. Every candidate must also come with a BEOL temperature and process compatible synthesis recipe that an expert review judges WOULD ATTEMPT — a candidate whose recipe is judged not worth attempting does not count. Novelty means that the material has never been deposited as a thin-film in BEOL compatible conditions in the reported literature.
We equipped the models with a set of tools to accomplish this task. These were designed to be similar to tools that would be available to a computational materials scientist. The tools we provided the model were:
Web search (using Exa as the provider)
A coding sandbox with python and bash capabilities and some relevant materials science packages — pymatgen , mp_api and ASE
Machine learning based tools to compute 1) the dynamic stability, 2) the lattice thermal conductivity, 3) the static dielectric constant, and 4) the compliance tensor
The model was given no stopping condition, and proceeds until it hits an error or exhausts its total token budget of 100 million tokens. We use the AI Security Institute’s open source Inspect framework to benchmark these models. In the next section we discuss in more detail the implemented tools used by the model to screen the proposed candidates. Then we discuss the synthesis scoring procedure.
Tools provided
Below, we list the machine learning based tools we used in this study to compute the various properties. We leverage machine learning interatomic potentials (MLIPs), in particular the universal point edge transformer (UPET) foundation machine learning model PET-MAD . In future work direct density functional theory calculations can be incorporated in place of these MLIP calculations, or a hybrid approach can be taken.
Dynamic stability and "harmonic" lattice thermal conductivity We use Pheasy and Phonopy to generate random configurations to fit the second order force constants using the compressed sensing method . This allows us to identify all the phonon modes. Any imaginary modes below − 1 THz ( − 4.14 meV) are determined to be dynamically unstable. We evaluate the energy and forces of every random configuration using PET-MAD. From quantities available from a harmonic phonon calculation in a unit cell volume V , with specific heat capacity C V ( 𝐪 , ν ) and group velocity 𝐯 g ( 𝐪 , ν ) of mode ν and irreducible wave vector 𝐪 with weight w 𝐪 , we approximate the thermal conductivity κ e s t as κ e s t = τ V ∑ 𝐪 w 𝐪 ∑ 𝐪 ∑ ν w 𝐪 C V ( 𝐪 , ν ) | 𝐯 g ( 𝐪 , ν ) | 2 , where τ is treated as a constant. This effectively screens out structures with extremely flat bands, avoiding the more expensive relaxation time computation. Lattice thermal conductivity (LTC) We use Pheasy and Phonopy to generate random configurations to fit the second and third order force constants using the compressed sensing method . We approximate the LTC in the three phonon scattering picture. To obtain the thermal conductivity we solve the Boltzmann transport equation in the relaxation time approximation . We include isotope effects in our evaluation of LTC, and neglect the electronic contribution due to the expected high band gap (low dielectric constant) of the proposed candidates. Static dielectric constant We use a General Materials Tensor Network (GMTNet) to obtain the static dielectric constant ε 0 , by fitting to the JARVIS DFPT database . Instead of fitting the static dielectric constant directly, we train two separate models — one for the electronic/high frequency dielectric constant ε ∞ , and one for the Born effective charges Z * . From these two models we can reconstruct ε i o n i c from the Γ point optical modes (computed using the MLIP above) as ( ε i o n i c ) α β = 4 π C V ∑ ν ∈ o p t i c a l Z ¯ ν , α Z ¯ ν , β ω ν 2 , where Z ¯ ν , β are phonon eigenvector weighted Born effective charges, 4 π C is the mode oscillator strength and V is the unit cell volume. We obtain the ε 0 tensor by adding the tensor contributions of ε ∞ and ε i o n i c . Mechanical properties We obtain the mechanical properties of Young’s and shear modulus from the compliance tensor of the material. This is evaluated by straining the unit cell in 12 different configurations and predicting the stresses σ associated with those configurations, using the MLIP once again.
... continue reading