GoKawiilWe tested the latest frontier models gpt-5.6-terra in our agent, on five modeling and simulation problems of varying difficulty. Here are the results:
claude-fable-5 0.889 gpt-5.6-sol 0.814 gpt-5.6-terra 0.786 gpt-5.6-luna 0.727
A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth. How one physical-AI evaluation works GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best? - Blog | JuliaHub THE CONTAINER PROBLEM · SEALED workspace + ground truth CONFIG xhigh · 1M ctx · 128k HARNESS Dyad Agent MODEL gpt-5.6-terra solve THE AGENT ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ▌ grade it THE GRADER — SIMULATE BOTH, THEN COMPARE time → ground truth submitted score GRADE 1.0 0 0.96 1 · the container MODEL gpt-5.6-terra HARNESS Dyad Agent CONFIG xhigh · 1M ctx · 128k PROBLEM · SEALED workspace + ground truth ↓ 2 · the agent ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ↓ 3 · the grader Simulate both, then compare ↓ 4 · grade trajectory match 0.96 A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth. How one physical-AI evaluation works GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best? - Blog | JuliaHub THE CONTAINER PROBLEM · SEALED workspace + ground truth CONFIG xhigh · 1M ctx · 128k HARNESS Dyad Agent MODEL gpt-5.6-terra solve THE AGENT ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ▌ grade it THE GRADER — SIMULATE BOTH, THEN COMPARE time → ground truth submitted score GRADE 1.0 0 0.96 1 · the container MODEL gpt-5.6-terra HARNESS Dyad Agent CONFIG xhigh · 1M ctx · 128k PROBLEM · SEALED workspace + ground truth ↓ 2 · the agent ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ↓ 3 · the grader Simulate both, then compare ↓ 4 · grade trajectory match 0.96 A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth. How one physical-AI evaluation works GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best? - Blog | JuliaHub THE CONTAINER PROBLEM · SEALED workspace + ground truth CONFIG xhigh · 1M ctx · 128k HARNESS Dyad Agent MODEL gpt-5.6-terra solve THE AGENT ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ▌ grade it THE GRADER — SIMULATE BOTH, THEN COMPARE time → ground truth submitted score GRADE 1.0 0 0.96 1 · the container MODEL gpt-5.6-terra HARNESS Dyad Agent CONFIG xhigh · 1M ctx · 128k PROBLEM · SEALED workspace + ground truth ↓ 2 · the agent ▸ read problem ▸ derive physics ▸ write dyad/model.dyad ▸ compile ▸ simulate · verify ↓ 3 · the grader Simulate both, then compare ↓ 4 · grade trajectory match 0.96
Physical AI lives or dies on whether the modeled physics is correct. A model of an aircraft, a separation column or a charged particle can compile and run cleanly while the physics it encodes is impossible. Agentic AI makes this failure mode worse, because agents steer by feedback from tests, and the tests are often written by the same agent. In domains like web development or compilers that loop works well enough, since correct behavior is contained and checkable. In engineering, the real question is “does this match the real world?”, and that is much harder to close: the agent can pass every test it wrote while those tests rest on simplifications that don’t hold in the regime where the model will actually be used.
There is also a trust problem with the numbers that already exist. Nobody takes the model providers’ self-reported benchmarks at face value, and for good reason: the provider publishing the score is the same party with every incentive to tune for it. Our incentives point the other way. JuliaHub ships Dyad with multiple agent backends across vendors, and we win when our users get the best possible Dyad experience, whichever lab’s model delivers it. If one model is better at physical modeling, it is in our interest to find that out and recommend it. So we investigated: which one actually is?
To measure which frontier model handles that work best, we hold everything else still. The Dyad AI agent harness - purpose-built for modeling & simulation workflows - is pinned across every trial, along with the problems, the reasoning effort (xhigh), the context window (1M) and the token budget (128k). The only variable is the model: OpenAI’s GPT 5.6 family (terra, sol, luna) against Anthropic’s claude-fable-5. Three trials per model on each of the four core problems, one long-horizon trial each on the fifth: 52 graded runs in all.
01 · The problems
How Physical AI Evaluation Works.
We ground our exploration in a subset of our internal evals: five sealed problems drawn from the daily work of modeling & simulation. We order them by difficulty, defined in terms of the stages a correct solution must chain through, the careful details where a small slip is fatal, the easy shortcuts within reach, and the engineering work around the modeling itself - from parsing specs to repairing harnesses. We selected each for the same property: they admit solutions that compile and run green while being physically wrong. Grading therefore ignores the code and scores the full simulated trajectory against sealed ground truth. The hardest of the five is NASA’s HL-20 flight vehicle. An easier version of the problem is walked through in detail in this video ; the study runs a harder, sealed variant.
P1 Constitutive consistency P2 Constrained consistency P3 Relativistic dynamics P4 Steady-state linearization P5 Long-horizon flight vehicle Every previous problem at onceRead the engineering spec and aerodynamic data, derive six-degree-of-freedom motion, simulate, verify, and iterate. P1 Constitutive consistency P2 Constrained consistency P3 Relativistic dynamics P4 Steady-state linearization P5 Long-horizon flight vehicle Every previous problem at onceRead the engineering spec and aerodynamic data, derive six-degree-of-freedom motion, simulate, verify, and iterate. P1 Constitutive consistency P2 Constrained consistency P3 Relativistic dynamics P4 Steady-state linearization P5 Long-horizon flight vehicle Every previous problem at onceRead the engineering spec and aerodynamic data, derive six-degree-of-freedom motion, simulate, verify, and iterate.
... continue reading