GPT-6 Astra outperforms GPT-5.6 Sol on Zstd and IMAP coding evals
An evaluation comparing GPT-6 Astra to GPT-5.6 Sol on software test/verification tasks found Astra achieved over 99% full test correctness on a Zstd implementation task at medium effort for about $1, versus Sol's 63-77% correctness at $3.50-$5.50. The Zstd eval has become saturated for Astra, nearly maxing out regardless of testing technique used, so evaluators also tried an IMAP RFC implementation task that had been too difficult for Sol to attempt meaningfully.
GoKawiil's interpretation of the reporting above, not reported fact.
The saturation of the Zstd benchmark suggests evaluators now need harder tasks to meaningfully differentiate model capability, which is why the IMAP RFC test was introduced. The observation that Astra applies techniques like fuzzing more purposefully than Sol hints at improved reasoning about testing strategy, though the evaluators note they cannot yet confirm this translates into measurably better correctness since the original benchmark no longer shows meaningful variation.
- GPT-6 Astra reached over 99% test correctness on the Zstd eval at roughly $1, versus Sol's 63-77% at higher cost.
- The Zstd implementation benchmark is now saturated for Astra, prompting use of a harder IMAP RFC task.
- Astra appears to apply techniques like structured fuzzing more effectively than Sol, though impact on correctness scores remains unclear.
Source: danluu.com, 2026-11-16
Published there as: “Test/verification technique effectiveness with Astra”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.