GoKawiilOpenAI showed GPT-6 Astra working in KiCad. We have been building a way to measure whether the circuits that come out of these models are any good.
We got pretty excited yesterday when OpenAI put a demo of GPT-6 Astra working on a circuit board in KiCad on the front page of its launch post. It is cool to see electronics show up in a major model release like this.
We are obviously still some distance from asking an AI to build an entire phone in one prompt. The demo does raise a question we have been thinking about for a while, though: how do we measure whether the electronics an AI produces are actually any good?
The models know a surprising amount about electronics
Our experience has been that current models know much more about electronics than their output in conventional design tools tends to show. They have read textbooks, datasheets, application notes and a lot of code.
You can have an agent operate a graphical CAD tool, but it spends a lot of time clicking around and keeping track of what is on screen. A lot of its context consists of coordinates, menus and application state.
EEBench uses atopile instead. The circuit lives in declarative code, so the agent can work directly on components, connections and electrical constraints. It can change the design, build it, run a simulation and inspect what failed without leaving the project.
This has worked much better for us than asking a model to draw lines in a GUI. It also means the benchmark can spend less time testing computer use and more time testing electronics.
A small part of the starter design for one public EEBench task, in ato v2 . ELEC : @STD :: Import { . project &= "electronics" . org &= "atopile" } . Submission : @type { . vin : ELEC :: ElectricPower . vhold : ELEC :: ElectricPower . vhold.lv ~ . vin.lv . c_bank : ELEC :: Capacitor { . capacitance &= 22uF +/- 20% . max_voltage &= 10V .. 25V . temperature_coefficient &= "X5R" . package &= "0805" } . vhold.hv ~> . c_bank ~> . vhold.lv }
The real world is messy
... continue reading