GoKawiilStarSkirmish Bench ranks LLMs on coding StarCraft bots in one hour
A new benchmark called StarSkirmish Bench v0.2 tests flagship LLMs from OpenAI, Anthropic, Google and Meta by giving each one hour to write a Protoss bot in C++ for StarCraft: Brood War, then scoring win rates against human-written and demo bots. GPT-6 Astra topped the leaderboard, with GPT-6.1 Sol second, while Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 scored closely behind between 37 and 39. Even the top model, GPT-6 Astra, won only 7% of games against the strongest human-written bots like Stardust, PurpleWave and BananaBrain.
GoKawiil's interpretation of the reporting above, not reported fact.
The benchmark suggests current LLMs still lag far behind specialized human-written game AI when forced to reason and code under a tight time limit, indicating long-horizon agentic coding remains a weak spot even for top models. The cost comparison, showing GPT-6.1 Sol achieving near-top performance at a fraction of GPT-6 Astra's price, could matter for developers weighing performance against API expense. The authors note scores show no sign of saturating, implying there's substantial room for improvement before LLMs rival expert-crafted game bots.
- GPT-6 Astra leads the StarSkirmish Bench v0.2 rankings, ahead of GPT-6.1 Sol in second place.
- GPT-6.1 Sol delivers near-top results at roughly one-tenth the API cost of GPT-6 Astra per run.
- Even the best-performing model wins only 7% of games against top human-written StarCraft bots.
Source: starskirmish.com — Kai Mcpheeters, 2026-10-16
Published there as: “Show HN: Gemini 4 Argon plays StarCraft”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.