GoKawiilNew Real-SWE benchmark tests AI coding agents on licensed enterprise codebases
A new benchmark called Real-SWE evaluates frontier AI coding models against tasks drawn from real, private production codebases licensed from actual companies, including billing, tax and customer-migration work. Top performer Fable 5.1 running on Claude Code resolved 38.8% of tasks, followed by GPT-6 Astra and Gemini 3.8 Flash, with several other models trailing well below that mark.
GoKawiil's interpretation of the reporting above, not reported fact.
Most existing coding benchmarks rely on public repositories that models may have already seen during training, inflating perceived competence. By using proprietary, business-critical code with company-specific conventions, Real-SWE offers a tougher and more realistic gauge of whether AI agents can genuinely replace or assist engineers on production systems, and the modest resolution rates suggest current models still struggle with real enterprise complexity.
- Real-SWE tests AI models on private, licensed enterprise codebases rather than public repos.
- Fable 5.1 on Claude Code led with a 38.8% resolution rate, the highest among tested models.
- Even top models solved fewer than 40% of tasks, showing real-world enterprise coding remains challenging for AI.
O'Reilly "Designing Data-Intensive Applications" by Martin Kleppmann — If benchmarks like Real-SWE have you thinking about billing systems, migrations and multi-service complexity, this book is the classic deep dive into how real production systems are actually built. It's a great companion for engineers evaluating what AI agents can and can't handle in messy enterprise codebases.
See O'Reilly "Designing Data-Intensive Applications" by Martin Kleppmann on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: withspecific.com, 2026-09-12
Published there as: “Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.