GoKawiil01 Introduction
Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product. Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet. Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services. Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there. Can a coding agent actually do the work of a software engineer in the real world?
1 Fable 5.1 Claude Code Resolution rate: 38.8% 2 GPT-6 Astra Codex CLI Resolution rate: 33.8% 3 Gemini 3.8 Flash Gemini CLI Resolution rate: 31.2% 4 GLM 5.3 Claude Code Resolution rate: 28.8% =5 Grok 4.6 Grok Build Resolution rate: 23.8% =5 Muse Spark 1.3 Muse Code Resolution rate: 23.8% 7 Kimi K3 Kimi Code Resolution rate: 18.8% 8 GPT-5.6 Sol Codex CLI Resolution rate: 16.2% # Model Harness Resolution rate 1 Fable 5.1 Fable 5.1 Claude Code 38.8% 2 GPT-6 Astra GPT-6 Astra Codex CLI 33.8% 3 Gemini 3.8 Flash Gemini 3.8 Flash Gemini CLI 31.2% 4 GLM 5.3 GLM 5.3 Claude Code 28.8% =5 Grok 4.6 Grok 4.6 Grok Build 23.8% =5 Muse Spark 1.3 Muse Spark 1.3 Muse Code 23.8% 7 Kimi K3 Kimi K3 Kimi Code 18.8% 8 GPT-5.6 Sol GPT-5.6 Sol Codex CLI 16.2% Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown.
Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models. We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.
Real company tasks require company-specific context
Correct billing depends on business rules and external services Fix invoice billing so each business charges the right tax and exempt customers aren't taxed. View full instruction Hide full instruction ▾ Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL. Services in the sandbox TJ TaxJar sandbox
TaxJar sandbox TJ TaxJar production
TaxJar production InfluxDB ledger
... continue reading