Tech News
← Home  ·  All topics

Terminal Bench

5 GoKawiil briefs on this topic

Anthropic launches Opus 5.5, OpenAI debuts GPT-6 Sol and Luna at lower prices

Anthropic released Opus 5.5, an update to Claude aimed at enterprise tasks like coding, financial analysis and business work, claiming better agentic coding benchmarks than GPT-6 Astra and reduced token pricing compared to Opus 5. OpenAI simultaneously released GPT-6 Sol and GPT-6 Luna, cheaper successors to GPT-5.6 that the company says cut factual errors roughly in half and match rival Fable 5.1's coding performance at lower cost.

Benchmark tests find RTK's token-saving claims don't hold up on cost

RTK, a popular tool with over 79,000 GitHub stars that compresses terminal output before AI coding agents read it, has been marketed as a way to cut AI coding costs, with one viral post claiming up to 60% token savings. But independent benchmark testing using Terminal-Bench 2.1 across 1,740 task attempts found mixed results: costs dropped 5% for one model setup but rose 5% for another, contradicting the widely shared savings figures.

Cognition's SWE-2 model scores 92.8 on Terminal-Bench 2.1, trails on longer tasks

Cognition released SWE-2, a coding model built by post-training on a 2.8-trillion-parameter Kimi K3 base with 104B active parameters, marking its first RL push into the multi-trillion-parameter range. The model posts a class-leading 92.8 on Terminal-Bench 2.1 and competitive FrontierCode scores at lower cost than rivals like Claude Fable 5.1 and GPT-6 Astra, but falls well behind both on the harder Terminal-Bench 4.0 benchmark (27.3 vs 55.8 and 57.9). SWE-2 is closed-weight and available now through Devin Desktop and CLI, with no local deployment option.

Qwen3.8 27B: 4-bit quantization matches full model, 1-bit quantization fails badly

A benchmarking study tested several GGUF quantizations of the Qwen3.8 27B model against benchmarks including GPQA Diamond, IFBench, and Terminal-Bench 2.1. The 17GB Q4_K_M version performed on par with the full 55GB BF16 model on coding tasks and fits on a consumer 24GB GPU, while the smallest 6.2GB 1-bit version dropped to near-random accuracy on reasoning tasks, with longer reasoning chains making results worse.

New Terminal-Bench-Science benchmark shows top AI agent solving just 30% of research tasks

Stanford researchers, working with the Terminal-Bench team and scientists across disciplines, launched Terminal-Bench-Science, a benchmark testing AI agents on real scientific research workflows. The initial version includes 70 tasks spanning life, physical, Earth, mathematical, and engineering sciences, with Claude Opus 5 running Claude Code topping the leaderboard at a 30% resolution rate.