GoKawiilStudy finds most testing prompts barely improve AI coding agents' Zstd implementations
A researcher evaluated 26 different prompt conditions—ranging from 'use TDD' to formal methods like Lean 4, TLA+, and SMT solvers—to see whether simply instructing coding agents to apply specific testing techniques or libraries improves the correctness of Rust implementations of Zstd. Four popular testing 'skills,' including ones with hundreds of thousands of GitHub stars, were also tested alongside a self-written skill, using the same Zstd benchmark from prior agentic programming comparisons.
GoKawiil's interpretation of the reporting above, not reported fact.
As AI coding agents become more common, many developers assume that telling an agent to 'use property-based testing' or 'use fuzzing' will meaningfully boost code quality, but this experiment probes whether that assumption holds. The findings matter because if lightweight technique-naming prompts don't reliably improve correctness, teams relying on such shortcuts may be getting a false sense of security about AI-generated code quality.
- 26 different prompt addendums, from TDD to formal verification tools like TLA+ and Verus, were tested on the same Zstd implementation task.
- Four externally popular testing skills, including one with 250k GitHub stars, were compared against a custom-written skill.
- The study responds to an earlier observation that software quality seems to be declining despite agents having easier access to effective test techniques.
Programming Rust" by Jim Blandy (O'Reilly) — Since every implementation in this eval was written in Rust, this O'Reilly book is a solid way to build the expertise needed to judge whether an agent's code — and its tests — are actually correct. It covers ownership, traits, error handling and concurrency in depth, which is exactly the background you need when reviewing generated compression or protocol code.
See Programming Rust" by Jim Blandy (O'Reilly) on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: danluu.com, 2026-09-08
Published there as: “How well do agents use test/verification techniques?”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.