GoKawiilAI agents tend to be, at least as of this writing, lousy at writing tests. The tests they write are often vague, cryptic, overcomplicated, hacky, disorganized, tautological, performative, perfunctory and downright pointless.
Unfortunately, I don't expect uncoached agents to get much better at writing tests anytime soon, because the agents learned by human-written example, and the human-written examples out there are often, I'm sorry to say, just as bad. Not only are the tests written by "amateurs" often poor quality, but, sadly, the testing practices preached by teachers are often pretty bad as well. It's truly rough out there.
The good news is that I've found that, with a bit of guidance, agents are capable of following a rational TDD process and of writing clear, meaningful tests. What exactly is that guidance? The short answer, which can serve as a close enough approximation to the truth, is Kent Beck's Canon TDD. If you give your agent a skill that says nothing more than "Follow Kent Beck's Canon TDD" then I suspect you'll be a good 60% of the way there. The longer answer is what I've baked into my own personal TDD skill.
My TDD skill
Since it's a living document, I don't want to bake my TDD skill into this blog post and freeze it in time. Instead, you can see my TDD skill here on GitHub. Having said that, I can certainly share the essence of the skill here since I'm sure that that isn't going to change.
First I clue the agent in to what I call the specify-encode-fulfill loop, which is my personal alternative to red-green-refactor. Specify-encode-fulfill (SEF) goes like this:
Specify: Come up with the specifications for what you want to build Encode: Encode those specifications as automated tests (executable specifications) Fulfill: Write the code to fulfill the specifications
SEF is the high-level view of what, to me, TDD is all about. At a slightly lower level is Kent Beck's Canon TDD, which I've described below in my own words.
Write a list of the specifications within scope of the current TDD session Encode each item in the list as an automated test Change the code just barely enough to make the current test failure go away. Avoid "speculative coding" - if we write more code than necessary to make the current test failure go away, we risk having code never exercised by any test Optionally refactor, but not before committing the behavior change. Never mix behavior changes with refactoring Until the list is empty, go back to #2
My TDD skill contains a bit more detail but this is the essence of the process. This process doesn't have much influence over the design of the tests themselves, though, so I have a different skill for that, Test Design Review. Test Design Review spawns a separate agent (in an effort to avoid bias), looks for violations of design principles (such as case is where a test focusing on means rather than ends) and makes suggestions for fixes. Sometimes the "fixes" are dubious but usually they're on the mark. When I'm not satisfied with the way my agent has written a certain test, I run Test Design Review to try to let the agent catch its own mistakes.
... continue reading