When Targeted Random Testing Finds Real Security Vulnerabilities
Security vulnerabilities often hide in the corners of our code that we never think to test. We write unit tests for the happy path, maybe a few edge cases we can imagine, but what about the inputs we'd never consider? Many times we assume that LLMs are handling these scenarios by default, however LLM-generated code can be as prone to contain bugs or vulnerabilities as human-written code. What happens when a user enters a malicious string into your application?
This is exactly what happened when we tested building a storage service for a chat application using AI with Kiro’s latest GA features. Following a specification-driven development (SDD) workflow, Kiro carefully defined the requirements, extracted testable properties, and implemented what seemed like straightforward code for storing and retrieving API keys. The implementation looked solid. Code review would likely have approved it. Traditional unit tests would have passed.
But on the 75th iteration of a property-based test, something unexpected happened: the entire round-trip property failed. What should have been a simple save-and-retrieve operation instead exposed a mishandling of javascript prototypes—a bug that can lead to security issues in the future if you’re not careful to eliminate the flaw early on.
This post tells the story of how Property-Based Testing (PBT) caught a security bug that human intuition and traditional testing methods would likely have missed. We'll walk through:
The specification and property that Kiro defined
The seemingly innocent implementation that contained a critical flaw
How PBT's systematic exploration of the input space uncovered the vulnerability
The fix that addresses the vulnerability
Why this matters for building secure software
... continue reading