GoKawiilDeepSeek V4.1 Flash produced an extraordinary result in our AI hacking benchmark. It gained code execution on all 11 vulnerable targets, while all four fixed targets remained secure. The accepted runs cost only $4.65.
A perfect score at that price deserves a detailed review. We looked into every command, request, and successful attack. The review confirmed six solutions that followed the planned attack path, and it also found five successful routes that the original scoring system did not distinguish from the planned solutions.
The result gave us two useful insights. DeepSeek showed strong hacking ability and the review showed where the benchmark needed stricter checks.
A large attack run for less than five dollars
DeepSeek worked inside isolated copies of Grafana, Jenkins, and Nextcloud. It read source code, compared vulnerable and fixed versions, started services, sent requests, tested ideas, and changed its approach when an attempt failed.
Across the full benchmark, the model used 2,349 Bash commands and almost two hours and 38 minutes of active model time. The median successful run took four minutes and 38 seconds. The provider reported 268.3 million input tokens and about two million output tokens.
Caching explains much of the low cost. Of the 268.3 million input tokens, 266.2 million were cached. The provider charged a lower price for this reused input. The accepted runs cost $4.65. Failed attempts and replacement runs increased the complete cost to $5.14.
DeepSeek completed a large amount of work for a remarkably low price.
Grafana fell in under 90 seconds
The Grafana challenge tested a problem in the plugin installation process. The planned attack used a file-path handling issue to place code in a protected location.
... continue reading