GoKawiil1Password's AI patch-benchmark study 'FLAWED' criticized for errors and thin citations
A security researcher publicly challenged 1Password's Off-by-1 Labs report 'Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D,' echoing earlier critiques from Trail of Bits and Davi Ottenheimer that called its AI patching benchmark misleading. The researcher's Twitter thread cited arithmetic mistakes, inconsistent diagrams, and a citation list of just 19 sources—mostly corporate blogs—compared to 73 in the concurrent academic PatchBench paper.