4 ms·
We hid backdoors in binaries – Opus 4.6 found 49% of them
- stared 8mo agoSee our BinaryAudit, in which we gave AI agents access to Ghidra by NSA to find malware in raw machine code: https://quesma.com/benchmarks/binaryaudit/ https://quesma.com/benchmarks/binaryaudit/ All tasks are open-source & we welcome contributions: https://github.com/QuesmaOrg/BinaryAudit https://github.com/QuesmaOrg/BinaryAudit Discussion on X: https://x.com/pmigdal/status/2021244382800760873 https://x.com/pmigdal/status/2021244382800760873
- bell-cot 8mo agoKudos for the article very quickly getting to the important stuff: > However, this approach is not ready for production. Even the best model, Claude Opus 4.6, found relatively obvious backdoors in small/mid-size binaries only 49% of the time. Worse yet, most models had a high false positive rate — flagging clean binaries.