4 ms·
I added this to a benchmark I've been doing of how well agents find security bugs, specifically security bugs originally found by Mythos. It performs poorly wit
by SwellJoe 4mo ago
I added this to a benchmark I've been doing of how well agents find security bugs, specifically security bugs originally found by Mythos. It performs poorly with only read/grep/ls tools, but in a follow-up test with a full shell and Python, it doubled its findings (still a poor showing, but it does at least indicate it is doing what it says on the tin: making tools to help it solve problems). It also did worse than Qwen AgentWorld, another recent post-train of Qwen 3.6 MoE intended for agentic use.
https://swelljoe.com/post/will-it-mythos/ https://swelljoe.com/post/will-it-mythos/
- kordlessagain 4mo agoGood to know. Thanks for the research!
- hedgehog 4mo agoIt would be really interesting to see how the Qwen 3.6 35B model compares to the 27B on your benchmark.