8 ms·

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs

by johnfn 6mo ago

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any search process, we can't know in advance which run will succeed. Mythos scoured the entire continent for gold and found some. For these small models, the authors pointed at a particular acre of land and said "any gold there? eh? eh?" while waggling their eyebrows suggestively. For a true apples-to-apples comparison, let's see it sweep the entire FreeBSD codebase. I hypothesize it will find the exploit, but it will also turn up so much irrelevant nonsense that it won't matter.

SoftTalker 6mo ago
How much of that is simply scale? Anthropic threw probably an entire data center at analyzing a code base. Has anyone done the same with a "small" model?
- jstanley 6mo ago
  It's still useful if $20k of consultants would be less effective.
hellcow 6mo ago
It seems feasible to use a small/cheap model to flag possible vulnerabilities, and then use a more expensive model to do a second-pass to confirm those, rather than on every file. Could dramatically reduce the total cost and speed up the process.
- conception 6mo ago
  Does it? I don’t see quality from small models being high enough to be able to effectively scour a code based like this.
notnullorvoid 6mo ago
> I hypothesize it will find the exploit, but it will also turn up so much irrelevant nonsense that it won't matter. The trick with Mythos wasn't that it didn't hallucinate nonsense vulnerabilities, it absolutely did. It was able to verify some were real though by testing them. The question is if smaller models can verify and test the vulnerabilities too, and can it be done cheaper than these Mythos experiments.
- iririririr 6mo ago
  so it's just better at hallucinations, but they added discrete code that works as a fuzzer/verifier?
- bredren 6mo ago
  The article positions the smaller models as capable under expert orchestration, which to be any kind of comparable must include validation.
  - Aurornis 6mo ago
    Calling it “expert orchestration” is misleading when they were pointing it at the vulnerable functions and giving it hints about what to look for because they already knew the vulnerability.
    - cyanydeez 6mo ago
      You know for loops exist and you can run opencode against any section of code with just a small amount of templating, right? There's zero stopping you from writing a harness that does what you're saying.
- hibikir 6mo ago
  People often undervalue scaffolding. I was looking at a bug yesterday, reported by a tester. He has access to Opus, but he's looking through a single repo, and Amazon Q. It provided some useful information, but the scaffolding wasn't good enough. I took its preliminary findings into Claude Code with the same model. But in mine it knows where every adjacent system is, the entire git history, deployment history, and state of the feature flags. So instead of pointing at a vague problem, it knew which flag had been flipped in a different service, see how it changed behavior, and how, if the flag was flipped in prod, it'd make the service under testing cry, and which code change to make to make sure it works both ways. It's not as if a modern Opus is a small model: Just a stronger scaffold, along with more CLI tools available in the context. The issue here in the security testing is to know exactly what was visible, and how much it failed, because it makes a huge difference. A middling chess player can find amazing combinations at a good speed when playing puzzle rush: You are handed a position where you know a decisive combination exist, and that it works. The same combination, however, might be really hard to find over the board, because in a typical chess game, it's rare for those combinations to exist, and the energy needed to thoroughly check for them, and calculate all the way through every possible thing. This is why chess grandmasters would consider just being able to see the computer score for a position to be massive cheating: Just knowing when the last move was a blunder would be a decisive advantage. When we ask a cheap model to look for a vulnerability with the right context to actually find it, we are already priming it, vs asking to find one when there's nothing.
celeritascelery 6mo ago
That was my thought exactly. If small models can find these same vulnerabilities, and your company is trying to find vulnerabilities, why didn’t you find them?
- rakejake 6mo ago
  Maybe they did use small models but you couldn't make the front page of HN with something like this until Anthropic made a big fuss out of it. Or perhaps it is just a question of compute. Not everyone has 20k$ or the GPU arsenal to task models to find vulnerabilities which may/may not be correct? Unless Anthropic makes it known exactly what model + harness/scaffolding + prompt + other engineering they did, these comparisons are pointless. Given the AI labs' general rate of doomsday predictions, who really knows?
  - replygirl 6mo ago
    papers are always coming out saying smaller models can do these amazing and terrifying things if you give them highly constrained problems and tailored instructions to bias them toward a known solution. most of these don't make the front page because people are rightfully unimpressed
- nullsanity 6mo ago
  [dead]
- echelon 6mo ago
  Who is spending millions of dollars on small models to find vulns? Nobody else is selling here or has the budget to sell quite like this. Anthropic spends millions - maybe significantly more. Then when they know where they are, they spend $20k to show how effective it is in a patch of land. They engineered this "discovery". What the small teams are doing is fair - it's just a scaled down version of what Anthropic already did.
  - paulddraper 6mo ago
    > What the small teams are doing is fair - it's just a scaled down version of what Anthropic already did. Do they find novel items? Or do they copy the areas already found by others?
alpha_squared 6mo ago
This is addressed elsewhere in the comments, but it appears this is actually a direct comparison to how Anthropic got their Mythos headline results. https://news.ycombinator.com/item?id=47732322 https://news.ycombinator.com/item?id=47732322
- Aurornis 6mo ago
  How is that a direct comparison? The link you gave has a quote that says it’s not: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints They pointed the models at the known vulnerable functions and gave them a hint. The hint part is what really breaks this comparison because they were basically giving the model the answer.
  - cyanydeez 6mo ago
    Does no one defending mythos understand how nested foreloops work? loop through each repo: loop through each file: opencode command /find_wraparoundvulnerability next file next repo I can run this on my local LLM and sure, I gotta wait some time for it to complete, but I see zero distinguishing facts here.
    - u_fucking_dork 6mo ago
      Please do so, looking forward to your write up
    - Dylan16807 6mo ago
      The question is how customized those hints were. That changes whether looping over an entire code base is possible or not.
    - johnfn 6mo ago
      No one is saying your nested for loop idea because it won't actually work in practice. In short, the signal to noise ratio will be too high - you will need to comb through a ton of false positives in order to find anything valuable, at which point it stops looking like "automated security research" and it starts looking like "normal security research". If you don't believe me, you should try it yourself, it's only a couple of dollars. Hey, maybe you're right, and you can prove us all wrong. But I'd bet you on great odds that you're not.
yorwba 6mo ago
We don't even need to hypothesize that much on the irrelevant nonsense, since they helpfully provide data with the detected vulnerability patched: https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier#patched-freebsd-sensitivity-vs-specificity-appendix-patched-freebsd https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jag... and half of the small models they touted as finding the vulnerability still found it in the patched code in 3/3 runs. A model that finds a vulnerability 100% of the time even when there is none is just as informative as a model that finds a vulnerability 0% of the time even when there is one. You could replace it with a rock that has "There's a vulnerability somewhere." engraved on it. They're a company selling a system for detecting vulnerabilities reliant on models trained by others, so they're strongly incentivized to claim that the moat is in the system, not the model, and this post really puts the thumb on the scale. They set up a test that can hardly distinguish between models (just three runs, really??) unless some are completely broken or work perfectly, the test indeed suggests that some are completely broken, and then they try to spin it as a win anyway! A high false-positive rate isn't necessarily an issue if you can produce a working PoC to demonstrate the true positives, where they kinda-sorta admit that you might need a stronger model for this (a.k.a. what they can't provide to their customers). Overall I rate Aisle intellectually dishonest hypemongers talking their own book.
kilpikaarna 6mo ago
Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. Have Anthropic actually said anything about the amount of false positives Mythos turned up? FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with other (public) SotA models, but each turned up only a subset of the ones Mythos found. I'd say that sounds plausible from the perspective of Mythos being an incremental (though an unusually large increment perhaps) improvement over previous models, but one that also brings with it a correspondingly significant increase in complexity. So the angle they choose to use for presenting it and the subsequent buzz is at least part hype -- saying "it's too powerful to release publicly" sounds a lot cooler than "it costs $20000 to run over your codebase, so we're going to offer this directly to enterprise customers (and a few token open source projects for marketing)". Keep in mind that the examples in Nicholas Carlini's presentation were using Opus, so security is clearly something they've been working on for a while (as they should, because it's a huge risk). They didn't just suddenly find themselves having accidentally created a super hacker.
- johnfn 6mo ago
  > Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. Both are worthless without human intervention. I definitely breathed a sigh of relief when I read it was $20,000 to find these vulnerabilities with Mythos. But I also don't think it's hype. $20,000 is, optimistically, a tenth the price of a security researcher, and that shift does change the calculus of how we should think about security vulnerabilities.
  - amazingamazing 6mo ago
    Citation needed for basically all of this. You basically are creating a double standard for small models vs mythos…
deleted 6mo ago
[deleted]
letitgo12345 6mo ago
Can't you execute the bug to see if the vulnerability is real? So you have a perfect filter. Maybe Mythos decided w/o executing but we don't know that.
cyanydeez 6mo ago
so what you're saying is no one could ever write a loop like: for githubProject in githubProjects opencode command /findvulnerability end for Seems like a silly thing to try and back up.
- tredre3 6mo ago
  What he's saying is that you should read the "Caveats and limitations" section of the article. Here's the first one: > Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). Mythos did no such thing, it was cut lose and told to find vulnerabilities. If the intent was to prove that small models are just as good, they haven't demonstrated that at all. The end.
  - cyanydeez 6mo ago
    ok, but you're missing the obvious: I could also give it the vulnerable function byt just looping over all functions and providing a small hint about what to look at. Until "Mythos" is compared with the most bland and straight forward harness vs small model, there's no great context god that can't be emulated with deterministic scanning and context pulls.
WhyNotHugo 6mo ago
OTOH, this article goes too far the opposite extreme: > We isolated the vulnerable svc_rpc_gss_validate function, provided architectural context (that it handles network-parsed RPC credentials, that oa_length comes from the packet), and asked eight models to assess it for security vulnerabilities. To follow your analogy, they pointed to the exact room where the gold was hidden, and their model found it. But finding the right room within the entire continent in honestly the hard part.
- mattmanser 6mo ago
  Or would it have any way if they hadn't pointed it at it? Who knows? Just like people paid by big tobacco found no link to cancer in cigarettes, researchers paid for by AI companies find amazing results for AI. Their job literally depends on them finding Mythos to be good, we can't trust a single word they say.
  - LordDragonfang 6mo ago
    > Their job literally depends on them finding Mythos to be good, we can't trust a single word they say. TFA article is literally from a company whose business is finding vulnerabilities with other people's AI. This article is the exact kind of incentive-driven bad study you're criticizing. Hell, the subtitle is literally "Why the moat is the system, not the model". It's literally them going, "pssh, we can do that too, invest in us instead"
hoppp 6mo ago
They pay me 20k and give me time maybe I find it also.
- LordDragonfang 6mo ago
  No, you wouldn't. The vulnerability has been in the codebase for 17 years. Orders of magnitude more than 20k in security professional salary-hours have been pointed at the FreeBSD codebase over the past decade and a half, so we already know a human is unlikely to have found it in any reasonable amount of time.
glerk 6mo ago
I'm having trouble finding this info (I assume they won't publish it), but could the secret sauce be much larger and more readily accessible context window? OpenBSD's code is in the 10s of millions of lines. Being able to hold all of it in context would make bug finding much easier.
- johnfn 6mo ago
  You can look at some of the bugs, if you'd like. They are (at least the ones I looked at) fairly self-contained, scoped to a single function, a hundred lines or less. There's no need for a massive amount of context.
  - glerk 6mo ago
    Interesting, and you are absolutely right (hehe). These are pretty self-contained and seems to be something more like "formal verification" where the model is able to simulate a large number of states and find incorrect ones, if I were to speculate, something akin to a reasoning loop that moved from the harness/orchestration layer down to the model itself.
lukev 6mo ago
This is a really interesting point though -- it's really scaffold-dependent. Because for the same price, you could point the small model at each function, one by one, N times each, across N prompts instructing it to look for a specific class of issue. It's not that there's no difference between models, but it's hard to judge exactly how much difference there is when so much depends on the scaffold used. For a properly scientific test, you'd need to use exactly the same one. Which isn't possible when Anthropic won't release the model.
lmeyerov 6mo ago
Instead of scanning more code, afaict what you seem to want is instead, scan on the same small area, and compare on how many FPs are found there. A common measure here is what % of the reported issues got labeled as security issues and fixed. I don't see Mythos publishing on relative FP rate, so dunno how to compare those. Maybe something substantively changed? At the same time, I'm not sure that really changes anything because I don't see a reason to believe attacks are constrained by the quality of source code vulnerability finding tools, at least for the last 10-15 years after open source fuzzing tools got a lot better, popular, and industrialized. This might sound like a grumpy reply, but as someone on both sides here, it's easy to maintain two positions: 1. This stuff is great, and doing code reviews has been one of my favorite claude code use cases for a year now, including security review. It is both easier to use than traditional tools, and opens up higher-level analysis too. 2. Finding bugs in source code was sufficiently cheap already for attackers. They don't need the ease of use or high-level thing in practice, there's enough tooling out there that makes enough of these. Likewise, groups have already industrialized. There's an element of vuln-pocalypse that may be coming with the ease of use going further than already happening with existing out-of-the-box blackbox & source code scanning tools . That's not really what I worry about though. Scarier to me, instead, is what this does to today's reliance on human response. AI rapidly industrializes what how attackers escalate access and wedge in once they're in. Even without AI, that's been getting faster and more comprehensive, and with AI, the higher-level orchestration can get much more aggressive for much less capable people. So the steady stream of existing vulns & takeovers into much more industrialized escalations is what worries me more. As coordination keeps moving into machine speed, the current reliance on human response is becoming less and less of an option.
rakel_rakel 6mo ago
Spending $20000 (and whatever other resources this thing consumes) on a denial of service vulnerability in OpenBSD seems very off balance to me. Given the tone with which the project communicates discussing other operating systems approaches to security, I understand that it can be seen as some kind of trophy for Mythos. But really, searching the number of erratas on the releases page that include "could crash the kernel" makes me think that investing in the OpenBSD project by donating to the foundation would be better than using your closed source model for peacocking around people who might think it's harder than it is to find such a bug.
- paulddraper 6mo ago
  You don’t see the value of vulnerabilities as on the order of 20k USD? When it’s a security researcher, HN says that’s a squalid amount. But when its a model, it’s exorbitant.
  - rakel_rakel 6mo ago
    If I understand you correctly, you're asking me if I would class this as a 20k USD (plus environmental and societal impact) bug? nope, I don't. I've not said anything else than that I think this specific bug isn't worth the attention it's getting, and that 20k USD would benefit the OpenBSD project (much) more through the foundation. > When it’s a security researcher, HN says that’s a squalid amount. But when its a model, it’s exorbitant. Not sure why you're projecting this onto me, for the project in question $20k is _a_lot_. The target fundraising goal for 2025 was $400k, 5% of that goes a very long way (and yes, this includes OpenSSH).
    - vel0city 6mo ago
      > you're asking me if I would class this as a 20k USD (plus environmental and societal impact) bug? Not this bug in particular as a single bug bounty, but as an entire codebase audit that exposed multiple bugs? Sure.
  - telotortium 6mo ago
    Denial of service isn’t worth that much generally, I think - you can’t use it to directly steal data or to install a payload for later exploitation. There are usually generic ways to mitigate denial of service as well - IP blocking and the like.
mehmetkerem 6mo ago
[dead]
andy_ppp 6mo ago
I wonder if you could just setup a small model and suggest a load of things and try every file and it might still end up being cheaper and just as good as Mythos at a specific task. Maybe this will be something that holds true for more things, formulating a small model to do specific things may well end up being as effective/efficient as a larger model looking at a huge solution space.
davemp 6mo ago
> Across a thousand runs through our scaffold, the total cost was under $20,000 Lots of questions about the $20k. Is that raw electricity costs, subsidized user token costs? If so, the actual costs to run these sorts of tasks sustainably could be something like $200k. Even at $50k, a FreeBSD DoS is not an extremely competitive price. That's like 2-4mo of labor. Don't get me wrong, I think this seems like a great use for LLMs. It intuitively feels like a much more powerful form of white box fuzzing that used techniques like symbolic execution to try to guide execution contexts to more important code paths.
Sparkyte 6mo ago
Why not just write many small models for explicit tasks than running one bigger model anyway? I prefer the agentic subject matter expert design anyway. I suppose because it wants to look at the whole code base?
klempner 6mo ago
The broad answer to the "irrelevant nonsense" for something like this is to use more expensive models to validate. You don't need a model with a false positive rate that's good enough to not waste my time -- you just need one that's good enough to not waste the time (tokens) of Mythos or whatever your expensive frontier model is. Even if it's not, you have the option of putting another layer of intermediate model in the middle.
mlmonkey 6mo ago
We can reduce this to an even more basic question: if these small models are equally comparable in finding vulnerabilities, why haven't they done so yet?. After all, the source code is out in the open, and has been for decades. Please go ahead, find (and report) the vulnerabilities.
shmagadee 6mo ago
I've read this statement a bunch of times and am still unclear what it is saying. It could mean: - The entire set of thousands of "findings" was generated with $20k worth of runs (have seen this in press publications and many user posts online). - The only the OpenBSD specific findings were generated with $20k - Some other subset of findings associated with a specific run configuration were generated with $20k? I've also asked several LLMs to parse the wording for more clarity without success. They all highlight it as ambiguous wording. Why not use more direct language and provide the supporting data? They also stated that they are providing $100M in credits to their partners. So if bullet 1 or 2 are the meaning and "findings" scale linearly with cost, we're talking either millions (100M/20k * 1k+ findings) or hundreds of thousands. Does that make any sense? Or is the idea that all of these companies will run scans across their critical codebases continuously? Anyone else have a better sense of the math going on here?
AbstractH24 6mo ago
So the real learning here is the cost of “using” GenAI to do things is declining at a rapid speed. We’re not doing anything that couldn’t be done before, we’re just doing it faster, easier and cheaper. Sounds like a recipe for a lot of junk being built. Also sounds like something that’s been true since the beginning of humanity. In the more near term, sounds like a reminder the datacenters and processing boom will look at lot like the fiber one.
coldtea 6mo ago
>Mythos scoured the entire continent for gold and found some. For these small models, the authors pointed at a particular acre of land and said "any gold there? eh? eh?" while waggling their eyebrows suggestively. Which sounds trivial for a hacker wanting to find vulnerabilities to replicate, so what's the huge advantage of Mython then? That you don't need to spend 5 minutes to nudge it to the most complex/ripe for vulnerabilities parts of a codebase?

by johnfn 6mo ago

SoftTalker 6mo ago

How much of that is simply scale? Anthropic threw probably an entire data center at analyzing a code base. Has anyone done the same with a "small" model?

jstanley 6mo ago
It's still useful if $20k of consultants would be less effective.

hellcow 6mo ago

It seems feasible to use a small/cheap model to flag possible vulnerabilities, and then use a more expensive model to do a second-pass to confirm those, rather than on every file. Could dramatically reduce the total cost and speed up the process.

conception 6mo ago
Does it? I don’t see quality from small models being high enough to be able to effectively scour a code based like this.

notnullorvoid 6mo ago

> I hypothesize it will find the exploit, but it will also turn up so much irrelevant nonsense that it won't matter. The trick with Mythos wasn't that it didn't hallucinate nonsense vulnerabilities, it absolutely did. It was able to verify some were real though by testing them. The question is if smaller models can verify and test the vulnerabilities too, and can it be done cheaper than these Mythos experiments.

iririririr 6mo ago
so it's just better at hallucinations, but they added discrete code that works as a fuzzer/verifier?
bredren 6mo ago
The article positions the smaller models as capable under expert orchestration, which to be any kind of comparable must include validation.
- Aurornis 6mo ago
  Calling it “expert orchestration” is misleading when they were pointing it at the vulnerable functions and giving it hints about what to look for because they already knew the vulnerability.
  - cyanydeez 6mo ago
    You know for loops exist and you can run opencode against any section of code with just a small amount of templating, right? There's zero stopping you from writing a harness that does what you're saying.
hibikir 6mo ago
People often undervalue scaffolding. I was looking at a bug yesterday, reported by a tester. He has access to Opus, but he's looking through a single repo, and Amazon Q. It provided some useful information, but the scaffolding wasn't good enough. I took its preliminary findings into Claude Code with the same model. But in mine it knows where every adjacent system is, the entire git history, deployment history, and state of the feature flags. So instead of pointing at a vague problem, it knew which flag had been flipped in a different service, see how it changed behavior, and how, if the flag was flipped in prod, it'd make the service under testing cry, and which code change to make to make sure it works both ways. It's not as if a modern Opus is a small model: Just a stronger scaffold, along with more CLI tools available in the context. The issue here in the security testing is to know exactly what was visible, and how much it failed, because it makes a huge difference. A middling chess player can find amazing combinations at a good speed when playing puzzle rush: You are handed a position where you know a decisive combination exist, and that it works. The same combination, however, might be really hard to find over the board, because in a typical chess game, it's rare for those combinations to exist, and the energy needed to thoroughly check for them, and calculate all the way through every possible thing. This is why chess grandmasters would consider just being able to see the computer score for a position to be massive cheating: Just knowing when the last move was a blunder would be a decisive advantage. When we ask a cheap model to look for a vulnerability with the right context to actually find it, we are already priming it, vs asking to find one when there's nothing.

celeritascelery 6mo ago

That was my thought exactly. If small models can find these same vulnerabilities, and your company is trying to find vulnerabilities, why didn’t you find them?

rakejake 6mo ago
Maybe they did use small models but you couldn't make the front page of HN with something like this until Anthropic made a big fuss out of it. Or perhaps it is just a question of compute. Not everyone has 20k$ or the GPU arsenal to task models to find vulnerabilities which may/may not be correct? Unless Anthropic makes it known exactly what model + harness/scaffolding + prompt + other engineering they did, these comparisons are pointless. Given the AI labs' general rate of doomsday predictions, who really knows?
- replygirl 6mo ago
  papers are always coming out saying smaller models can do these amazing and terrifying things if you give them highly constrained problems and tailored instructions to bias them toward a known solution. most of these don't make the front page because people are rightfully unimpressed
nullsanity 6mo ago
[dead]
echelon 6mo ago
Who is spending millions of dollars on small models to find vulns? Nobody else is selling here or has the budget to sell quite like this. Anthropic spends millions - maybe significantly more. Then when they know where they are, they spend $20k to show how effective it is in a patch of land. They engineered this "discovery". What the small teams are doing is fair - it's just a scaled down version of what Anthropic already did.
- paulddraper 6mo ago
  > What the small teams are doing is fair - it's just a scaled down version of what Anthropic already did. Do they find novel items? Or do they copy the areas already found by others?

alpha_squared 6mo ago

This is addressed elsewhere in the comments, but it appears this is actually a direct comparison to how Anthropic got their Mythos headline results. https://news.ycombinator.com/item?id=47732322 https://news.ycombinator.com/item?id=47732322

Aurornis 6mo ago
How is that a direct comparison? The link you gave has a quote that says it’s not: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints They pointed the models at the known vulnerable functions and gave them a hint. The hint part is what really breaks this comparison because they were basically giving the model the answer.
- cyanydeez 6mo ago
  Does no one defending mythos understand how nested foreloops work? loop through each repo: loop through each file: opencode command /find_wraparoundvulnerability next file next repo I can run this on my local LLM and sure, I gotta wait some time for it to complete, but I see zero distinguishing facts here.
  - u_fucking_dork 6mo ago
    Please do so, looking forward to your write up
  - Dylan16807 6mo ago
    The question is how customized those hints were. That changes whether looping over an entire code base is possible or not.
  - johnfn 6mo ago
    No one is saying your nested for loop idea because it won't actually work in practice. In short, the signal to noise ratio will be too high - you will need to comb through a ton of false positives in order to find anything valuable, at which point it stops looking like "automated security research" and it starts looking like "normal security research". If you don't believe me, you should try it yourself, it's only a couple of dollars. Hey, maybe you're right, and you can prove us all wrong. But I'd bet you on great odds that you're not.

yorwba 6mo ago

We don't even need to hypothesize that much on the irrelevant nonsense, since they helpfully provide data with the detected vulnerability patched: https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier#patched-freebsd-sensitivity-vs-specificity-appendix-patched-freebsd https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jag... and half of the small models they touted as finding the vulnerability still found it in the patched code in 3/3 runs. A model that finds a vulnerability 100% of the time even when there is none is just as informative as a model that finds a vulnerability 0% of the time even when there is one. You could replace it with a rock that has "There's a vulnerability somewhere." engraved on it. They're a company selling a system for detecting vulnerabilities reliant on models trained by others, so they're strongly incentivized to claim that the moat is in the system, not the model, and this post really puts the thumb on the scale. They set up a test that can hardly distinguish between models (just three runs, really??) unless some are completely broken or work perfectly, the test indeed suggests that some are completely broken, and then they try to spin it as a win anyway! A high false-positive rate isn't necessarily an issue if you can produce a working PoC to demonstrate the true positives, where they kinda-sorta admit that you might need a stronger model for this (a.k.a. what they can't provide to their customers). Overall I rate Aisle intellectually dishonest hypemongers talking their own book.

kilpikaarna 6mo ago

Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. Have Anthropic actually said anything about the amount of false positives Mythos turned up? FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with other (public) SotA models, but each turned up only a subset of the ones Mythos found. I'd say that sounds plausible from the perspective of Mythos being an incremental (though an unusually large increment perhaps) improvement over previous models, but one that also brings with it a correspondingly significant increase in complexity. So the angle they choose to use for presenting it and the subsequent buzz is at least part hype -- saying "it's too powerful to release publicly" sounds a lot cooler than "it costs $20000 to run over your codebase, so we're going to offer this directly to enterprise customers (and a few token open source projects for marketing)". Keep in mind that the examples in Nicholas Carlini's presentation were using Opus, so security is clearly something they've been working on for a while (as they should, because it's a huge risk). They didn't just suddenly find themselves having accidentally created a super hacker.

johnfn 6mo ago
> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. Both are worthless without human intervention. I definitely breathed a sigh of relief when I read it was $20,000 to find these vulnerabilities with Mythos. But I also don't think it's hype. $20,000 is, optimistically, a tenth the price of a security researcher, and that shift does change the calculus of how we should think about security vulnerabilities.
- amazingamazing 6mo ago
  Citation needed for basically all of this. You basically are creating a double standard for small models vs mythos…

deleted 6mo ago

[deleted]

letitgo12345 6mo ago

Can't you execute the bug to see if the vulnerability is real? So you have a perfect filter. Maybe Mythos decided w/o executing but we don't know that.

cyanydeez 6mo ago

so what you're saying is no one could ever write a loop like: for githubProject in githubProjects opencode command /findvulnerability end for Seems like a silly thing to try and back up.

tredre3 6mo ago
What he's saying is that you should read the "Caveats and limitations" section of the article. Here's the first one: > Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). Mythos did no such thing, it was cut lose and told to find vulnerabilities. If the intent was to prove that small models are just as good, they haven't demonstrated that at all. The end.
- cyanydeez 6mo ago
  ok, but you're missing the obvious: I could also give it the vulnerable function byt just looping over all functions and providing a small hint about what to look at. Until "Mythos" is compared with the most bland and straight forward harness vs small model, there's no great context god that can't be emulated with deterministic scanning and context pulls.

WhyNotHugo 6mo ago

OTOH, this article goes too far the opposite extreme: > We isolated the vulnerable svc_rpc_gss_validate function, provided architectural context (that it handles network-parsed RPC credentials, that oa_length comes from the packet), and asked eight models to assess it for security vulnerabilities. To follow your analogy, they pointed to the exact room where the gold was hidden, and their model found it. But finding the right room within the entire continent in honestly the hard part.

mattmanser 6mo ago
Or would it have any way if they hadn't pointed it at it? Who knows? Just like people paid by big tobacco found no link to cancer in cigarettes, researchers paid for by AI companies find amazing results for AI. Their job literally depends on them finding Mythos to be good, we can't trust a single word they say.
- LordDragonfang 6mo ago
  > Their job literally depends on them finding Mythos to be good, we can't trust a single word they say. TFA article is literally from a company whose business is finding vulnerabilities with other people's AI. This article is the exact kind of incentive-driven bad study you're criticizing. Hell, the subtitle is literally "Why the moat is the system, not the model". It's literally them going, "pssh, we can do that too, invest in us instead"

hoppp 6mo ago

They pay me 20k and give me time maybe I find it also.

LordDragonfang 6mo ago
No, you wouldn't. The vulnerability has been in the codebase for 17 years. Orders of magnitude more than 20k in security professional salary-hours have been pointed at the FreeBSD codebase over the past decade and a half, so we already know a human is unlikely to have found it in any reasonable amount of time.

glerk 6mo ago

I'm having trouble finding this info (I assume they won't publish it), but could the secret sauce be much larger and more readily accessible context window? OpenBSD's code is in the 10s of millions of lines. Being able to hold all of it in context would make bug finding much easier.

johnfn 6mo ago
You can look at some of the bugs, if you'd like. They are (at least the ones I looked at) fairly self-contained, scoped to a single function, a hundred lines or less. There's no need for a massive amount of context.
- glerk 6mo ago
  Interesting, and you are absolutely right (hehe). These are pretty self-contained and seems to be something more like "formal verification" where the model is able to simulate a large number of states and find incorrect ones, if I were to speculate, something akin to a reasoning loop that moved from the harness/orchestration layer down to the model itself.

lukev 6mo ago

This is a really interesting point though -- it's really scaffold-dependent. Because for the same price, you could point the small model at each function, one by one, N times each, across N prompts instructing it to look for a specific class of issue. It's not that there's no difference between models, but it's hard to judge exactly how much difference there is when so much depends on the scaffold used. For a properly scientific test, you'd need to use exactly the same one. Which isn't possible when Anthropic won't release the model.

lmeyerov 6mo ago

Instead of scanning more code, afaict what you seem to want is instead, scan on the same small area, and compare on how many FPs are found there. A common measure here is what % of the reported issues got labeled as security issues and fixed. I don't see Mythos publishing on relative FP rate, so dunno how to compare those. Maybe something substantively changed? At the same time, I'm not sure that really changes anything because I don't see a reason to believe attacks are constrained by the quality of source code vulnerability finding tools, at least for the last 10-15 years after open source fuzzing tools got a lot better, popular, and industrialized. This might sound like a grumpy reply, but as someone on both sides here, it's easy to maintain two positions: 1. This stuff is great, and doing code reviews has been one of my favorite claude code use cases for a year now, including security review. It is both easier to use than traditional tools, and opens up higher-level analysis too. 2. Finding bugs in source code was sufficiently cheap already for attackers. They don't need the ease of use or high-level thing in practice, there's enough tooling out there that makes enough of these. Likewise, groups have already industrialized. There's an element of vuln-pocalypse that may be coming with the ease of use going further than already happening with existing out-of-the-box blackbox & source code scanning tools . That's not really what I worry about though. Scarier to me, instead, is what this does to today's reliance on human response. AI rapidly industrializes what how attackers escalate access and wedge in once they're in. Even without AI, that's been getting faster and more comprehensive, and with AI, the higher-level orchestration can get much more aggressive for much less capable people. So the steady stream of existing vulns & takeovers into much more industrialized escalations is what worries me more. As coordination keeps moving into machine speed, the current reliance on human response is becoming less and less of an option.

rakel_rakel 6mo ago

Spending $20000 (and whatever other resources this thing consumes) on a denial of service vulnerability in OpenBSD seems very off balance to me. Given the tone with which the project communicates discussing other operating systems approaches to security, I understand that it can be seen as some kind of trophy for Mythos. But really, searching the number of erratas on the releases page that include "could crash the kernel" makes me think that investing in the OpenBSD project by donating to the foundation would be better than using your closed source model for peacocking around people who might think it's harder than it is to find such a bug.

paulddraper 6mo ago
You don’t see the value of vulnerabilities as on the order of 20k USD? When it’s a security researcher, HN says that’s a squalid amount. But when its a model, it’s exorbitant.
- rakel_rakel 6mo ago
  If I understand you correctly, you're asking me if I would class this as a 20k USD (plus environmental and societal impact) bug? nope, I don't. I've not said anything else than that I think this specific bug isn't worth the attention it's getting, and that 20k USD would benefit the OpenBSD project (much) more through the foundation. > When it’s a security researcher, HN says that’s a squalid amount. But when its a model, it’s exorbitant. Not sure why you're projecting this onto me, for the project in question $20k is _a_lot_. The target fundraising goal for 2025 was $400k, 5% of that goes a very long way (and yes, this includes OpenSSH).
  - vel0city 6mo ago
    > you're asking me if I would class this as a 20k USD (plus environmental and societal impact) bug? Not this bug in particular as a single bug bounty, but as an entire codebase audit that exposed multiple bugs? Sure.
- telotortium 6mo ago
  Denial of service isn’t worth that much generally, I think - you can’t use it to directly steal data or to install a payload for later exploitation. There are usually generic ways to mitigate denial of service as well - IP blocking and the like.

mehmetkerem 6mo ago

[dead]

andy_ppp 6mo ago

I wonder if you could just setup a small model and suggest a load of things and try every file and it might still end up being cheaper and just as good as Mythos at a specific task. Maybe this will be something that holds true for more things, formulating a small model to do specific things may well end up being as effective/efficient as a larger model looking at a huge solution space.

davemp 6mo ago

> Across a thousand runs through our scaffold, the total cost was under $20,000 Lots of questions about the $20k. Is that raw electricity costs, subsidized user token costs? If so, the actual costs to run these sorts of tasks sustainably could be something like $200k. Even at $50k, a FreeBSD DoS is not an extremely competitive price. That's like 2-4mo of labor. Don't get me wrong, I think this seems like a great use for LLMs. It intuitively feels like a much more powerful form of white box fuzzing that used techniques like symbolic execution to try to guide execution contexts to more important code paths.

Sparkyte 6mo ago

Why not just write many small models for explicit tasks than running one bigger model anyway? I prefer the agentic subject matter expert design anyway. I suppose because it wants to look at the whole code base?

klempner 6mo ago

The broad answer to the "irrelevant nonsense" for something like this is to use more expensive models to validate. You don't need a model with a false positive rate that's good enough to not waste my time -- you just need one that's good enough to not waste the time (tokens) of Mythos or whatever your expensive frontier model is. Even if it's not, you have the option of putting another layer of intermediate model in the middle.

mlmonkey 6mo ago

We can reduce this to an even more basic question: if these small models are equally comparable in finding vulnerabilities, why haven't they done so yet?. After all, the source code is out in the open, and has been for decades. Please go ahead, find (and report) the vulnerabilities.

shmagadee 6mo ago

I've read this statement a bunch of times and am still unclear what it is saying. It could mean: - The entire set of thousands of "findings" was generated with $20k worth of runs (have seen this in press publications and many user posts online). - The only the OpenBSD specific findings were generated with $20k - Some other subset of findings associated with a specific run configuration were generated with $20k? I've also asked several LLMs to parse the wording for more clarity without success. They all highlight it as ambiguous wording. Why not use more direct language and provide the supporting data? They also stated that they are providing $100M in credits to their partners. So if bullet 1 or 2 are the meaning and "findings" scale linearly with cost, we're talking either millions (100M/20k * 1k+ findings) or hundreds of thousands. Does that make any sense? Or is the idea that all of these companies will run scans across their critical codebases continuously? Anyone else have a better sense of the math going on here?

AbstractH24 6mo ago

So the real learning here is the cost of “using” GenAI to do things is declining at a rapid speed. We’re not doing anything that couldn’t be done before, we’re just doing it faster, easier and cheaper. Sounds like a recipe for a lot of junk being built. Also sounds like something that’s been true since the beginning of humanity. In the more near term, sounds like a reminder the datacenters and processing boom will look at lot like the fiber one.

coldtea 6mo ago

>Mythos scoured the entire continent for gold and found some. For these small models, the authors pointed at a particular acre of land and said "any gold there? eh? eh?" while waggling their eyebrows suggestively. Which sounds trivial for a hacker wanting to find vulnerabilities to replicate, so what's the huge advantage of Mython then? That you don't need to spend 5 minutes to nudge it to the most complex/ripe for vulnerabilities parts of a codebase?