5 ms·

This is from the first of the caveats that they list: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.

by elicash 6mo ago

This is from the first of the caveats that they list: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints. The models' performance here is an upper bound on what they'd achieve in a fully autonomous scan. That said, a well-designed scaffold naturally produces this kind of scoped context through its targeting and iterative prompting stages, which is exactly what both AISLE's and Anthropic's systems do. That's why their point is what the subheadline says, that the moat is the system, not the model. Everybody so far here seems to be misunderstanding the point they are making.

wat10000 6mo ago
If that’s the case, why didn’t they do it that way?
- e12e 6mo ago
  Tunnel vision? If your model can handle big context, why divide into lesser problems to conquer - even if such splitting might be quite trivial and obvious? It's the difference of "achieve the goal", and "achieve the goal in this one particular way" (leverage large context).
  - wat10000 6mo ago
    I meant, if the claim here is that small models can accomplish the same things with good scaffolding, why didn’t they demonstrate finding those problem with good scaffolding rather than directly pointing them at the problem?
    - mattmanser 6mo ago
      They don't have to. Lot of people in this thread don't seem to be getting that. If another model can find the vulnerability if you point it at the right place, it would also find the vulnerability if you scanned each place individually. People are talking about false positives, but that also doesn't matter. Again, they're not thinking it through. False positives don't matter, as you can just automatically try and exploit the "exploit" and if it doesn't work, it's a false positive. Worse, we have no idea how Mythos actually worked, it could have done the process I've outlined above, "found" 1,000s of false positives and just got rid of them by checking them. The fundamental point is it doesn't matter how the cheap models identified the exploit, it's that they can identify the exploit. When it turns out the harness is just acting as a glorified for-each brute force, it's not the model being intelligent, it's simply the harness covering more ground. It's millions of monkeys bashing type-writers, not Shakespeare at one.
      - Oranguru 6mo ago
        [dead]
      - LordDragonfang 6mo ago
        > If another model can find the vulnerability if you point it at the right place, it would also find the vulnerability if you scanned each place individually. They didn't just point it at the right place, they pointed it at the right place and gave it hints. That's a huge difference, even for humans.
      - wat10000 6mo ago
        It’s strange to see this constant “I could do that too, I just don’t want to” response. Finding an important decades-old vulnerability in OpenBSD is extremely impressive. That’s the sort of thing anyone would be proud to put on their resume. Small models are available for anyone to use. Scaffolding isn’t that hard to build. So why didn’t someone use this technique to find this vulnerability and make some headlines before Anthropic did? Either this technique with small models doesn’t actually work, or it does work but nobody’s out there trying it for some reason. I find the second possibility a lot less plausible than the first.
        cycomanic 6mo ago
        From the article: >At AISLE, we've been running a discovery and remediation system against live targets since mid-2025: 15 CVEs in OpenSSL (including 12 out of 12 in a single security release, with bugs dating back 25+ years and a CVSS 9.8 Critical), 5 CVEs in curl, over 180 externally validated CVEs across 30+ projects spanning deep infrastructure, cryptography, middleware, and the application layer. They have been doing it (and likely others as well), but they are not anthropic which a million dollar marketing budget and a trillion dollar hype behind it, so you just didn't hear about it.
        roywiggins 6mo ago
        They could have linked their replication in this blog post, which we did all see, if they have one.
anotheryou 6mo ago
huh, running it over each function in theory but testing just the specific ones here makes sense, but that hint?!
- elicash 6mo ago
  I agree. To clarify, I don't necessarily agree with the post or their approach. I just thought folks were misreading it. I also think it adds something useful to the conversation.
TacticalCoder 6mo ago
> That's why their point is what the subheadline says, that the moat is the system, not the model. Can you expand a bit more on this? What is the system then in this case? And how was that model created? By AI? By humans?
- SCHiM 6mo ago
  You can imagine a pipeline that looks at individual source files or functions. And first "extracts" what is going on. You ask the model: - "Is the code doing arithmetic in this file/function?" - "Is the code allocating and freeing memory in this file/function?" - "Is the code the code doing X/Y/Z? etc etc" For each question, you design the follow-up vulnerability searchers. For a function you see doing arithmetic, you ask: - "Does this code look like integer overflow could take place?", For memory: - "Do all the pointers end up being freed?" _or_ - "Do all pointers only get freed once?" I think that's the harness part in terms of generating the "bug reports". From there on, you'll need a bunch of tools for the model to interact with the code. I'd imagine you'll want to build a harness/template for the file/code/function to be loaded into, and executed under ASAN. If you have an agent that thinks it found a bug: "Yes file xyz looks like it could have integer overflow in function abc at line 123, because...", you force another agent to load it in the harness under ASAN and call it. If ASAN reports a bug, great, you can move the bug to the next stage, some sort of taint analysis or reach-ability analysis. So at this point you're running a pipeline to: 1) Extract "what this code does" at the file, function or even line level. 2) Put code you suspect of being vulnerable in a harness to verify agent output. 3) Put code you confirmed is vulnerable into a queue to perform taint analysis on, to see if it can be reached by attackers. Traditionally, I guess a fuzzer approached this from 3 -> 2, and there was no "stage 1". Because LLMs "understand" code, you can invert this system, and work if up from "understanding", i.e. approach it from the other side. You ask, given this code, is there a bug, and if so can we reach it?, instead of asking: given this public interface and a bunch of data we can stuff in it, does something happen we consider exploitable?
  - ang_cire 6mo ago
    That's funny, this is how I've been doing security testing in my code for a while now, minus the 'taint analysis'. Who knew I was ahead of the game. :P In all seriousness though, it scares me that a lot of security-focused people seemingly haven't learned how LLMs work best for this stuff already. You should always be breaking your code down into testable chunks, with sets of directions about how to chunk them and what to do with those chunks. Anyone just vaguely gesturing at their entire repo going, "find the security vulns" is not a serious dev/tester; we wouldn't accept that approach in manual secure coding processes/ SSDLCs.
tehryanx 6mo ago
I get what you're saying, but I think this is still missing something pretty critical. The smaller models can recognize the bug when they're looking right at it, that seems to be verified. And with AISLE's approach you can iteratively feed the models one segment at a time cheaply. But if a bug spans multiple segments, the small model doesn't have the breadth of context to understand those segments in composite. The advantage of the larger model is that it can retain more context and potentially find bugs that require more code context than one segment at a time. That said, the bugs showcased in the mythos paper all seemed to be shallow bugs that start and end in a single input segment, which is why AISLE was able to find them. But having more context in the window theoretically puts less shallow bugs within range for the model. I think the point they are making, that the model doesn't matter as much as the harness, stands for shallow bugs but not for vulnerability discovery in general.
- cyanydeez 6mo ago
  OK, consider a for loop that goes through your repo, then goes through each file, and then goes through each common vulnerability... Is Mythos some how more powerful than just a recursive foreloop aka, "agentic" review. You can run `open code run --command` with a tailored command for whatever vulnerabilities you're looking for.
  - tehryanx 6mo ago
    newer models have larger context windows, and more stable reasoning across larger context windows. If you point your model directly at the thing you want it to assess, and it doesn't have to gather any additional context you're not really testing those things at all. Say you point kimi and opus at some code and give them an agentic looping harness with code review tools. They're going to start digging into the code gathering context by mapping out references and following leads. If the bug is really shallow, the model is going to get everything it needs to find it right away, neither of them will have any advantage. If the bug is deeper, requires a lot more code context, Opus is going to be able to hold onto a lot more information, and it's going to be a lot better at reasoning across all that information. That's a test that would actually compare the models directly. Mythos is just a bigger model with a larger context window and, presumably, better prioritization and stronger attention mechanisms.
lelanthran 6mo ago
> That's why their point is what the subheadline says, that the moat is the system, not the model. I'm skeptical; they provided a tiny piece of code and a hint to the possible problem, and their system found the bug using a small model. That is hardly useful, is it? In order to get the same result , they had to know both where the bug is and what the bug is. All these companies in the business of "reselling tokens, but with a markup" aren't going to last long. The only strategy is "get bought out and cash out before the bubble pops".
epistasis 6mo ago
If that's the point they are making, let's see their false positive rate that it produces on the entire codebase. They measured false negatives on a handful of cases, but that is not enough to hint at the system you suggest. And based on my experiences with $$$ focused eval products that you can buy right now, e.g. greptile, the false positive rate will be so high that it won't be useful to do full codebase scans this way.
- zelphirkalt 6mo ago
  How do we know the false positives for this "Mythos" thingamabob? Since they didn't release it, and we cannot reproduce it, are we to simply believe their word on this? What if the author of the featured article simply made a claim about that? We also simply believe their word? To me these AI tech companies are not any more trustworthy than a random blog author, maybe even less so, due to all the shady stuff they are pulling and especially since they have not released. Show or it didn't happen.
  - epistasis 6mo ago
    That they were able to use it for security scanning puts the false positive rate at a useable level, inherently. Maybe they spent more on labor to comb through reports than they did on the hardware costs of discovery, but if so I think we'd be hearing from third parties about how useless those millions in Mythos credits were that they got.
nimchimpsky 6mo ago
[dead]
LordDragonfang 6mo ago
> That said, a well-designed scaffold naturally produces this kind of scoped context through its targeting and iterative prompting stages, which is exactly what both AISLE's and Anthropic's systems do. Unless the context they added to get the small model to find it was generated fully by their own scaffold (which I assume it was not, since they'd have bragged about it if it was), either they're admitting theirs isn't well designed, or they're outright lying. People aren't missing the point, they're saying the point is dishonest.
scotty79 6mo ago
> Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). To be fair, nothing stops anyone from feeding each function of given codebase separately with one out of the predefined set of hints. It's just AST and a for loop. Calling it a system is a bit much.

by elicash 6mo ago

wat10000 6mo ago

If that’s the case, why didn’t they do it that way?

e12e 6mo ago
Tunnel vision? If your model can handle big context, why divide into lesser problems to conquer - even if such splitting might be quite trivial and obvious? It's the difference of "achieve the goal", and "achieve the goal in this one particular way" (leverage large context).
- wat10000 6mo ago
  I meant, if the claim here is that small models can accomplish the same things with good scaffolding, why didn’t they demonstrate finding those problem with good scaffolding rather than directly pointing them at the problem?
  - mattmanser 6mo ago
    They don't have to. Lot of people in this thread don't seem to be getting that. If another model can find the vulnerability if you point it at the right place, it would also find the vulnerability if you scanned each place individually. People are talking about false positives, but that also doesn't matter. Again, they're not thinking it through. False positives don't matter, as you can just automatically try and exploit the "exploit" and if it doesn't work, it's a false positive. Worse, we have no idea how Mythos actually worked, it could have done the process I've outlined above, "found" 1,000s of false positives and just got rid of them by checking them. The fundamental point is it doesn't matter how the cheap models identified the exploit, it's that they can identify the exploit. When it turns out the harness is just acting as a glorified for-each brute force, it's not the model being intelligent, it's simply the harness covering more ground. It's millions of monkeys bashing type-writers, not Shakespeare at one.
    - Oranguru 6mo ago
      [dead]
    - LordDragonfang 6mo ago
      > If another model can find the vulnerability if you point it at the right place, it would also find the vulnerability if you scanned each place individually. They didn't just point it at the right place, they pointed it at the right place and gave it hints. That's a huge difference, even for humans.
    - wat10000 6mo ago
      It’s strange to see this constant “I could do that too, I just don’t want to” response. Finding an important decades-old vulnerability in OpenBSD is extremely impressive. That’s the sort of thing anyone would be proud to put on their resume. Small models are available for anyone to use. Scaffolding isn’t that hard to build. So why didn’t someone use this technique to find this vulnerability and make some headlines before Anthropic did? Either this technique with small models doesn’t actually work, or it does work but nobody’s out there trying it for some reason. I find the second possibility a lot less plausible than the first.
      - cycomanic 6mo ago
        From the article: >At AISLE, we've been running a discovery and remediation system against live targets since mid-2025: 15 CVEs in OpenSSL (including 12 out of 12 in a single security release, with bugs dating back 25+ years and a CVSS 9.8 Critical), 5 CVEs in curl, over 180 externally validated CVEs across 30+ projects spanning deep infrastructure, cryptography, middleware, and the application layer. They have been doing it (and likely others as well), but they are not anthropic which a million dollar marketing budget and a trillion dollar hype behind it, so you just didn't hear about it.
        roywiggins 6mo ago
        They could have linked their replication in this blog post, which we did all see, if they have one.

anotheryou 6mo ago

huh, running it over each function in theory but testing just the specific ones here makes sense, but that hint?!

elicash 6mo ago
I agree. To clarify, I don't necessarily agree with the post or their approach. I just thought folks were misreading it. I also think it adds something useful to the conversation.

TacticalCoder 6mo ago

> That's why their point is what the subheadline says, that the moat is the system, not the model. Can you expand a bit more on this? What is the system then in this case? And how was that model created? By AI? By humans?

SCHiM 6mo ago
You can imagine a pipeline that looks at individual source files or functions. And first "extracts" what is going on. You ask the model: - "Is the code doing arithmetic in this file/function?" - "Is the code allocating and freeing memory in this file/function?" - "Is the code the code doing X/Y/Z? etc etc" For each question, you design the follow-up vulnerability searchers. For a function you see doing arithmetic, you ask: - "Does this code look like integer overflow could take place?", For memory: - "Do all the pointers end up being freed?" _or_ - "Do all pointers only get freed once?" I think that's the harness part in terms of generating the "bug reports". From there on, you'll need a bunch of tools for the model to interact with the code. I'd imagine you'll want to build a harness/template for the file/code/function to be loaded into, and executed under ASAN. If you have an agent that thinks it found a bug: "Yes file xyz looks like it could have integer overflow in function abc at line 123, because...", you force another agent to load it in the harness under ASAN and call it. If ASAN reports a bug, great, you can move the bug to the next stage, some sort of taint analysis or reach-ability analysis. So at this point you're running a pipeline to: 1) Extract "what this code does" at the file, function or even line level. 2) Put code you suspect of being vulnerable in a harness to verify agent output. 3) Put code you confirmed is vulnerable into a queue to perform taint analysis on, to see if it can be reached by attackers. Traditionally, I guess a fuzzer approached this from 3 -> 2, and there was no "stage 1". Because LLMs "understand" code, you can invert this system, and work if up from "understanding", i.e. approach it from the other side. You ask, given this code, is there a bug, and if so can we reach it?, instead of asking: given this public interface and a bunch of data we can stuff in it, does something happen we consider exploitable?
- ang_cire 6mo ago
  That's funny, this is how I've been doing security testing in my code for a while now, minus the 'taint analysis'. Who knew I was ahead of the game. :P In all seriousness though, it scares me that a lot of security-focused people seemingly haven't learned how LLMs work best for this stuff already. You should always be breaking your code down into testable chunks, with sets of directions about how to chunk them and what to do with those chunks. Anyone just vaguely gesturing at their entire repo going, "find the security vulns" is not a serious dev/tester; we wouldn't accept that approach in manual secure coding processes/ SSDLCs.

tehryanx 6mo ago

I get what you're saying, but I think this is still missing something pretty critical. The smaller models can recognize the bug when they're looking right at it, that seems to be verified. And with AISLE's approach you can iteratively feed the models one segment at a time cheaply. But if a bug spans multiple segments, the small model doesn't have the breadth of context to understand those segments in composite. The advantage of the larger model is that it can retain more context and potentially find bugs that require more code context than one segment at a time. That said, the bugs showcased in the mythos paper all seemed to be shallow bugs that start and end in a single input segment, which is why AISLE was able to find them. But having more context in the window theoretically puts less shallow bugs within range for the model. I think the point they are making, that the model doesn't matter as much as the harness, stands for shallow bugs but not for vulnerability discovery in general.

cyanydeez 6mo ago
OK, consider a for loop that goes through your repo, then goes through each file, and then goes through each common vulnerability... Is Mythos some how more powerful than just a recursive foreloop aka, "agentic" review. You can run `open code run --command` with a tailored command for whatever vulnerabilities you're looking for.
- tehryanx 6mo ago
  newer models have larger context windows, and more stable reasoning across larger context windows. If you point your model directly at the thing you want it to assess, and it doesn't have to gather any additional context you're not really testing those things at all. Say you point kimi and opus at some code and give them an agentic looping harness with code review tools. They're going to start digging into the code gathering context by mapping out references and following leads. If the bug is really shallow, the model is going to get everything it needs to find it right away, neither of them will have any advantage. If the bug is deeper, requires a lot more code context, Opus is going to be able to hold onto a lot more information, and it's going to be a lot better at reasoning across all that information. That's a test that would actually compare the models directly. Mythos is just a bigger model with a larger context window and, presumably, better prioritization and stronger attention mechanisms.

lelanthran 6mo ago

> That's why their point is what the subheadline says, that the moat is the system, not the model. I'm skeptical; they provided a tiny piece of code and a hint to the possible problem, and their system found the bug using a small model. That is hardly useful, is it? In order to get the same result , they had to know both where the bug is and what the bug is. All these companies in the business of "reselling tokens, but with a markup" aren't going to last long. The only strategy is "get bought out and cash out before the bubble pops".

epistasis 6mo ago

If that's the point they are making, let's see their false positive rate that it produces on the entire codebase. They measured false negatives on a handful of cases, but that is not enough to hint at the system you suggest. And based on my experiences with $$$ focused eval products that you can buy right now, e.g. greptile, the false positive rate will be so high that it won't be useful to do full codebase scans this way.

zelphirkalt 6mo ago
How do we know the false positives for this "Mythos" thingamabob? Since they didn't release it, and we cannot reproduce it, are we to simply believe their word on this? What if the author of the featured article simply made a claim about that? We also simply believe their word? To me these AI tech companies are not any more trustworthy than a random blog author, maybe even less so, due to all the shady stuff they are pulling and especially since they have not released. Show or it didn't happen.
- epistasis 6mo ago
  That they were able to use it for security scanning puts the false positive rate at a useable level, inherently. Maybe they spent more on labor to comb through reports than they did on the hardware costs of discovery, but if so I think we'd be hearing from third parties about how useless those millions in Mythos credits were that they got.

nimchimpsky 6mo ago

[dead]

LordDragonfang 6mo ago

> That said, a well-designed scaffold naturally produces this kind of scoped context through its targeting and iterative prompting stages, which is exactly what both AISLE's and Anthropic's systems do. Unless the context they added to get the small model to find it was generated fully by their own scaffold (which I assume it was not, since they'd have bragged about it if it was), either they're admitting theirs isn't well designed, or they're outright lying. People aren't missing the point, they're saying the point is dishonest.

scotty79 6mo ago

> Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). To be fair, nothing stops anyone from feeding each function of given codebase separately with one out of the predefined set of hints. It's just AST and a for loop. Calling it a system is a bit much.