3 ms·

>> Test it yourself, GPT 120B OSS is cheap and available. BTW, this is why with this bug, the stronger the model you pick (but not enough to discover the true b

>> Test it yourself, GPT 120B OSS is cheap and available. BTW, this is why with this bug, the stronger the model you pick (but not enough to discover the true bug), the less likely it is it will claim there is a bug. I guess this is the crux of the debate. All the claims are comparing models that are available freely with a model that is available only to limited customers (Mythos). The problem here is with the phrase "better model". Better how? Is it trained specifically on cybersecurity? Is it simply a large model with a higher token/thinking budget? Is it a better harness/scaffold? Is it simply a better prompt? I don't doubt that some models are stronger that other models (a Gemini Pro or a Claude Opus has more parameters, higher context sizes and probably trained for longer and on more data than their smaller counterparts (Flash and Sonnet respectively). Unless we know the exact experimental setup (which in this case is impossible because Mythos is completely closed off and not even accessible via API), all of this is hand wavy. Anthropic is definitely not going to reveal their setup because whether or not there is any secret sauce, there is more value to letting people's imaginations fly and the marketing machine work. Anthropic must be jumping with joy at all the free publicity they are getting.

solenoid0937 6mo ago
Mythos isn't restricted for marketing purposes - that would be incredibly dumb because Anthropic would be giving up first mover advantage for next gen models. It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run. That's it. That's all there is to it. It is not some grand marketing play.
- rakejake 6mo ago
  Sure, I am not precluding the possibility that they've trained a genuinely great model. All I am saying is that the "this model better than that model" is moot when on one side you have model weights, and on the other side a whitepaper and some accompanying comments on the danger. I'm not that old but have been here long enough that I remember when GPT-3 was considered too dangerous to release. Now you have models 10x as good, 1/10th the size and run on 8GB VRAM.
- the_snooze 6mo ago
  >It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run. It's a possibility, but it doesn't eliminate the possibility that it's hype. If these claims were indeed serious, they would submit it for independent analysis somewhere. This isn't some crazy process. Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded.
  - afthonos 6mo ago
    > If these claims were indeed serious, they would submit it for independent analysis somewhere. They have. 40 different companies that have all committed resources to patching their systems based on vulnerabilities found by Mythos. One of them, Google, is a frontier AI lab that pointedly did not say that their own models have found similar vulnerabilities. > Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded. Does this look something like having 40 separate companies look at the outputs of the system, deciding that it’s real and they should do something about it, and committing resources to it? At some point, “cynicism” is another word for “lalala can’t hear you”.
    - jerf 6mo ago
      Another cross-check I've run is, are the claims Anthropic is making for Mythos that out of line with the current status of AI coding assistents? To which my answer is clearly, no, not even remotely. If Anthropic is outright lying about what Mythos can do, someone else will have it in a year. In fact the security world would have to seriously consider the possibility that even if Mythos didn't exist that nation states have the equivalent in hand already. And of course, if Mythos does exist, nation states have it now. The odds that Antropic (and every other AI vendor) isn't penetrated enough by every major intelligence agency such that they have access to their choice of model approach zero. I wonder about the overlap between people being skeptical of Mythos' capabilities, and those who are too skeptical of AI to have spent any time with it because they assume it can't be any good. If you are not aware of what frontier models routinely do, you may not realize that Mythos is just an evolution of existing capabilities, not a revolution. Even just taking a publicly-available frontier model, pointing it at a code base and telling it to "find the vulnerabilities and write exploits" produces disturbingly good results. I can see the weaknesses referenced by the Mythos numbers, especially around the actual writing of the exploits, but it's not like the current frontier models fall on their face and hallucinate wildly for this task. Most everything they produce when I try this is at least a "yeah, that's worth thinking about" rather than an instant dismissal.
- 2983592 6mo ago
  How do you know? If you have access you are not unbiased, otherwise you cannot know by definition. AI companies routinely claim that something is too dangerous to release (I think GPT-2 was the first case) for marketing reasons. There are at least 10 documented high profile cases. They keep it secret because they now sell to the MIC with China and North Korea bullshit stories as well as to companies who are invested in the AI hype themselves.
  - afthonos 6mo ago
    > How do you know? If you have access you are not unbiased, otherwise you cannot know by definition. The platonic ideal of how to dismiss any argument by anyone about anything.
  - SpicyLemonZest 6mo ago
    GPT-2 was obviously too dangerous to release at the time! It's OK-ish now, when the knowledge that AI can produce arbitrary text is widely shared. It would have been a disaster for scammers and phishers to get GPT-2 at a time when almost everyone still assumed that large volumes of detailed text proved there's a real human being on the other end of the conversation.
    - jayd16 6mo ago
      And, as we all know, humans can't be scammers. They need the robots to lie.
  - Glemllksdf 6mo ago
    I prefer a more cautios approach than the musk style were stuff gets fixed after. And with gpt-2 the worry was mass emails a lot better and more detailed and personal, social media campaigns etc. How many bots are deployed today on X and influencing democrazy around the globe? Its fair to say it had an impact and LLMs still have.
- zzzeek 6mo ago
  it seems likely it's both a better model to some unknown extent and doing this "we have to give it to the defenders first" thing is super great marketing material. it seems an entirely natural marketing campaign "announce that we can't even give the model to everyone at first, it's so great!", plus there's some truth to it, even better. unless you are an employee at anthropic and shouldn't be talking about any of this at all, there's no way to know what the model's capabilities are.
- frank-romita 6mo ago
  Or, They created the illusion that it's restricted for security reasons but in reality they just lack the necessary for this to be used widespread!
- jayd16 6mo ago
  If it wasn't marketing it wouldn't have fancy branding... It wouldn't even be announced.
- louiereederson 6mo ago
  I don't think you can say this with confidence, outside-in. It's not just about safety. The additional unknown is cost - I don't just mean API cost, but fully loaded cost for a given task. Is the model cost effective for tasks such that it has product market fit? We don't yet know if Mythos was a level shift in the capability/cost frontier, or a continued extension of the same logarithmic capability/cost curve.
  - solenoid0937 6mo ago
    Some people have access to the model for red team purposes as part of Glasswing and they came away quite spooked according to what I heard
    - louiereederson 6mo ago
      I don't doubt it, I just mean the decision to release/not release generally may also be informed by the commercial/economic viability of the model for general usage patterns versus extremely high value patterns like vulnerability assessment
- dmix 6mo ago
  That safety stuff is almost always quacks whose job it is to exaggerate LLMs at their non profits or marketing hype that "our models are so powerful you should fear them". Then they release them and the world moves on and adapts. Mythos will benefit security in the long run more than hackers, if it can do what they claim. And there's nothing that will stop an LLM like it from being released in the near term so it's very likely just resource constraints or marketing
antirez 6mo ago
In the Anthropic Mythos model cards they explicitly remarked that they didn't want Mythos to be specifically good at security. They trained it to be good at coding, and as a side effect the model is (obviously) good at security. This what happens with flesh hackers too, mostly. Hackers are very good programmers, as a side effect they understand systems well enough that their understanding has security implications.
- Hendrikto 6mo ago
  Model cards are just marketing material. I wouldn’t trust them one bit.
  - 2983592 6mo ago
    But they are treated as holy scripture ...
  - Yokohiii 6mo ago
    The whole discussion started out as an attempt to disprove/verify anthropics (model card) claims. He also transfers the logic of their claims to the actual real world. You can say that model cards are marketing garbage. You have to prove that experienced programmers are not significantly better at security.
    - root_axis 6mo ago
      > You have to prove that experienced programmers are not significantly better at security. That has not been my experience. It's true that they are "better at security" in the sense that they know to avoid common security pitfalls like unparamaterized SQL, but essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems.
      - inetknght 6mo ago
        > essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems. I've found it to be the opposite. Many of them do have the ability to apply their knowledge in that fashion. They're just either not incentivised to do so, or incentivised to not do so.
Glemllksdf 6mo ago
If its really more expensive per token, it might have more parameters and is then able to hold more context/scope of code. Rumors say it has 10 trillion parameter vs. 1 trillion.
- rakejake 6mo ago
  Yes, that does track with my personal experience. More context, more params and no quantization is probably it. But my hunch is that all the training data they've been getting in the past year also plays a part here. More than any other lab, anthropic's focus on coding right from the beginning gives them access to the best training data (several githubs worth). Most of this code comes with human feedback and anthropic even has data on how many went to production, got reverted etc. No need to pay for human labeling when your customers are doing it for you. This is their secret sauce.

solenoid0937 6mo ago

Mythos isn't restricted for marketing purposes - that would be incredibly dumb because Anthropic would be giving up first mover advantage for next gen models. It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run. That's it. That's all there is to it. It is not some grand marketing play.

rakejake 6mo ago
Sure, I am not precluding the possibility that they've trained a genuinely great model. All I am saying is that the "this model better than that model" is moot when on one side you have model weights, and on the other side a whitepaper and some accompanying comments on the danger. I'm not that old but have been here long enough that I remember when GPT-3 was considered too dangerous to release. Now you have models 10x as good, 1/10th the size and run on 8GB VRAM.
the_snooze 6mo ago
>It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run. It's a possibility, but it doesn't eliminate the possibility that it's hype. If these claims were indeed serious, they would submit it for independent analysis somewhere. This isn't some crazy process. Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded.
- afthonos 6mo ago
  > If these claims were indeed serious, they would submit it for independent analysis somewhere. They have. 40 different companies that have all committed resources to patching their systems based on vulnerabilities found by Mythos. One of them, Google, is a frontier AI lab that pointedly did not say that their own models have found similar vulnerabilities. > Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded. Does this look something like having 40 separate companies look at the outputs of the system, deciding that it’s real and they should do something about it, and committing resources to it? At some point, “cynicism” is another word for “lalala can’t hear you”.
  - jerf 6mo ago
    Another cross-check I've run is, are the claims Anthropic is making for Mythos that out of line with the current status of AI coding assistents? To which my answer is clearly, no, not even remotely. If Anthropic is outright lying about what Mythos can do, someone else will have it in a year. In fact the security world would have to seriously consider the possibility that even if Mythos didn't exist that nation states have the equivalent in hand already. And of course, if Mythos does exist, nation states have it now. The odds that Antropic (and every other AI vendor) isn't penetrated enough by every major intelligence agency such that they have access to their choice of model approach zero. I wonder about the overlap between people being skeptical of Mythos' capabilities, and those who are too skeptical of AI to have spent any time with it because they assume it can't be any good. If you are not aware of what frontier models routinely do, you may not realize that Mythos is just an evolution of existing capabilities, not a revolution. Even just taking a publicly-available frontier model, pointing it at a code base and telling it to "find the vulnerabilities and write exploits" produces disturbingly good results. I can see the weaknesses referenced by the Mythos numbers, especially around the actual writing of the exploits, but it's not like the current frontier models fall on their face and hallucinate wildly for this task. Most everything they produce when I try this is at least a "yeah, that's worth thinking about" rather than an instant dismissal.
2983592 6mo ago
How do you know? If you have access you are not unbiased, otherwise you cannot know by definition. AI companies routinely claim that something is too dangerous to release (I think GPT-2 was the first case) for marketing reasons. There are at least 10 documented high profile cases. They keep it secret because they now sell to the MIC with China and North Korea bullshit stories as well as to companies who are invested in the AI hype themselves.
- afthonos 6mo ago
  > How do you know? If you have access you are not unbiased, otherwise you cannot know by definition. The platonic ideal of how to dismiss any argument by anyone about anything.
- SpicyLemonZest 6mo ago
  GPT-2 was obviously too dangerous to release at the time! It's OK-ish now, when the knowledge that AI can produce arbitrary text is widely shared. It would have been a disaster for scammers and phishers to get GPT-2 at a time when almost everyone still assumed that large volumes of detailed text proved there's a real human being on the other end of the conversation.
  - jayd16 6mo ago
    And, as we all know, humans can't be scammers. They need the robots to lie.
- Glemllksdf 6mo ago
  I prefer a more cautios approach than the musk style were stuff gets fixed after. And with gpt-2 the worry was mass emails a lot better and more detailed and personal, social media campaigns etc. How many bots are deployed today on X and influencing democrazy around the globe? Its fair to say it had an impact and LLMs still have.
zzzeek 6mo ago
it seems likely it's both a better model to some unknown extent and doing this "we have to give it to the defenders first" thing is super great marketing material. it seems an entirely natural marketing campaign "announce that we can't even give the model to everyone at first, it's so great!", plus there's some truth to it, even better. unless you are an employee at anthropic and shouldn't be talking about any of this at all, there's no way to know what the model's capabilities are.
frank-romita 6mo ago
Or, They created the illusion that it's restricted for security reasons but in reality they just lack the necessary for this to be used widespread!
jayd16 6mo ago
If it wasn't marketing it wouldn't have fancy branding... It wouldn't even be announced.
louiereederson 6mo ago
I don't think you can say this with confidence, outside-in. It's not just about safety. The additional unknown is cost - I don't just mean API cost, but fully loaded cost for a given task. Is the model cost effective for tasks such that it has product market fit? We don't yet know if Mythos was a level shift in the capability/cost frontier, or a continued extension of the same logarithmic capability/cost curve.
- solenoid0937 6mo ago
  Some people have access to the model for red team purposes as part of Glasswing and they came away quite spooked according to what I heard
  - louiereederson 6mo ago
    I don't doubt it, I just mean the decision to release/not release generally may also be informed by the commercial/economic viability of the model for general usage patterns versus extremely high value patterns like vulnerability assessment
dmix 6mo ago
That safety stuff is almost always quacks whose job it is to exaggerate LLMs at their non profits or marketing hype that "our models are so powerful you should fear them". Then they release them and the world moves on and adapts. Mythos will benefit security in the long run more than hackers, if it can do what they claim. And there's nothing that will stop an LLM like it from being released in the near term so it's very likely just resource constraints or marketing

antirez 6mo ago

In the Anthropic Mythos model cards they explicitly remarked that they didn't want Mythos to be specifically good at security. They trained it to be good at coding, and as a side effect the model is (obviously) good at security. This what happens with flesh hackers too, mostly. Hackers are very good programmers, as a side effect they understand systems well enough that their understanding has security implications.

Hendrikto 6mo ago
Model cards are just marketing material. I wouldn’t trust them one bit.
- 2983592 6mo ago
  But they are treated as holy scripture ...
- Yokohiii 6mo ago
  The whole discussion started out as an attempt to disprove/verify anthropics (model card) claims. He also transfers the logic of their claims to the actual real world. You can say that model cards are marketing garbage. You have to prove that experienced programmers are not significantly better at security.
  - root_axis 6mo ago
    > You have to prove that experienced programmers are not significantly better at security. That has not been my experience. It's true that they are "better at security" in the sense that they know to avoid common security pitfalls like unparamaterized SQL, but essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems.
    - inetknght 6mo ago
      > essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems. I've found it to be the opposite. Many of them do have the ability to apply their knowledge in that fashion. They're just either not incentivised to do so, or incentivised to not do so.

Glemllksdf 6mo ago

If its really more expensive per token, it might have more parameters and is then able to hold more context/scope of code. Rumors say it has 10 trillion parameter vs. 1 trillion.

rakejake 6mo ago
Yes, that does track with my personal experience. More context, more params and no quantization is probably it. But my hunch is that all the training data they've been getting in the past year also plays a part here. More than any other lab, anthropic's focus on coding right from the beginning gives them access to the best training data (several githubs worth). Most of this code comes with human feedback and anthropic even has data on how many went to production, got reverted etc. No need to pay for human labeling when your customers are doing it for you. This is their secret sauce.