9 ms·

All the bugs they found

84 points by ziggy42 5mo ago

grey-area 5mo ago
[flagged]
vachanmn123 5mo ago
> Trying to work around Anthropic blocking security-related prompts does get pretty tiring though. Didn't know this is a thing... interesting for a company that's marketing their Mythos so hard not allowing security prompts. I am also curious how the cheaper Chinese models do, I have an Opencode Go plan, so I'll let 'em rip over the weekend, hopefully I get to see a few bugs!
- simonw 5mo ago
  I think that's consistent of Anthropic. The whole point of Mythos/Glasswing is "our best models are scary good at security research, so much so that we won't let them help you find vulnerabilities unless you are a trusted partner".
  - tptacek 5mo ago
    You can just fill out a form and be in their security researcher program, which dials back the refusals.
- deleted 5mo ago
  [deleted]
shandilyaharsh 5mo ago
sometimes i feel mythos is just that a myth
- nullbio 5mo ago
  Agreed. Considering Anthropic had a sandbox bypass vulnerability in CC for a year, silently patched it, and still hasn't made a disclosure statement, no one on Earth should trust them or believe a word they say. https://www.securityweek.com/anthropic-silently-patches-claude-code-sandbox-bypass/ https://www.securityweek.com/anthropic-silently-patches-clau...
- dv_dt 5mo ago
  The "too many security issues" meme feels like a form of product placement marketing. How many of the bugs found would have also been found if you said to a security team - ok you now have a project with independent time to go spelunking for bugs - this is your highest priority for the next month. Now do the same with a bunch of security teams across multiple organizations across the industry doing that at the same time. What is the differential in actuality with and without Mythos. The brilliant part is now those discoveries have a Anthropic mythos tag on them. Even if it is marketing, at least there is some positive side effects of identified and closed security flaws.
- keyle 5mo ago
  I think you meant marketing*
MarStudio 5mo ago
[dead]
xeyownt 5mo ago
Nice writeup. A practical example of a project, what was found, how it was found, the quality of the findings, reproducible.
NitpickLawyer 5mo ago
The cool thing about LLMs is that once a capability is "good enough" you can always "chain" them together for better overall results. On the client side this means "write an API that does x y z" -> "analyse this API for security concerns" -> "PoC for each finding from this report" -> "fix this code according to these verified claims". On the "server side" (i.e. training) you can use the current gen models to improve the training data by running many parallel environments with a similar loop as above. Then incorporate the new data and repeat. Reminiscent of the old GAN approach, where the generator and discriminator are trained together in an adversarial regime. The end result should be safer code on "vanilla" prompts. "Write an API that does x y z" should now contain the learnings from this loop, and the models should produce better code. Works really well for every verifiable scenario. And as the models become better, they can also more reliably create environments that closely match real-world scenarios. If you also have some data from human devs (say you run a subsidised coding model for a few months), even better. An example of turning a "normal" repo into a verifiable environment that I read recently in the Cursor blog: take a repo, ask an LLM to remove a feature, verify that the app still works w/o the feature, verify that the tests for that feature fail. Ask a generator to "add feature x". Verify with the original tests. If pass -> give carrot :) The key is composition. Once you unlock a new capability, that gets implemented and incorporated into the next training run. Pretty neat, I would say, and the main driver for the recent increase in the breadth of capabilities for new models.
- amelius 5mo ago
  > If pass -> give carrot :) More like, give $$$ pass or not.
- onlyrealcuzzo 5mo ago
  The bad thing about software is that there's infinite ways to solve the same problem, and the vast majority of them are terrible and unmaintainable, so "working" is a prerequisite, but not really "good enough". It's good if no LLMs can find a bug. It certainly does not mean there isn't one... I've found LLMs to be very disappointing at identifying overly complex code (that they've written) and the correct architectural decisions to 1) make the code actually work, and 2) be simple, maintainable, and future proof. They can certainly find some bugs, which definitely has value, but I've not had much success with them writing code that simply has no bugs... That requires simplicity and architectural correctness, something LLMs are good at vaguely bullshitting, but not very good at getting correct. I think this can be solved by feeding them the right metrics, but I haven't found prior art for how to algorithmically pinpoint: 1) what is actually complex in a bad way (there's a lot of ways to do this roughly), and 2) where exactly the problem is most acutely (less prior art here, but some), and 3) what viable solutions are. If you can get better at 1 and 2, the LLMs can get much better at 3. Anybody who has ideas, I'd love to hear them, as this is what I'm working on now.
Makiaveli 5mo ago
[dead]
keybored 5mo ago
I don’t really care about posting in bold 20 bugs when it comes to a hobby project. (In before “Linux was just a hobby project”) No need to LLM post over what this tells us about the trajectory of society, oh my. We can save that dialogue for finding bugs in widely used projects. Edit: Something I tried to reply to a now-dead top level comment here: Whoever claims that new accounts alone is a signal for submission-boosting comments etc. needs to update their heuristics.
jcarrano 5mo ago
I found, in my rather recent experience with Go, that using anything other than zero for invalid, default or "sentinel" values is a source of potential problems due to the lack of real constructors.
- grey-area 5mo ago
  Yes I'd say 0 should always be treated as a None or Invalid value. The upside of the lack of real constructors is less incidental complexity which every object having a constructor written which then has to be read and maintained. Another option of course is to write constructors - there's nothing to stop you doing so in go and using those when creating objects (e.g. foo.New() whenever you want one of these things), but it'd be a convention rather than something required.
rashar 5mo ago
He describes himself as "Software engineer. Writing code prompts at Google". So throwing his own, apparently poorly written, creation under the bus will get him applause and promotions by the AI lunatics. It is a currently popular strategy among AI boosters.
- hootz 5mo ago
  Seems to be working for a lot of people, and I won't really blame them that much. People want promotions, money and a job in general, and they will do stupid stuff to keep their jobs and increase their pay. Unfortunately, it's an incentives problem of our current mode of production.
- simonw 5mo ago
  What a weird comment. You think he cynically decided to boost his career by writing a detailed description of the exploits found in his own software. Is there no room in your model of the world for someone to figure out something interesting using AI tools and then write about it just because they like sharing interesting information?
  - keybored 5mo ago
    It is not at all, in the slightest, weird heuristic to deploy in the Agentic Era. It’s a heuristic after all. There is no proof one way or the other.
  - tptacek 5mo ago
    The irritating thing about this thread is that it's a human-authored post about vulnerabilities in a WASM interpreter, which isn't a common genre --- it's interesting on its own terms. But as usual, instead of discussing the interesting thing, we're rehashing the high-salience culture war issue instead.
- some_furry 5mo ago
  I can't speak to the larger trend of AI boosters, as I don't go out of my way to pay attention to them, but the practice of being vocally self-critical and openly discussing the flaws found in one's own software (whether from AI or human analysts) is a damn good idea for reasons that can best be understood by rehashing the vulnerability disclosure debate in one's own mind.
AznHisoka 5mo ago
… running thru my head. All the bugs they found…
- ggambetta 5mo ago
  Not gonna fix 'em!
- ziggy42 5mo ago
  indeed this was the inspiration for the title :D
andai 5mo ago
I thought this was going to be a list of all the zero days that popped up on HN in the last ten days.
jedisct1 5mo ago
Have you tried Swival /audit? https://swival.dev/pages/audit.html https://swival.dev/pages/audit.html
r0ze-at-hn 5mo ago
> It is also very extensively tested against the official WASM testsuite. I was hoping that near the end the author would have tried to contributed any new tests to the official testsuite to help catch these same errors elsewhere. For the first one, Zero Is Not Null maybe there is a missing test in in the call_indirect test? https://github.com/WebAssembly/testsuite/blob/main/call_indirect.wast https://github.com/WebAssembly/testsuite/blob/main/call_indi...
- zephen 5mo ago
  My takeaway from the article is to stay away from wasm, because the official testsuite is dogshit that doesn't even make sure that you popped the same number of parameters that were pushed.

84 points by ziggy42 5mo ago

[flagged]

> Trying to work around Anthropic blocking security-related prompts does get pretty tiring though. Didn't know this is a thing... interesting for a company that's marketing their Mythos so hard not allowing security prompts. I am also curious how the cheaper Chinese models do, I have an Opencode Go plan, so I'll let 'em rip over the weekend, hopefully I get to see a few bugs!

simonw 5mo ago
I think that's consistent of Anthropic. The whole point of Mythos/Glasswing is "our best models are scary good at security research, so much so that we won't let them help you find vulnerabilities unless you are a trusted partner".
- tptacek 5mo ago
  You can just fill out a form and be in their security researcher program, which dials back the refusals.
deleted 5mo ago
[deleted]

shandilyaharsh 5mo ago

sometimes i feel mythos is just that a myth

nullbio 5mo ago
Agreed. Considering Anthropic had a sandbox bypass vulnerability in CC for a year, silently patched it, and still hasn't made a disclosure statement, no one on Earth should trust them or believe a word they say. https://www.securityweek.com/anthropic-silently-patches-claude-code-sandbox-bypass/ https://www.securityweek.com/anthropic-silently-patches-clau...
dv_dt 5mo ago
The "too many security issues" meme feels like a form of product placement marketing. How many of the bugs found would have also been found if you said to a security team - ok you now have a project with independent time to go spelunking for bugs - this is your highest priority for the next month. Now do the same with a bunch of security teams across multiple organizations across the industry doing that at the same time. What is the differential in actuality with and without Mythos. The brilliant part is now those discoveries have a Anthropic mythos tag on them. Even if it is marketing, at least there is some positive side effects of identified and closed security flaws.
keyle 5mo ago
I think you meant marketing*

MarStudio 5mo ago

[dead]

xeyownt 5mo ago

Nice writeup. A practical example of a project, what was found, how it was found, the quality of the findings, reproducible.

NitpickLawyer 5mo ago

The cool thing about LLMs is that once a capability is "good enough" you can always "chain" them together for better overall results. On the client side this means "write an API that does x y z" -> "analyse this API for security concerns" -> "PoC for each finding from this report" -> "fix this code according to these verified claims". On the "server side" (i.e. training) you can use the current gen models to improve the training data by running many parallel environments with a similar loop as above. Then incorporate the new data and repeat. Reminiscent of the old GAN approach, where the generator and discriminator are trained together in an adversarial regime. The end result should be safer code on "vanilla" prompts. "Write an API that does x y z" should now contain the learnings from this loop, and the models should produce better code. Works really well for every verifiable scenario. And as the models become better, they can also more reliably create environments that closely match real-world scenarios. If you also have some data from human devs (say you run a subsidised coding model for a few months), even better. An example of turning a "normal" repo into a verifiable environment that I read recently in the Cursor blog: take a repo, ask an LLM to remove a feature, verify that the app still works w/o the feature, verify that the tests for that feature fail. Ask a generator to "add feature x". Verify with the original tests. If pass -> give carrot :) The key is composition. Once you unlock a new capability, that gets implemented and incorporated into the next training run. Pretty neat, I would say, and the main driver for the recent increase in the breadth of capabilities for new models.

amelius 5mo ago
> If pass -> give carrot :) More like, give $$$ pass or not.
onlyrealcuzzo 5mo ago
The bad thing about software is that there's infinite ways to solve the same problem, and the vast majority of them are terrible and unmaintainable, so "working" is a prerequisite, but not really "good enough". It's good if no LLMs can find a bug. It certainly does not mean there isn't one... I've found LLMs to be very disappointing at identifying overly complex code (that they've written) and the correct architectural decisions to 1) make the code actually work, and 2) be simple, maintainable, and future proof. They can certainly find some bugs, which definitely has value, but I've not had much success with them writing code that simply has no bugs... That requires simplicity and architectural correctness, something LLMs are good at vaguely bullshitting, but not very good at getting correct. I think this can be solved by feeding them the right metrics, but I haven't found prior art for how to algorithmically pinpoint: 1) what is actually complex in a bad way (there's a lot of ways to do this roughly), and 2) where exactly the problem is most acutely (less prior art here, but some), and 3) what viable solutions are. If you can get better at 1 and 2, the LLMs can get much better at 3. Anybody who has ideas, I'd love to hear them, as this is what I'm working on now.

Makiaveli 5mo ago

[dead]

keybored 5mo ago

I don’t really care about posting in bold 20 bugs when it comes to a hobby project. (In before “Linux was just a hobby project”) No need to LLM post over what this tells us about the trajectory of society, oh my. We can save that dialogue for finding bugs in widely used projects. Edit: Something I tried to reply to a now-dead top level comment here: Whoever claims that new accounts alone is a signal for submission-boosting comments etc. needs to update their heuristics.

jcarrano 5mo ago

I found, in my rather recent experience with Go, that using anything other than zero for invalid, default or "sentinel" values is a source of potential problems due to the lack of real constructors.

grey-area 5mo ago
Yes I'd say 0 should always be treated as a None or Invalid value. The upside of the lack of real constructors is less incidental complexity which every object having a constructor written which then has to be read and maintained. Another option of course is to write constructors - there's nothing to stop you doing so in go and using those when creating objects (e.g. foo.New() whenever you want one of these things), but it'd be a convention rather than something required.

rashar 5mo ago

He describes himself as "Software engineer. Writing code prompts at Google". So throwing his own, apparently poorly written, creation under the bus will get him applause and promotions by the AI lunatics. It is a currently popular strategy among AI boosters.

hootz 5mo ago
Seems to be working for a lot of people, and I won't really blame them that much. People want promotions, money and a job in general, and they will do stupid stuff to keep their jobs and increase their pay. Unfortunately, it's an incentives problem of our current mode of production.
simonw 5mo ago
What a weird comment. You think he cynically decided to boost his career by writing a detailed description of the exploits found in his own software. Is there no room in your model of the world for someone to figure out something interesting using AI tools and then write about it just because they like sharing interesting information?
- keybored 5mo ago
  It is not at all, in the slightest, weird heuristic to deploy in the Agentic Era. It’s a heuristic after all. There is no proof one way or the other.
- tptacek 5mo ago
  The irritating thing about this thread is that it's a human-authored post about vulnerabilities in a WASM interpreter, which isn't a common genre --- it's interesting on its own terms. But as usual, instead of discussing the interesting thing, we're rehashing the high-salience culture war issue instead.
some_furry 5mo ago
I can't speak to the larger trend of AI boosters, as I don't go out of my way to pay attention to them, but the practice of being vocally self-critical and openly discussing the flaws found in one's own software (whether from AI or human analysts) is a damn good idea for reasons that can best be understood by rehashing the vulnerability disclosure debate in one's own mind.

AznHisoka 5mo ago

… running thru my head. All the bugs they found…

ggambetta 5mo ago
Not gonna fix 'em!
ziggy42 5mo ago
indeed this was the inspiration for the title :D

andai 5mo ago

I thought this was going to be a list of all the zero days that popped up on HN in the last ten days.

jedisct1 5mo ago

Have you tried Swival /audit? https://swival.dev/pages/audit.html https://swival.dev/pages/audit.html

r0ze-at-hn 5mo ago

> It is also very extensively tested against the official WASM testsuite. I was hoping that near the end the author would have tried to contributed any new tests to the official testsuite to help catch these same errors elsewhere. For the first one, Zero Is Not Null maybe there is a missing test in in the call_indirect test? https://github.com/WebAssembly/testsuite/blob/main/call_indirect.wast https://github.com/WebAssembly/testsuite/blob/main/call_indi...

zephen 5mo ago
My takeaway from the article is to stay away from wasm, because the official testsuite is dogshit that doesn't even make sure that you popped the same number of parameters that were pushed.