2 ms·

Do you really think current LLMs can implement a large application in C as securely as they can in e.g. Go? I will freely admit that I have not tried to do so b

by traes 2mo ago

Do you really think current LLMs can implement a large application in C as securely as they can in e.g. Go? I will freely admit that I have not tried to do so but I have a hard time imagining it. Perhaps my biases are outdated...

brabel 2mo ago
Danluu had an article circulating on HN a few days ago that apparently shows that yes, they can. He implemented Zstd and most models did it almost perfectly in most languages. Pandoc was a challenge they still cold not one shot though, which seems correct to me as the LLM would need proper guidance still for such a large enterprise.
- WJW 2mo ago
  Zstd is decidedly not a "large C application" though, more like a smallish library at best. The C reference implementation is also open source and thus likely part of the training set for coding LLMs. Any model re-implementing Zstd would probably be "remembering" how it was done originally, instead of implementing it from scratch.
  - brabel 2mo ago
    By your argument it should have been able to just remember Pandoc existing implementation then? But it didn’t, you think they can only “remember” small code based well enough? I doubt that. Especially since after using LLMs quite a bit I am confident I could write my own specifications for something and LLMs would be able to do it properly, despite definitively not having seen it before.
guenthert 2mo ago
Why not? The vulnerabilities you read about are due to oversights, not inherently lacking capabilities of the language. For every program exhibiting any given of such, you'll find thousands which don't make the same mistake at the same place. After all, most reported vulnerabilities are just a short patch away from being fixed. Now human programmers might find it more difficult to get certain things right than others, but to a LLM only quantity of examples matters, no?
- traes 2mo ago
  Out of curiosity, what is the largest C program you have written pre-LLM, and did you ever try to run it through valgrind*? The "oversights" in C tend to be extremely subtle and dangerous, and of a type that would be impossible to make in Go (or Rust or Common Lisp or whatever.) I do not have confidence in LLMs not falling victim to these subtleties at least once in a large application. I could be totally wrong! Perhaps they can all be found with a sufficient adversarial loops or something. I just have a really hard time imagining no problems of this sort occuring. *(Memory leaks can probably be found by just having the LLM run valgrind itself and chase them down, but this gives you a good feel for the difficulties of writing safe C. Again, never tried any of this with LLMs myself, and I haven't written a nontrivial C program in years.)
- regularfry 2mo ago
  It's more opportunities for a screw-up to lead to a vulnerability in ways the tooling won't catch by default, in a system where P(screw-up) > 0.
renox 2mo ago
> Do you really think current LLMs can implement a large application in C as securely as they can in e.g. Go? That's not my experience: I've seen an LLM generate a C++ use-after-free (1.5 month ago).

by traes 2mo ago

brabel 2mo ago

Danluu had an article circulating on HN a few days ago that apparently shows that yes, they can. He implemented Zstd and most models did it almost perfectly in most languages. Pandoc was a challenge they still cold not one shot though, which seems correct to me as the LLM would need proper guidance still for such a large enterprise.

WJW 2mo ago
Zstd is decidedly not a "large C application" though, more like a smallish library at best. The C reference implementation is also open source and thus likely part of the training set for coding LLMs. Any model re-implementing Zstd would probably be "remembering" how it was done originally, instead of implementing it from scratch.
- brabel 2mo ago
  By your argument it should have been able to just remember Pandoc existing implementation then? But it didn’t, you think they can only “remember” small code based well enough? I doubt that. Especially since after using LLMs quite a bit I am confident I could write my own specifications for something and LLMs would be able to do it properly, despite definitively not having seen it before.

guenthert 2mo ago

Why not? The vulnerabilities you read about are due to oversights, not inherently lacking capabilities of the language. For every program exhibiting any given of such, you'll find thousands which don't make the same mistake at the same place. After all, most reported vulnerabilities are just a short patch away from being fixed. Now human programmers might find it more difficult to get certain things right than others, but to a LLM only quantity of examples matters, no?

traes 2mo ago
Out of curiosity, what is the largest C program you have written pre-LLM, and did you ever try to run it through valgrind*? The "oversights" in C tend to be extremely subtle and dangerous, and of a type that would be impossible to make in Go (or Rust or Common Lisp or whatever.) I do not have confidence in LLMs not falling victim to these subtleties at least once in a large application. I could be totally wrong! Perhaps they can all be found with a sufficient adversarial loops or something. I just have a really hard time imagining no problems of this sort occuring. *(Memory leaks can probably be found by just having the LLM run valgrind itself and chase them down, but this gives you a good feel for the difficulties of writing safe C. Again, never tried any of this with LLMs myself, and I haven't written a nontrivial C program in years.)
regularfry 2mo ago
It's more opportunities for a screw-up to lead to a vulnerability in ways the tooling won't catch by default, in a system where P(screw-up) > 0.

renox 2mo ago

> Do you really think current LLMs can implement a large application in C as securely as they can in e.g. Go? That's not my experience: I've seen an LLM generate a C++ use-after-free (1.5 month ago).