3 ms·

if machine-learning can find all these holes why can't machine-learning write a product from scratch that is flawless?

by ck2 5mo ago

yjftsjthsd-h 5mo ago
Who said it can't? https://news.ycombinator.com/item?id=47759709 https://news.ycombinator.com/item?id=47759709 appears to be a nearly flawless (per spec) zip implementation.
- PunchyHamster 5mo ago
  [flagged]
  - yjftsjthsd-h 5mo ago
    No, a collection of fuzzers and the lean proof assistant found (almost) no bugs.
  - dang 5mo ago
    Could you please stop breaking the site guidelines? We've already asked you once. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - PunchyHamster 5mo ago
      Sorry, I got carried away, the whole blind trust in AI tools gets me wound up a bit too fast, I'll try to be better. > We've already asked you once. there are no notifications of any kind about that or the fact the comment has been flagged so I genuinely didn't notice previous one and only noticed this after I noticed rate limiting.
_flux 5mo ago
Just because something is good at finding bugs, it may not find all the bugs. Finding a bug only tells you there was one bug you found, it doesn't tell if the rest is solid.
hnlmorg 5mo ago
It’s easier to break something than it is to make something that cannot be broken.
jonhohle 5mo ago
Have you ever met a security engineer? I’ve never met one who was also a good engineer (not saying they don’t exist, I just haven’t met one). Do they find vulnerabilities? Sure. Could they write the tools they use to find vulnerabilities, most probably not.
perlgeek 5mo ago
LLMs certainly make it more feasible to rewrite a product in a memory-safe language, eliminating a whole class of bugs. Flawless software is hard for an LLM to write, because all the programs they have been trained on are flawed as well. As a fun exercise, you could give a coding agent a hunk of non-trivial software (such as the Linux kernel, or postgresql, or whatever), and tell it over and over again: find a flaw in this, fix it. I'm pretty sure it won't ever tell you "now it's perfect" (and do this reproducibly).
tclancy 5mo ago
Because the problem is asymmetric: the attacker only needs to find one hole at one time. The defender has to be flawless forever.
chromacity 5mo ago
If humans can find bugs, why can't humans write flawless code? Whatever the answer to that conundrum might be, LLMs are trained on these patterns and replicate them pretty faithfully.
duped 5mo ago
You could argue the answer to this question depends on if you believe P=NP
tetha 5mo ago
How do you define flawless though? The CVEs here have their fair share of silly C problems, but also more rigid input validation and handling. These more rigid validations exclude stuff which may even be valid by the spec, but entirely problematic in practice. As examples, take a look how many valid XML documents are practically considered unsafe and not parsed, for example due to recursive entity expansion. This renders the parsers not flawless and in fact not in spec. Or, my favorite bait - there should be a maximum length limit on passwords. Why would you ever need a kilobyte sized password?

if machine-learning can find all these holes why can't machine-learning write a product from scratch that is flawless?

by ck2 5mo ago

if machine-learning can find all these holes why can't machine-learning write a product from scratch that is flawless?

Who said it can't? https://news.ycombinator.com/item?id=47759709 https://news.ycombinator.com/item?id=47759709 appears to be a nearly flawless (per spec) zip implementation.

PunchyHamster 5mo ago
[flagged]
- yjftsjthsd-h 5mo ago
  No, a collection of fuzzers and the lean proof assistant found (almost) no bugs.
- dang 5mo ago
  Could you please stop breaking the site guidelines? We've already asked you once. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - PunchyHamster 5mo ago
    Sorry, I got carried away, the whole blind trust in AI tools gets me wound up a bit too fast, I'll try to be better. > We've already asked you once. there are no notifications of any kind about that or the fact the comment has been flagged so I genuinely didn't notice previous one and only noticed this after I noticed rate limiting.

_flux 5mo ago

Just because something is good at finding bugs, it may not find all the bugs. Finding a bug only tells you there was one bug you found, it doesn't tell if the rest is solid.

hnlmorg 5mo ago

It’s easier to break something than it is to make something that cannot be broken.

jonhohle 5mo ago

Have you ever met a security engineer? I’ve never met one who was also a good engineer (not saying they don’t exist, I just haven’t met one). Do they find vulnerabilities? Sure. Could they write the tools they use to find vulnerabilities, most probably not.

perlgeek 5mo ago

LLMs certainly make it more feasible to rewrite a product in a memory-safe language, eliminating a whole class of bugs. Flawless software is hard for an LLM to write, because all the programs they have been trained on are flawed as well. As a fun exercise, you could give a coding agent a hunk of non-trivial software (such as the Linux kernel, or postgresql, or whatever), and tell it over and over again: find a flaw in this, fix it. I'm pretty sure it won't ever tell you "now it's perfect" (and do this reproducibly).

tclancy 5mo ago

Because the problem is asymmetric: the attacker only needs to find one hole at one time. The defender has to be flawless forever.

chromacity 5mo ago

If humans can find bugs, why can't humans write flawless code? Whatever the answer to that conundrum might be, LLMs are trained on these patterns and replicate them pretty faithfully.

duped 5mo ago

You could argue the answer to this question depends on if you believe P=NP

tetha 5mo ago

How do you define flawless though? The CVEs here have their fair share of silly C problems, but also more rigid input validation and handling. These more rigid validations exclude stuff which may even be valid by the spec, but entirely problematic in practice. As examples, take a look how many valid XML documents are practically considered unsafe and not parsed, for example due to recursive entity expansion. This renders the parsers not flawless and in fact not in spec. Or, my favorite bait - there should be a maximum length limit on passwords. Why would you ever need a kilobyte sized password?