4 ms·

Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.

by dmix 1mo ago

deleted 1mo ago
[deleted]
bewareofscams 1mo ago
[flagged]
- shevy-java 1mo ago
  > LLMs do find bugs, do save time They find bugs but whether they save time is nowhere near as clear as you try to insinuate here.
  - pixl97 1mo ago
    They save time in finding bugs.
    - lukan 1mo ago
      And for me they also save time in fixing bugs.
    - owebmaster 1mo ago
      Unless it's finding a bug it added then it's time wasted x2
hombre_fatal 1mo ago
The missing part of this is that verifying the bug with LLMs is also easy, and so is adversarially reviewing the proposed fix with LLMs. The only thing left for you to do should be directional decisions. The LLMs should pause and rope you in if the fix involves directional/invariant changes.
- deleted 1mo ago
  [deleted]
nonethewiser 1mo ago
No one can keep up with the volume of code AI produces. We wont stop using AI. We will use AI to check AI. Of course this is crazy, but it will also unlock pretty insane scaling and productivity and ultimately we will manage it on either end via requirements and tests.
- krona 1mo ago
  You're suggesting that LLMs get better at fixing bugs/vulnerabilities, but at the same time stop getting better at finding them? What if this difference is inherent and essential?
  - TacticalCoder 1mo ago
    > You're suggesting that LLMs get better at fixing bugs/vulnerabilities, but at the same time stop getting better at finding them? Are you implying that all code writing by LLMs atm is bug-free?
    - krona 1mo ago
      Absolutely not. By most accounts they're terrible at fixing anything other than trivial bugs in complex codebases e.g. Linux kernel, but they're much better at finding them.
      - a2ff6eeb0 1mo ago
        So you put it in a loop and tell it to find the bugs in the code it wrote. What's the issue?
        krona 1mo ago
        This is a self confession if I ever saw one.
        a2ff6eeb0 1mo ago
black_knight 1mo ago
This is where I believe strong typing (like, Haskell-strong or stronger) and functional programming in general will be a win. The confidence I have that my fixes are localised when fixing Haskell code is infinitely stronger than fixing even Java, not speak about C, code.
- astrange 1mo ago
  Haskell's type system would not easily prevent this bug. It's not good at numeric/logic issues like that. When people say "Haskell makes it impossible to write bugs" they mean "Haskell has enums" (ADTs).
  - black_knight 1mo ago
    I am not claiming you cant write buggy code in Haskell! But following good functional style, your bug will more likely be compartmentalised, and fixing it will not break some other part of your program.
    - StilesCrisis 1mo ago
      You can write good functional code in many languages. (Even C++!)
      - black_knight 1mo ago
        Sure! I have done my fair share of pretending Java and C++ support my functional style. But at the end of the day, you have better support for writing that style in a real functional programming language. And I wonder how well one can enforce a functional style in say Java or C++ upon the LLMs. Who knows, they might be great at it?
  - _jackdk_ 1mo ago
    Liquid Haskell might require you to prove that the divisor is nonzero, but even in standard Haskell there's common idioms for ensuring that a list is non-empty (data NonEmpty a = a :| [a]) or that text is non-empty (newtype NonEmptyText = NonEmptyText Text, with non-exported constructor, helpers like make :: Text -> NonEmptyText, or more advanced tricks like https://exploring-better-ways.bellroy.com/haskell-koan-type-checked-non-empty-strings.html https://exploring-better-ways.bellroy.com/haskell-koan-type-... ). The big problem preventing this approach from working for numbers is that it's just so cumbersome there. Most of this is because all the arithmetic operators are bundled into a single Num typeclass, and `fromInteger :: Num a => Integer -> a` has a type that's impossible for a "non-zero number" wrapper to satisfy.
macless 1mo ago
[flagged]
sadfgknerknksdf 1mo ago
If finding the bugs with LLMs is easy. Then making sure it doesn't break something else is just LLMs finding no bugs. Easy.
BikiniPrince 1mo ago
That hasn’t been that bad. My real issue has been the time sink involved in following along with the maintainer and jumper through their hoops. Even after I demonstrate a flaw and a potential fix. My schedule is just so busy I need to pencil in time to deal with them.

Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.

by dmix 1mo ago

Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.

deleted 1mo ago

[deleted]

bewareofscams 1mo ago

[flagged]

shevy-java 1mo ago
> LLMs do find bugs, do save time They find bugs but whether they save time is nowhere near as clear as you try to insinuate here.
- pixl97 1mo ago
  They save time in finding bugs.
  - lukan 1mo ago
    And for me they also save time in fixing bugs.
  - owebmaster 1mo ago
    Unless it's finding a bug it added then it's time wasted x2

hombre_fatal 1mo ago

The missing part of this is that verifying the bug with LLMs is also easy, and so is adversarially reviewing the proposed fix with LLMs. The only thing left for you to do should be directional decisions. The LLMs should pause and rope you in if the fix involves directional/invariant changes.

deleted 1mo ago
[deleted]

nonethewiser 1mo ago

No one can keep up with the volume of code AI produces. We wont stop using AI. We will use AI to check AI. Of course this is crazy, but it will also unlock pretty insane scaling and productivity and ultimately we will manage it on either end via requirements and tests.

krona 1mo ago
You're suggesting that LLMs get better at fixing bugs/vulnerabilities, but at the same time stop getting better at finding them? What if this difference is inherent and essential?
- TacticalCoder 1mo ago
  > You're suggesting that LLMs get better at fixing bugs/vulnerabilities, but at the same time stop getting better at finding them? Are you implying that all code writing by LLMs atm is bug-free?
  - krona 1mo ago
    Absolutely not. By most accounts they're terrible at fixing anything other than trivial bugs in complex codebases e.g. Linux kernel, but they're much better at finding them.
    - a2ff6eeb0 1mo ago
      So you put it in a loop and tell it to find the bugs in the code it wrote. What's the issue?
      - krona 1mo ago
        This is a self confession if I ever saw one.
        a2ff6eeb0 1mo ago

black_knight 1mo ago

This is where I believe strong typing (like, Haskell-strong or stronger) and functional programming in general will be a win. The confidence I have that my fixes are localised when fixing Haskell code is infinitely stronger than fixing even Java, not speak about C, code.

astrange 1mo ago
Haskell's type system would not easily prevent this bug. It's not good at numeric/logic issues like that. When people say "Haskell makes it impossible to write bugs" they mean "Haskell has enums" (ADTs).
- black_knight 1mo ago
  I am not claiming you cant write buggy code in Haskell! But following good functional style, your bug will more likely be compartmentalised, and fixing it will not break some other part of your program.
  - StilesCrisis 1mo ago
    You can write good functional code in many languages. (Even C++!)
    - black_knight 1mo ago
      Sure! I have done my fair share of pretending Java and C++ support my functional style. But at the end of the day, you have better support for writing that style in a real functional programming language. And I wonder how well one can enforce a functional style in say Java or C++ upon the LLMs. Who knows, they might be great at it?
- _jackdk_ 1mo ago
  Liquid Haskell might require you to prove that the divisor is nonzero, but even in standard Haskell there's common idioms for ensuring that a list is non-empty (data NonEmpty a = a :| [a]) or that text is non-empty (newtype NonEmptyText = NonEmptyText Text, with non-exported constructor, helpers like make :: Text -> NonEmptyText, or more advanced tricks like https://exploring-better-ways.bellroy.com/haskell-koan-type-checked-non-empty-strings.html https://exploring-better-ways.bellroy.com/haskell-koan-type-... ). The big problem preventing this approach from working for numbers is that it's just so cumbersome there. Most of this is because all the arithmetic operators are bundled into a single Num typeclass, and `fromInteger :: Num a => Integer -> a` has a type that's impossible for a "non-zero number" wrapper to satisfy.

macless 1mo ago

[flagged]

sadfgknerknksdf 1mo ago

If finding the bugs with LLMs is easy. Then making sure it doesn't break something else is just LLMs finding no bugs. Easy.

BikiniPrince 1mo ago

That hasn’t been that bad. My real issue has been the time sink involved in following along with the maintainer and jumper through their hoops. Even after I demonstrate a flaw and a potential fix. My schedule is just so busy I need to pencil in time to deal with them.