4 ms·

AI is a great use for this kind of boring, rote translation where precision is important. Humans are quite bad at it and tend to make mistakes. In either case t

by rjh29 3mo ago

AI is a great use for this kind of boring, rote translation where precision is important. Humans are quite bad at it and tend to make mistakes. In either case the focus should be on improving testing, not trying to manually verify if the translation was correct by eye.

IsTom 3mo ago
With programs large enough tests aren't going to ever be enough. Formal verification might work, but then who checks the specification for bugs?
- copperx 3mo ago
  In the case of rewrites, the specification is the original behavior, no? bugs and all.
  - hoppp 3mo ago
    There is often no spec, just the old code, copied with the old bugs and with new ones sprinkled on top.
  - IsTom 3mo ago
    Deciding if two programs do the same thing is provably impossible in the strict mathematical sense.
- cyphar 3mo ago
  I really wonder where all of these people who believe that tests perfectly encapsulate the behaviour of software come from. Maybe it's because LLMs happen to work better when you give them acceptance criteria and people struggle to distinguish between "better" and "good"?
  - 27183 3mo ago
    The real test is years in production. Over time your test suite grows when bugs are found and fixed, but not every bugfix necessarily gets a test, and it's very rare that a bugfix is exhaustively tested. Relying on the test suite as a directional indicator that your vibecoded rewrite functions something like the original is probably sensible. But it isn't "done" until you've run it in production for at least as long as the original. And that's where it all falls apart, because maintenance will be a nightmare. Nobody knows how the new thing works.
    - rjh29 3mo ago
      Ladybird's Rust port of the JS engine was a good example. Compare the output byte-for-byte, run both in production (with the new code disabled but checked) before releasing. It was LLM-translated but done carefully.
hoppp 3mo ago
I have an issue with the precision of generated code. LLMs sometimes confidently leave things out or they will overbuild. I use them all the time but mistakes happen. It's not exactly a scalpel, more like a sledge hammer.
pier25 3mo ago
If precision is important then non deterministic AI is simply not a good tool.

by rjh29 3mo ago

IsTom 3mo ago

With programs large enough tests aren't going to ever be enough. Formal verification might work, but then who checks the specification for bugs?

copperx 3mo ago
In the case of rewrites, the specification is the original behavior, no? bugs and all.
- hoppp 3mo ago
  There is often no spec, just the old code, copied with the old bugs and with new ones sprinkled on top.
- IsTom 3mo ago
  Deciding if two programs do the same thing is provably impossible in the strict mathematical sense.
cyphar 3mo ago
I really wonder where all of these people who believe that tests perfectly encapsulate the behaviour of software come from. Maybe it's because LLMs happen to work better when you give them acceptance criteria and people struggle to distinguish between "better" and "good"?
- 27183 3mo ago
  The real test is years in production. Over time your test suite grows when bugs are found and fixed, but not every bugfix necessarily gets a test, and it's very rare that a bugfix is exhaustively tested. Relying on the test suite as a directional indicator that your vibecoded rewrite functions something like the original is probably sensible. But it isn't "done" until you've run it in production for at least as long as the original. And that's where it all falls apart, because maintenance will be a nightmare. Nobody knows how the new thing works.
  - rjh29 3mo ago
    Ladybird's Rust port of the JS engine was a good example. Compare the output byte-for-byte, run both in production (with the new code disabled but checked) before releasing. It was LLM-translated but done carefully.

hoppp 3mo ago

I have an issue with the precision of generated code. LLMs sometimes confidently leave things out or they will overbuild. I use them all the time but mistakes happen. It's not exactly a scalpel, more like a sledge hammer.

pier25 3mo ago

If precision is important then non deterministic AI is simply not a good tool.