3 ms·

Sure, but behaviors that never have a bug or regression don't get a test. Software of this kind of complexity has all kinds of behavior that has never been bro

by kelnos 3mo ago

Sure, but behaviors that never have a bug or regression don't get a test. Software of this kind of complexity has all kinds of behavior that has never been broken, and doesn't have a specific test written for it. Getting an extensive test suite passing is certainly orders of magnitude better than having no test suite at all, but it still doesn't tell you as much as you need to know. I would absolutely never trust an LLM Postgres rewrite (in any language) in production based on "only" Postgres's test suite passing.

bob1029 3mo ago
> Software of this kind of complexity has all kinds of behavior that has never been broken This space of things is astronomically larger than the space of things expressly covered by any test suite. "Program testing can be used to show the presence of bugs, but never to show their absence." -Edsger W. Dijkstra
- w4der 3mo ago
  I've also seen situations where a customer reports a bug, the fix breaks some regression, and the updated behavior to work around the fix breaking the regressions turns into an undocumented feature.
  - derdi 3mo ago
    How do you break a regression? A regression is breakage. Are you one of the people who use "regression" to mean "regression test"? Did Codex learn this from you? I hate it.
    - w4der 3mo ago
      Yeah, sorry, it was a codebase with _only_ regression tests dating to the 80s, so we called everything a "regression".
- mbrock 3mo ago
  The same basically holds for proofs in the absence of coherent global correctness criteria like, say, confluence and normalization for a lambda calculus, or soundness and completeness for a logic. Fable's napkin estimate of the effort required to produce a passable reference semantics for Postgres, which would involve novel discoveries in denotational semantics of concurrent transactions and so on, might be in the ballpark of 30–60 years of PhD level work. So realistically I think the only way to validate a Postgres implementation involves differential testing, fuzzing, acceptance test suites, etc. And still you'll have bugs that need to be hammered out the good old fashioned way.
gblargg 3mo ago
Or even a human rewrite merely because some language is the current fad. A rewrite in a different language should be done for very good reasons, to solve problems that are bigger than the costs of all the bugs that will be introduced.
gb2d_hn 3mo ago
Agreed.And a rewrite in another language creates a high probability of a change in behaviour
jghn 3mo ago
Perhaps before embarking on one of these rewrites the first step should be a heavy round of mutation testing and property based testing. Contribute any new testing code from this back to the original project. And *then* embark on the rewrite.
vitally3643 3mo ago
If that's your concern, then your argument becomes "software should never change". Why dare patch any bug ever? It might be load-bearing in some unknown, undocumented, unsupported workflow somewhere in the world. No test imaginable can catch that apart from the scream test. There are reasonable arguments against language ports, but this is not one. You're making an argument against code changing at all ever.

by kelnos 3mo ago

bob1029 3mo ago

> Software of this kind of complexity has all kinds of behavior that has never been broken This space of things is astronomically larger than the space of things expressly covered by any test suite. "Program testing can be used to show the presence of bugs, but never to show their absence." -Edsger W. Dijkstra

w4der 3mo ago
I've also seen situations where a customer reports a bug, the fix breaks some regression, and the updated behavior to work around the fix breaking the regressions turns into an undocumented feature.
- derdi 3mo ago
  How do you break a regression? A regression is breakage. Are you one of the people who use "regression" to mean "regression test"? Did Codex learn this from you? I hate it.
  - w4der 3mo ago
    Yeah, sorry, it was a codebase with _only_ regression tests dating to the 80s, so we called everything a "regression".
mbrock 3mo ago
The same basically holds for proofs in the absence of coherent global correctness criteria like, say, confluence and normalization for a lambda calculus, or soundness and completeness for a logic. Fable's napkin estimate of the effort required to produce a passable reference semantics for Postgres, which would involve novel discoveries in denotational semantics of concurrent transactions and so on, might be in the ballpark of 30–60 years of PhD level work. So realistically I think the only way to validate a Postgres implementation involves differential testing, fuzzing, acceptance test suites, etc. And still you'll have bugs that need to be hammered out the good old fashioned way.

gblargg 3mo ago

Or even a human rewrite merely because some language is the current fad. A rewrite in a different language should be done for very good reasons, to solve problems that are bigger than the costs of all the bugs that will be introduced.

gb2d_hn 3mo ago

Agreed.And a rewrite in another language creates a high probability of a change in behaviour

jghn 3mo ago

Perhaps before embarking on one of these rewrites the first step should be a heavy round of mutation testing and property based testing. Contribute any new testing code from this back to the original project. And *then* embark on the rewrite.

vitally3643 3mo ago

If that's your concern, then your argument becomes "software should never change". Why dare patch any bug ever? It might be load-bearing in some unknown, undocumented, unsupported workflow somewhere in the world. No test imaginable can catch that apart from the scream test. There are reasonable arguments against language ports, but this is not one. You're making an argument against code changing at all ever.