3 ms·

I'm skeptical. Most of us maintaining medium sized codebases or larger are constantly fighting nondeterminism in the form of flaky tests. I can't imagine choosi

I'm skeptical. Most of us maintaining medium sized codebases or larger are constantly fighting nondeterminism in the form of flaky tests. I can't imagine choosing a design that starts with nondeterminism baked in. And if you're really dead-set on paying nondeterminism to get more coverage, property-based testing has existed for a long time and has a comparatively solid track record.

mrkeen 1y ago
Couldn't put it better myself. I have the toughest time trying to communicate why f(x) should equal f(x) in the general case.
Garlef 1y ago
Hm... I think you have a good point. Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes? I think there are also other problems (inlining a verbal description makes the codebase verbose, writing a precise, non-ambiguous verbal description might be more work than writing unit tests)
- carlmr 1y ago
  >Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes? That would be good anyway to keep the costs reasonable.
IshKebab 1y ago
I agree. I want this as a code review tool to check if people forgot to update comments - "it looks like this now adds instead of multiplies, but the comment says otherwise; did you forget to update it?". Seems of dubious value as unit tests. LLMs don't seem to be quite smart enough for that in my experience, unless your bugs are really as trivial as adding instead of multiplying, in which case god help you.
Davidbrcz 1y ago
Many good and prolific approaches are non deterministic such as fuzzing or property-based testing,
- RainyDayTmrw 1y ago
  Both fuzzing and property-based testing, under certain assumptions, produce only false negatives, never false positives. That is a very desirable trait, because then you can at least confidently fail CI on a positive, because either your code is wrong or your test is wrong, and you've got something to fix no matter what.

mrkeen 1y ago

Couldn't put it better myself. I have the toughest time trying to communicate why f(x) should equal f(x) in the general case.

Garlef 1y ago

Hm... I think you have a good point. Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes? I think there are also other problems (inlining a verbal description makes the codebase verbose, writing a precise, non-ambiguous verbal description might be more work than writing unit tests)

carlmr 1y ago
>Maybe the non-determinism can be reduced by caching: Just reevaluate the spec if the code actually changes? That would be good anyway to keep the costs reasonable.

IshKebab 1y ago

I agree. I want this as a code review tool to check if people forgot to update comments - "it looks like this now adds instead of multiplies, but the comment says otherwise; did you forget to update it?". Seems of dubious value as unit tests. LLMs don't seem to be quite smart enough for that in my experience, unless your bugs are really as trivial as adding instead of multiplying, in which case god help you.

Davidbrcz 1y ago

Many good and prolific approaches are non deterministic such as fuzzing or property-based testing,

RainyDayTmrw 1y ago
Both fuzzing and property-based testing, under certain assumptions, produce only false negatives, never false positives. That is a very desirable trait, because then you can at least confidently fail CI on a positive, because either your code is wrong or your test is wrong, and you've got something to fix no matter what.