4 ms·

> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. The problem I have with LLM generated tests is that it seem

by tivert 3y ago

> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test. At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.

kromem 3y ago
Keeping them separated would also improve future training.
chii 3y ago
> you've got someone who can judge if it's the system being stupid or if it's the test. but why couldn't this be done even with the llm generated test cases?
- anoopelias 3y ago
  In that case, I think the point is the difference between what is LLM "assisted" tests (like say Copilot) vs. LLM "owned" tests.
- deleted 3y ago
  [deleted]
christkv 3y ago
I mean if I can use it to generate coverage for a method then prune the tests it will still save me hours. Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.
renewiltord 3y ago
Yeah, but that means you can really live Hyrum's Law.
js8 3y ago
> One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test. This is an instance of a more general problem, which I call "the problem of unwanted change". If you have an automated system which can change itself, how do you know a change is actually intended/correct or merely a symptom of a bug, failure or imperfect knowledge the automation has? That's why I think human supervision is always needed to an extent, to determine what scenario has occured. This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).
ithkuil 3y ago
Writing tests is indeed a great opportunity for finding bugs. But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that's useful property even if you have bugs and the refactoring preserves them faithfully. The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you've done is to encode the current behaviour, as advertised. Perhaps this problem can be just solved by not calling these tests "tests" but something like "behavioural snapshots" or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)
- drothlis 3y ago
  https://en.wikipedia.org/wiki/Characterization_test https://en.wikipedia.org/wiki/Characterization_test aka snapshot tests.
  - ithkuil 3y ago
    I'm a huge fan. But sometimes it's hard to produce the snapshots.
- the_gipsy 3y ago
  Sometimes you could replace snapshots with checksums. In other words, they're just alerts that code changed, or make changing code unnecessarily tedious.
- ape_key 3y ago
  Unit tests are dead weight during refactoring. Integration tests are useful. Code coverage metrics are most easily met by writing unit tests. Unit tests are tedious to write. If you have a robot writing unit tests, I do not want to see your codebase. Refactor all day long. I’m not going near it.
  - TillE 3y ago
    It is not at all uncommon for me to refactor the internals of a unit (ie, a class) in order to add a feature. Unit tests are extremely useful here.
adrianN 3y ago
In sufficiently large systems there is some value in test that just detect changed behavior, even if the behavior is buggy. Parts of the code probably rely on the bugs and accidentally (or intentionally) fixing them can lead to more severe problems. Of course these kinds of tests are no replacement for tests that check actual requirements.
- sixstringtheory 3y ago
  > Parts of the code probably rely on the bugs The combination of pieces is what should be tested then, not the behavior of each individual piece “just in case something needs it that way”. You’ll never be able to change anything with that approach. And why shouldn’t bugs be fixed anyways?
  - therealdrag0 3y ago
    It’s not that bugs shouldn’t be fixed it’s that they should be fixed intentionally, and with awareness of what the fix might break so you can make it backwards compatible if necessary.
MichaelNolan 3y ago
> LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior For a new project or a project under active development, I agree that auto generating tests is probably a bad idea. But there are countless legacy systems with low coverage that are in maintenance mode. For those, generating tests that verify the current behavior is super useful. It lets someone make a change, and see that everything else stayed the same.
ape_key 3y ago
If a test is worth having, it’s worth writing by hand. Throw away mandatory code coverage tech debt instead of adding artificially-intelligent tech debt on top. The best I can see a use for something like this is more like a linter than a test writer. “Robot, find weird things in the code and bring them to me for review.” Then you write the tests yourself.
- sixstringtheory 3y ago
  But this tool doesn’t know what is weird and what isn’t. What you’re talking about is something more like a fuzzer or static analyzer.
theptip 3y ago
On the other hand, in a codebase where test coverage is poor and average tenure of engineers is ~1yr, a significant impediment is getting the initial test scaffold set up; maybe I don’t know how to build factories for all the incidental inputs required to test my code, but I do know how the code itself should behave. If LLMs can help you to scaffold a test, and make it easy for you to write the business logic validation, I think that would be a big win. On the other hand, if the generated tests are like most UTs, they will be over-coupled to the implementation and therefore slow down development. You might even see folks start deleting all the tests and regenerating them as part of a big diff, if it’s too hard to fix individual test cases that are failing due to coupling rather than any logic issue.

by tivert 3y ago

kromem 3y ago

Keeping them separated would also improve future training.

chii 3y ago

> you've got someone who can judge if it's the system being stupid or if it's the test. but why couldn't this be done even with the llm generated test cases?

anoopelias 3y ago
In that case, I think the point is the difference between what is LLM "assisted" tests (like say Copilot) vs. LLM "owned" tests.
deleted 3y ago
[deleted]

christkv 3y ago

I mean if I can use it to generate coverage for a method then prune the tests it will still save me hours. Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.

renewiltord 3y ago

Yeah, but that means you can really live Hyrum's Law.

js8 3y ago

> One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test. This is an instance of a more general problem, which I call "the problem of unwanted change". If you have an automated system which can change itself, how do you know a change is actually intended/correct or merely a symptom of a bug, failure or imperfect knowledge the automation has? That's why I think human supervision is always needed to an extent, to determine what scenario has occured. This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).

ithkuil 3y ago

Writing tests is indeed a great opportunity for finding bugs. But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that's useful property even if you have bugs and the refactoring preserves them faithfully. The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you've done is to encode the current behaviour, as advertised. Perhaps this problem can be just solved by not calling these tests "tests" but something like "behavioural snapshots" or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)

drothlis 3y ago
https://en.wikipedia.org/wiki/Characterization_test https://en.wikipedia.org/wiki/Characterization_test aka snapshot tests.
- ithkuil 3y ago
  I'm a huge fan. But sometimes it's hard to produce the snapshots.
the_gipsy 3y ago
Sometimes you could replace snapshots with checksums. In other words, they're just alerts that code changed, or make changing code unnecessarily tedious.
ape_key 3y ago
Unit tests are dead weight during refactoring. Integration tests are useful. Code coverage metrics are most easily met by writing unit tests. Unit tests are tedious to write. If you have a robot writing unit tests, I do not want to see your codebase. Refactor all day long. I’m not going near it.
- TillE 3y ago
  It is not at all uncommon for me to refactor the internals of a unit (ie, a class) in order to add a feature. Unit tests are extremely useful here.

adrianN 3y ago

In sufficiently large systems there is some value in test that just detect changed behavior, even if the behavior is buggy. Parts of the code probably rely on the bugs and accidentally (or intentionally) fixing them can lead to more severe problems. Of course these kinds of tests are no replacement for tests that check actual requirements.

sixstringtheory 3y ago
> Parts of the code probably rely on the bugs The combination of pieces is what should be tested then, not the behavior of each individual piece “just in case something needs it that way”. You’ll never be able to change anything with that approach. And why shouldn’t bugs be fixed anyways?
- therealdrag0 3y ago
  It’s not that bugs shouldn’t be fixed it’s that they should be fixed intentionally, and with awareness of what the fix might break so you can make it backwards compatible if necessary.

MichaelNolan 3y ago

> LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior For a new project or a project under active development, I agree that auto generating tests is probably a bad idea. But there are countless legacy systems with low coverage that are in maintenance mode. For those, generating tests that verify the current behavior is super useful. It lets someone make a change, and see that everything else stayed the same.

ape_key 3y ago

If a test is worth having, it’s worth writing by hand. Throw away mandatory code coverage tech debt instead of adding artificially-intelligent tech debt on top. The best I can see a use for something like this is more like a linter than a test writer. “Robot, find weird things in the code and bring them to me for review.” Then you write the tests yourself.

sixstringtheory 3y ago
But this tool doesn’t know what is weird and what isn’t. What you’re talking about is something more like a fuzzer or static analyzer.

theptip 3y ago

On the other hand, in a codebase where test coverage is poor and average tenure of engineers is ~1yr, a significant impediment is getting the initial test scaffold set up; maybe I don’t know how to build factories for all the incidental inputs required to test my code, but I do know how the code itself should behave. If LLMs can help you to scaffold a test, and make it easy for you to write the business logic validation, I think that would be a big win. On the other hand, if the generated tests are like most UTs, they will be over-coupled to the implementation and therefore slow down development. You might even see folks start deleting all the tests and regenerating them as part of a big diff, if it’s too hard to fix individual test cases that are failing due to coupling rather than any logic issue.