3 ms·
> 500 test cases were successfully converted, executed, and passed. Wonder what "successfully converted" means? A converted test executing and passing doesn't
by BigJono 2y ago
> 500 test cases were successfully converted, executed, and passed.
Wonder what "successfully converted" means? A converted test executing and passing doesn't tell you whether it's still testing the same thing as before.
- Phemist 2y agoThat suggests a test suite to test the test suite, which again would suggest another test suite to test the test suite testing test suite... In the end, it is test suites all the way down.
- eru 2y agoYou could evaluate the quality of your test suite with something other than a test suite.
- netdevnet 2y agosuch as?
- cdogl 2y agoA developer who knows the code and will own the consequences can review and merge it - easy. Just not sure why the LLM needed to get involved in the first place.
- michaelt 2y agoPresumably because they've got 500 test cases?
- eru 2y agoWell, you need someone to write the code, and someone (else) to review the code. In this situation you would replace the author with the LLM, but leave the reviewer as human. It's not as pointless as you make it out to be. You still save one human.
- deleted 2y ago[deleted]
- ZeroGravitas 2y agoYou can run mutation tests that intentionally seed errors into the codebase (flipping Boolean and to or, for example) and rerun the tests. A good test suite will catch these errors. Something that is effectively a noop to get to green will not.
- lvncelot 2y agoThere actually is a way to test your test suite: mutation tests [1]. Basically, you change the covered codebase (invert an if statement, change a variable, etc.) and expect the tests to then fail. If the tests actually survive the mutations, they might not be good enough. [1] https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing