5 ms·
The actual efficiency claim (which is also likely incorrect) is inverted from the original article, "We examined the conversion rates of approximately 2,300 ind
by anymouse123456 2y ago
The actual efficiency claim (which is also likely incorrect) is inverted from the original article, "We examined the conversion rates of approximately 2,300 individual test cases spread out within 338 files. Among these, approximately 500 test cases were successfully converted, executed, and passed. This highlights how effective AI can be, leading to a significant saving of 22% of developer time."
Reading that leads me to believe that 22% of the conversions succeeded and someone at Slack is making up numbers about developer time.
- taylorfinley 2y ago> approximately 500 test cases were successfully converted, executed, and passed. How many of these passing tests are still actually testing anything? Do they test the tests?
- BigJono 2y ago> 500 test cases were successfully converted, executed, and passed. Wonder what "successfully converted" means? A converted test executing and passing doesn't tell you whether it's still testing the same thing as before.
- Phemist 2y agoThat suggests a test suite to test the test suite, which again would suggest another test suite to test the test suite testing test suite... In the end, it is test suites all the way down.
- eru 2y agoYou could evaluate the quality of your test suite with something other than a test suite.
- netdevnet 2y agosuch as?
- cdogl 2y agoA developer who knows the code and will own the consequences can review and merge it - easy. Just not sure why the LLM needed to get involved in the first place.
- michaelt 2y agoPresumably because they've got 500 test cases?
- eru 2y agoWell, you need someone to write the code, and someone (else) to review the code. In this situation you would replace the author with the LLM, but leave the reviewer as human. It's not as pointless as you make it out to be. You still save one human.
- deleted 2y ago[deleted]
- ZeroGravitas 2y agoYou can run mutation tests that intentionally seed errors into the codebase (flipping Boolean and to or, for example) and rerun the tests. A good test suite will catch these errors. Something that is effectively a noop to get to green will not.
- lvncelot 2y agoThere actually is a way to test your test suite: mutation tests [1]. Basically, you change the covered codebase (invert an if statement, change a variable, etc.) and expect the tests to then fail. If the tests actually survive the mutations, they might not be good enough. [1] https://en.wikipedia.org/wiki/Mutation_testing https://en.wikipedia.org/wiki/Mutation_testing
- Scea91 2y agoAlso given that there are huge differences in complexity between tests, how do we know that the successful 22% are not just trivial one-liner tests? Thinking about the test suite in my current project there is a clear Pareto distribution with majority of tests being simple or almost trivial.
- suplix 2y ago[flagged]
- ghostzilla 2y ago> You don’t need unit tests if you have integration tests. Which is why, as per Jim Coplien, most unit testing is waste. But converting one type of unit tests into another is a perfect showcase for AI-generated code. They could have even kept just the prompts in the source and regenerate the tests on every run, were it not for inaccuracy, temperature, and the high cost of running.
- desdenova 2y agoSomeone will still have to review and validate all the tests, which may take more time than rewriting the code.
- robertlagrant 2y agoYep. Honestly, I don't know when I last read an InfoQ article all the way through. Too painful.
- madeofpalk 2y agoIf you're reviewing the tests (and not just the results) after conversion by automation, wouldn't someone else also review the tests converted by a person rewriting them manually?
- imtringued 2y agoThis is something people seemingly don't grasp about LLMs. If you want "Human alignment", then you will need humans in the loop either way.
- jacobr1 2y agoFor a test that is already passing, the validation is already there. The human is reviewing the PR, but reviewing a diff is much less time intensive when you can presume it basically works, and you are just assessing if the logic make makes sense relative to the prior code.
- AmalgatedAmoeba 2y agoYes, the 80% claim comes from comparing 9 tests converted by both the conversion tool and humans and comparing the quality - “80% of the content within these files was accurately converted, while the remaining 20% required manual intervention.” Not sure what to make of it since they claim only 16% of files get fully converted.