3 ms·

I don't want to review this whole thing but one part in particular seems way off. [Caveat: I sorta-read the original paper shortly after it was posted, my memor

I don't want to review this whole thing but one part in particular seems way off. [Caveat: I sorta-read the original paper shortly after it was posted, my memory is fuzzy and I am only skimming it now.] From the blog: > Most of the test cases created by Meta’s TestGen-LLM only covered an extra 2.5 lines. However, one test case covered 1326 lines! The value of that one test case is exponentially more valuable than most of the previous test cases and exponentially improves the value of TestGen-LLM. LLMs can vigorously “think outside the box” and the value of catching unexpected edge cases is very high here. Of course "exponentially more valuable" should set off your BS detector. But to verify, from the paper: > However, this result arose due to a single test case, which achieved 1,326 lines covered. This test case managed to ‘hit the jackpot’ in terms of unit test coverage for a single test case. Because TestGen-LLM is typically adding to the existing coverage, and seeking to cover corner cases, the typical expected number of lines of code covered per test case is much lower....The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5. This is a more realistic assessment of the expected additional line coverage from a single test generated by TestGen-LLM. Nowhere do the authors mention "unexpected edge cases" or "thinking outside the box." They clearly present this 1,326 lines of coverage test as a fluke, e.g. maybe the test case checked one branch of a horrible switch statement, or perhaps it was even a fluke in how code coverage is counted. It is noteworthy that the authors do not seem to have looked into it any further, even in the "qualitative results" section. Inaccurate editorializing really doesn't help anyone. The internet is too damn full of people pretending to understand things they pretended to read.

engineercodex 3y ago
Hey! Thanks for your comment - I'm the one who wrote this article. I wasn't trying to say that the paper authors talked about "unexpected edge cases" or "thinking outside the box." I edited the post to be more clear that some of these takeaways are my own opinions. This article is less of a summary of a paper and rather commentary on what the results of the paper entails. After all, Hacker News is meant for discussion :) I will say though that I do believe that I still stand by the "exponentially more valuable" portion. I think the fact that LLMs can fluke their way into "hitting a jackpot" in terms of test coverage is exactly why they're so valuable. When you have something constantly trying out different combinations, if it hits even one jackpot, like in the paper, it's extremely valuable to the team. It's a case that could have been either non-obvious or simply too tedious to write a test for manually. I think there's tremendous value in that, especially speaking as someone who has spend way too much time simply figuring out how to test something within a Big Tech codebase (F/G) when I already knew what to test.
- digdugdirk 3y ago
  Thanks for engaging with the above constructive criticism, it's a refreshing change from what is sadly the norm. One additional question - do you forsee any issues with this application where LLMs enter a non-value add "doom loop"? I can imagine a scenario where a test generation LLM gets hooked on the lower value simplistic tests, and yet management sees such a huge increase on the test metric ("100x increase in unit tests in an afternoon? Let's do it again!") that they continue to bloat the test suite to near-infinity. Now we're in a situation where all future training data is now training on complete cesspool of meaningless tests that technically add coverage, but mostly just to cover an edge case that only an LLM would create. Not sure if that makes sense, but tl;dr - having LLMs in the loop for both code creation and code testing seems like it's a feedback loop waiting to happen, with what seems like solely negative repercussions for future LLM training data.
  - samstave 3y ago
    Perhaps there should be domains of focus for the test LLMs - even if they are clones, but assigned to only a particular domain, then their results have to be PR's etc... Why not treat every LLM as a dev contributing to git such that Humans, or other LLms need to gatekeep in case something like that happens? (start by treating them as Interns, rather than Professors with office hours)
  - engineercodex 3y ago
    I could see that, and I wouldn't want LLMs just generating tests willy-nilly with no human oversight. I don't have any trust in the reasoning ability of LLMs at all. Rather, I prefer to view LLMs as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.
- camkego 3y ago
  Pedantic warning here. In fast and loose day-to-day common English language "exponentially more" means "fast growth" or "a whole lot". But that usage is meaningless! Why?, technically, you can't have exponential growth without a dependent variable. You can have exponential growth as a function of time, height, spend, distance, any freaking metric or variable. But it has to be as a function of a value. You CAN'T have exponential growth that is not a function of some value or variable or input. I suppose in this case you could argue you have exponential growth as a function of the discrete using-an-LLM or not-using-an-LLM, but I've never heard of exponential growth as a function of a discrete. Often people using the term "exponential growth" in common English don't understand what it means. Sorry.
fermentation 3y ago
The incentives at meta around code production are all wrong. The team behind this is absolutely gearing this around lines of code and number of diffs produced. This'll just be another codegen tool creating another mountain of code that is difficult to debug.

engineercodex 3y ago

Hey! Thanks for your comment - I'm the one who wrote this article. I wasn't trying to say that the paper authors talked about "unexpected edge cases" or "thinking outside the box." I edited the post to be more clear that some of these takeaways are my own opinions. This article is less of a summary of a paper and rather commentary on what the results of the paper entails. After all, Hacker News is meant for discussion :) I will say though that I do believe that I still stand by the "exponentially more valuable" portion. I think the fact that LLMs can fluke their way into "hitting a jackpot" in terms of test coverage is exactly why they're so valuable. When you have something constantly trying out different combinations, if it hits even one jackpot, like in the paper, it's extremely valuable to the team. It's a case that could have been either non-obvious or simply too tedious to write a test for manually. I think there's tremendous value in that, especially speaking as someone who has spend way too much time simply figuring out how to test something within a Big Tech codebase (F/G) when I already knew what to test.

digdugdirk 3y ago
Thanks for engaging with the above constructive criticism, it's a refreshing change from what is sadly the norm. One additional question - do you forsee any issues with this application where LLMs enter a non-value add "doom loop"? I can imagine a scenario where a test generation LLM gets hooked on the lower value simplistic tests, and yet management sees such a huge increase on the test metric ("100x increase in unit tests in an afternoon? Let's do it again!") that they continue to bloat the test suite to near-infinity. Now we're in a situation where all future training data is now training on complete cesspool of meaningless tests that technically add coverage, but mostly just to cover an edge case that only an LLM would create. Not sure if that makes sense, but tl;dr - having LLMs in the loop for both code creation and code testing seems like it's a feedback loop waiting to happen, with what seems like solely negative repercussions for future LLM training data.
- samstave 3y ago
  Perhaps there should be domains of focus for the test LLMs - even if they are clones, but assigned to only a particular domain, then their results have to be PR's etc... Why not treat every LLM as a dev contributing to git such that Humans, or other LLms need to gatekeep in case something like that happens? (start by treating them as Interns, rather than Professors with office hours)
- engineercodex 3y ago
  I could see that, and I wouldn't want LLMs just generating tests willy-nilly with no human oversight. I don't have any trust in the reasoning ability of LLMs at all. Rather, I prefer to view LLMs as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.
camkego 3y ago
Pedantic warning here. In fast and loose day-to-day common English language "exponentially more" means "fast growth" or "a whole lot". But that usage is meaningless! Why?, technically, you can't have exponential growth without a dependent variable. You can have exponential growth as a function of time, height, spend, distance, any freaking metric or variable. But it has to be as a function of a value. You CAN'T have exponential growth that is not a function of some value or variable or input. I suppose in this case you could argue you have exponential growth as a function of the discrete using-an-LLM or not-using-an-LLM, but I've never heard of exponential growth as a function of a discrete. Often people using the term "exponential growth" in common English don't understand what it means. Sorry.

fermentation 3y ago

The incentives at meta around code production are all wrong. The team behind this is absolutely gearing this around lines of code and number of diffs produced. This'll just be another codegen tool creating another mountain of code that is difficult to debug.