4 ms·

> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage

> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. That doesn't seem great?

sampli 3y ago
Considering human tests should already have a high coverage rate, if the 25% that increased coverage were actual good tests, I think it’s a useful tool
- bornfreddy 3y ago
  I was actually surprised they had less than 75% coverage to start with.
  - sanxiyn 3y ago
    25% of test cases increased coverage, not coverage was increased by 25%. For example, if they started with 90% coverage and each test case increased coverage by 0.1% and there were 100 test cases, final coverage was 92.5%.
reissbaker 3y ago
It looks like it only gets committed if passes all of those checks (i.e. no humans have to look at it unless it's actually reliably passing and does increase code coverage). 25% code coverage improvement that passes reliably, for the cost of the electricity required for the GPUs, seems pretty cheap. Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.
- sanxiyn 3y ago
  > 25% code coverage improvement The paper does not report code coverage improvement and it is probably not 25%. The paper does say this: > The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5.
Jtsummers 3y ago
Their abstract doesn't match their actual paper contents. That's unfortunate. Their summary indicates rates in terms of test cases (repeating your quote a bit): > 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication] The actual report talks about test classes, where each class has one or more test cases. > (1) 75% of test classes had at least one new test case that builds correctly. > (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably. > (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target. Those are two very different statements. They even have a footnote acknowledging this: > For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class. But then in their conclusion they misrepresent their findings again, like the abstract: > When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.
Retr0id 3y ago
All three are machine-verifiable, so you can easily filter out the ones that don't work, right?

> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. That doesn't seem great?

sampli 3y ago

Considering human tests should already have a high coverage rate, if the 25% that increased coverage were actual good tests, I think it’s a useful tool

bornfreddy 3y ago
I was actually surprised they had less than 75% coverage to start with.
- sanxiyn 3y ago
  25% of test cases increased coverage, not coverage was increased by 25%. For example, if they started with 90% coverage and each test case increased coverage by 0.1% and there were 100 test cases, final coverage was 92.5%.

reissbaker 3y ago

It looks like it only gets committed if passes all of those checks (i.e. no humans have to look at it unless it's actually reliably passing and does increase code coverage). 25% code coverage improvement that passes reliably, for the cost of the electricity required for the GPUs, seems pretty cheap. Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.

sanxiyn 3y ago
> 25% code coverage improvement The paper does not report code coverage improvement and it is probably not 25%. The paper does say this: > The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5.

Jtsummers 3y ago

Their abstract doesn't match their actual paper contents. That's unfortunate. Their summary indicates rates in terms of test cases (repeating your quote a bit): > 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication] The actual report talks about test classes, where each class has one or more test cases. > (1) 75% of test classes had at least one new test case that builds correctly. > (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably. > (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target. Those are two very different statements. They even have a footnote acknowledging this: > For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class. But then in their conclusion they misrepresent their findings again, like the abstract: > When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.

Retr0id 3y ago

All three are machine-verifiable, so you can easily filter out the ones that don't work, right?