10 ms·

Automated Unit Test Improvement Using Large Language Models at Meta

301 points by mfiguiere 3y ago

newzisforsukas 3y ago
> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. That doesn't seem great?
- sampli 3y ago
  Considering human tests should already have a high coverage rate, if the 25% that increased coverage were actual good tests, I think it’s a useful tool
  - bornfreddy 3y ago
    I was actually surprised they had less than 75% coverage to start with.
    - sanxiyn 3y ago
      25% of test cases increased coverage, not coverage was increased by 25%. For example, if they started with 90% coverage and each test case increased coverage by 0.1% and there were 100 test cases, final coverage was 92.5%.
- reissbaker 3y ago
  It looks like it only gets committed if passes all of those checks (i.e. no humans have to look at it unless it's actually reliably passing and does increase code coverage). 25% code coverage improvement that passes reliably, for the cost of the electricity required for the GPUs, seems pretty cheap. Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.
  - sanxiyn 3y ago
    > 25% code coverage improvement The paper does not report code coverage improvement and it is probably not 25%. The paper does say this: > The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5.
- Jtsummers 3y ago
  Their abstract doesn't match their actual paper contents. That's unfortunate. Their summary indicates rates in terms of test cases (repeating your quote a bit): > 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication] The actual report talks about test classes, where each class has one or more test cases. > (1) 75% of test classes had at least one new test case that builds correctly. > (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably. > (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target. Those are two very different statements. They even have a footnote acknowledging this: > For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class. But then in their conclusion they misrepresent their findings again, like the abstract: > When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.
- Retr0id 3y ago
  All three are machine-verifiable, so you can easily filter out the ones that don't work, right?
bbor 3y ago
In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. …is that a good rate? I guess I have to read more and see if the unacceptable ones were silly mistakes like the ones that make us all do code review, or serious ones. I don’t think a human engineer with 25% failure rate would be very helpful, if it’s a certain kind of failure. As part of our overall mission to automate unit test generation for Android code, we have developed an automated test class improver, TestGen-LLM. Is that a good mission? I feel like the TDD people are turning over in their graves, or at least in their beds at home. But again something tells me that they caveat this later
- bagels 3y ago
  There is a lot of testless code in Facebook, nobody gets PSC points for that.
bilekas 3y ago
I'll admit its interesting, a 12 page paper by Meta employees to promote AI for developers. Even brought out the Sankey diagram. I'm probably wrong but if its published in this way, shouldn't the information be given to reproduce it ? Edit: This is not tinfoil hat, I just don't have the kind of data that meta has to learn from. So, maybe they released something ?
- seanmcdirmid 3y ago
  It’s an FSE 2024 paper, so I’m guessing the artifacts need theory or formal evaluation.
- refulgentis 3y ago
  If it's anything like Google, it's way too intimately tied to their infra and monorepo to release
  - oars 3y ago
    For these big tech companies, infrastructure is optimized for high scale & reliability over velocity & flexibility.
Roritharr 3y ago
I am currently looking into building something like this for a client with large (several > 3M LoCs) and old (started in 2001) Java Projects with low coverage. Interesting to read how it would work it you already have good coverage (I assume).
tivert 3y ago
> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test. At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.
- kromem 3y ago
  Keeping them separated would also improve future training.
- chii 3y ago
  > you've got someone who can judge if it's the system being stupid or if it's the test. but why couldn't this be done even with the llm generated test cases?
  - anoopelias 3y ago
    In that case, I think the point is the difference between what is LLM "assisted" tests (like say Copilot) vs. LLM "owned" tests.
  - deleted 3y ago
    [deleted]
- christkv 3y ago
  I mean if I can use it to generate coverage for a method then prune the tests it will still save me hours. Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.
- renewiltord 3y ago
  Yeah, but that means you can really live Hyrum's Law.
- 3y ago
TheChaplain 3y ago
In my experience writing tests is generally an outstanding method to determine code quality. If a test is complicated or coverage is hard to achieve, it's likely your tested code needs improvement.
- jillesvangurp 3y ago
  Testability of code is indeed a good benchmark for code quality. The things that make it hard to test code are also the things that are associated with low quality code. Something with low coupling, high cohesion, and low complexity should be easy to unit tests.
  - Jensson 3y ago
    Competitive programming code is extremely easy to test, but many would argue it isn't high quality.
planetjones 3y ago
From reading the PDF it seems that this ‘merely’ generates tests that will repeatedly pass i.e. that are not flaky. The main purpose is to create a regression test suite by having tests that pin the behaviour of existing code. This isn’t a replacement for developer written tests, which one would hope come with the knowledge of what the functional requirement is. Almost 20 years ago the company I worked for trialled AgitarOne - its promise was automagically generating test cases for Java code that help explore its behaviour. But also Agitar could create passing tests more or less automatically, which you could then use as a regression suite. Personally I never liked it, as it just led to too much stuff and it was something management didn’t really understand - to them if the test coverage had gone up then the quality must have too. I wonder how much better the LLM approach FB talk about here is compared to that though… http://www.agitar.com/solutions/products/agitarone.html http://www.agitar.com/solutions/products/agitarone.html
- seanmcdirmid 3y ago
  A lot of unit tests generated that way will simply be change detectors (fail when code changes) rather than regression tests (fail when bug is re-introduced). Those are pretty big distinctions, I don’t see LML’s getting here until they can ascertain tear correctness without just assuming good tests pass or depending on an oracle (the prompt will have to include behavior expectations somehow).
  - planetjones 3y ago
    That’s what I believe Facebook have created here, so you’re right ‘regression’ is a big word - the tests are more likely detecting change e.g. by asserting the existing behaviour of conditionals previously not executed.
  - sixstringtheory 3y ago
    This articulates the problem I’m having right now in an interesting way. I’m fine writing unit tests that validate business logic requirements or bug fixes, but writing tests that validate implementations to the point that they reimplement the same logic is a bit much. I want to figure out how to count the number of times a test has had to change with updated requirements vs how many defects they’ve prevented (vs how much wall clock time / compute resources they’ve consumed in running them).
hubraumhugo 3y ago
At a large insurance company I worked for, management set a target of 80% test coverage across our entire codebase. So people started writing stupid unit tests for getters and setters in Java DTOs to reach the goal. Of course devs also weren't allowed to change the coverage measuring rules in Sonar. As a young dev, it taught me that focusing only on KPIs can sometimes drive behaviors that don't align with the intended goals. A few well-thought out E2E test scenarios would probably have had a better impact on the software quality.
- awwaiid 3y ago
  True. However we might think about this differently if we flipped it to "our standard is 20% completely untested". Uncoverage communicates the value much better.
- farhanhubble 3y ago
  True of any metric when it becomes the goal in itself.
- oneshtein 3y ago
  Is it hard to write ONE test case for ALL getters and setters using reflection?
  - reactordev 3y ago
    at that point what are you testing? allocation/deallocation? Nothing is actually happening unless you depend on constructors/destructors when you use reflection to gen a mock type or real value type to inject. Or do you mean just assert true that all getters are not null? I agree that unit testing for the sake of KPI's is the wrong approach and unit testing functionality (as a means of documenting it, proving it still works as intended) is far better.
    - oneshtein 3y ago
      > at that point what are you testing? Memory. > Nothing is actually happening unless you depend on constructors/destructors when you use reflection to gen a mock type or real value type to inject. Just get an object from IoC container, then test that getFoo() == getFoo(setFoo(getFoo())) for every Foo with get and set methods, so you will have 80% of coverage for those getters and setters. For read-only properties, just get value and throw it away. > I agree that unit testing for the sake of KPI's is the wrong approach and unit testing functionality (as a means of documenting it, proving it still works as intended) is far better. Unit tests are for proving correctness of logic in a unit of code. Data structures are not logic. IMHO, such trivial parts of program should be ignored by coverage tool, but topic starter said that they cannot change rules. :-/
curtis3389 3y ago
TestGen-LLM is such a strange creation. I can see how it could be used as a first step in a refactoring or rewrite, but the emphasis on code coverage in the paper seems totally brain-broke. I suppose it'd be great if your org is already brain-broke and demanding high coverage, but TestGen-LLM won't make your project's code better in any way, and it'll increase the friction involved in actually implementing improvements. It'd be much more useful to generate edge-case tests that may or may not be passing, but TestGen-LLM relies on compiler errors and failing tests to filter out LLM garbage. The lack of any examples of the generated tests in the paper makes me suspect that they're the same as the rest of the LLM-generated code I have seen: amateurish.
- thomasahle 3y ago
  I recently had to refractory a project that had no tests what so ever. Having LLMs automatically generate a first draft of tests was very helpful. Even just to understand what the code was supposed to do.
Lio 3y ago
It’s papers like this that will act as the justification of the next round of FAANG lAIoffs. Regardless of how successful this approach is in the long term.
shardullavekar 3y ago
At unlogged.io, for some time - our primary focus was to auto-generate junit tests. The approach didn't take off for a few reasons: 1. A Lot of generated test code that no devs wanted to maintain. 2. The generated tests didn't simulate real-world scenarios. 3. Code coverage was a vanity metric. Devs worked around to reach their goals with scenarios that didn't matter. We are currently working on offering no-code replay tests that simulate all unique production scenarios and developers can replay locally while mocking external dependencies. Disclaimer: I am a founder at unlogged.io
pearjuice 3y ago
So write unit tests automatically, change code later and then regenerate the unit tests? Now the code has a bug but the unit tests pass. I'm already seeing this today with devs using ChatGPT to quickly get the "test boilerplate" over and over.
- afro88 3y ago
  You still need a human in the loop. I doubt Meta are letting it run blindly. More just automating part of the process and having a human decide what is and isn't committed.
regularfry 3y ago
I want to go the other way. Let me feed acceptance criteria in, have it generate tests that check them, and only then generate code that passes the tests. You can get close to this with Copilot, sometimes, in a fairly limited way, but why do I feel like nobody is focusing on doing it that way round?
oslac 3y ago
Is there still no type theoretic answer to unit testing? Does not the type or the class generally contain all the necessary information to unit test itself, assuming its a unit? That is, we should not have to even write these "theoretically". Just hit "compiler --unit_test <type>"
- deleted 3y ago
  [deleted]
- afro88 3y ago
  What you're describing is more or less fuzzing [1], at the unit level. I can't remember the names, but there are tools around that work like this at runtime (ie you define a test that executes functions from the test library that run tests based on input/output types and other user defined constraints). There's almost always more business logic to what a unit should do than it's types though. Depending on the language, the type system can only encode so much of that logic. Consider the opposite: can't the compiler generate implementations from types and interfaces? In most cases, no. LLMs are filling some of that gap though because they can use some surrounding context to return the high probability implementation (completion) from the interface or type definition. [1] https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing
aussieguy1234 3y ago
Not sure about improving. But for Tunnelmole (https://github.com/robbie-cahill/tunnelmole-client https://github.com/robbie-cahill/tunnelmole-client) I have used GPT-4 to generate unit tests just by showing it a TypeScript module and asking it to create tests.
Alifatisk 3y ago
Meta have been publishing fire content recent years
deleted 3y ago
[deleted]
emmender2 3y ago
Happy to see such a huge team collaborating on a project at any company. Perhaps becoz, it involves LLM and LLMs are hot, and everyone wants a piece of it.
tdiff 3y ago
I wonder what would be the cost of maintaining some huge auto-generated corpus of tests in the future. They need to provide some automated way not only to generate cases, but also to update them.
MeteorMarc 3y ago
So, assign to: LLM when lots of tests are broken after the next refactoring!
westurner 3y ago
"Automated Unit Test Improvement using Large Language Models at Meta" (2024) https://arxiv.org/abs/2402.09171 https://arxiv.org/abs/2402.09171 : > This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. [...] We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement. Coverage-guided unit test improvement might [with LLMs] be efficient too. https://github.com/topics/coverage-guided-fuzzing https://github.com/topics/coverage-guided-fuzzing : - e.g. Google/syzkaller is a coverage-guided syscall fuzzer: https://github.com/google/syzkaller https://github.com/google/syzkaller - Gitlab CI supports coverage-guided fuzzing: https://docs.gitlab.com/ee/user/application_security/coverage_fuzzing/ https://docs.gitlab.com/ee/user/application_security/coverag... - oss-fuzz, osv Additional ways to improve tests: Hypothesis and pynguin generate tests from type annotations. There are various tools to generate type annotations for Python code; > pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram) [3] all do dynamic / runtime PEP-484 type annotation type inference [4] to generate type annotations. https://news.ycombinator.com/item?id=39139198 https://news.ycombinator.com/item?id=39139198 icontract-hypothesis generates tests from icontract DbC Design by Contract type, value, and invariance constraints specified as precondition and postcondition @decorators: https://github.com/mristin/icontract-hypothesis https://github.com/mristin/icontract-hypothesis Nagini and deal-solver attempt to Formally Verify Python code with or without unit tests: https://news.ycombinator.com/item?id=39139198 https://news.ycombinator.com/item?id=39139198 Additional research: "Fuzz target generation using LLMs" (2023) https://google.github.io/oss-fuzz/research/llms/target_generation/ https://google.github.io/oss-fuzz/research/llms/target_gener... https://security.googleblog.com/2023/08/ai-powered-fuzzing-breaking-bug-hunting.html https://security.googleblog.com/2023/08/ai-powered-fuzzing-b... https://hn.algolia.com/?q=AI-Powered+Fuzzing%3A+Breaking+the+Bug+Hunting+Barrier https://hn.algolia.com/?q=AI-Powered+Fuzzing%3A+Breaking+the... OSSF//fuzz-introspector//doc/Features.md: https://github.com/ossf/fuzz-introspector/blob/main/doc/Features.md https://github.com/ossf/fuzz-introspector/blob/main/doc/Feat... https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=Fuzz+target+generation+using+LLMs&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=Fuz... : - "Large Language Models Based Fuzzing Techniques: A Survey" (2024) https://arxiv.org/abs/2402.00350 https://arxiv.org/abs/2402.00350 : > This survey provides a systematic overview of the approaches that fuse LLMs and fuzzing tests for software testing. In this paper, a statistical analysis and discussion of the literature in three areas, namely LLMs, fuzzing test, and fuzzing test generated based on LLMs, are conducted by summarising the state-of-the-art methods up until 2024
DeanMey 3y ago
[flagged]

301 points by mfiguiere 3y ago

> In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. That doesn't seem great?

sampli 3y ago
Considering human tests should already have a high coverage rate, if the 25% that increased coverage were actual good tests, I think it’s a useful tool
- bornfreddy 3y ago
  I was actually surprised they had less than 75% coverage to start with.
  - sanxiyn 3y ago
    25% of test cases increased coverage, not coverage was increased by 25%. For example, if they started with 90% coverage and each test case increased coverage by 0.1% and there were 100 test cases, final coverage was 92.5%.
reissbaker 3y ago
It looks like it only gets committed if passes all of those checks (i.e. no humans have to look at it unless it's actually reliably passing and does increase code coverage). 25% code coverage improvement that passes reliably, for the cost of the electricity required for the GPUs, seems pretty cheap. Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.
- sanxiyn 3y ago
  > 25% code coverage improvement The paper does not report code coverage improvement and it is probably not 25%. The paper does say this: > The median number of lines of code added by a TestGen-LLM test in the test-a-thon was 2.5.
Jtsummers 3y ago
Their abstract doesn't match their actual paper contents. That's unfortunate. Their summary indicates rates in terms of test cases (repeating your quote a bit): > 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication] The actual report talks about test classes, where each class has one or more test cases. > (1) 75% of test classes had at least one new test case that builds correctly. > (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably. > (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target. Those are two very different statements. They even have a footnote acknowledging this: > For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class. But then in their conclusion they misrepresent their findings again, like the abstract: > When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.
Retr0id 3y ago
All three are machine-verifiable, so you can easily filter out the ones that don't work, right?

bbor 3y ago

In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. …is that a good rate? I guess I have to read more and see if the unacceptable ones were silly mistakes like the ones that make us all do code review, or serious ones. I don’t think a human engineer with 25% failure rate would be very helpful, if it’s a certain kind of failure. As part of our overall mission to automate unit test generation for Android code, we have developed an automated test class improver, TestGen-LLM. Is that a good mission? I feel like the TDD people are turning over in their graves, or at least in their beds at home. But again something tells me that they caveat this later

bagels 3y ago
There is a lot of testless code in Facebook, nobody gets PSC points for that.

bilekas 3y ago

I'll admit its interesting, a 12 page paper by Meta employees to promote AI for developers. Even brought out the Sankey diagram. I'm probably wrong but if its published in this way, shouldn't the information be given to reproduce it ? Edit: This is not tinfoil hat, I just don't have the kind of data that meta has to learn from. So, maybe they released something ?

seanmcdirmid 3y ago
It’s an FSE 2024 paper, so I’m guessing the artifacts need theory or formal evaluation.
refulgentis 3y ago
If it's anything like Google, it's way too intimately tied to their infra and monorepo to release
- oars 3y ago
  For these big tech companies, infrastructure is optimized for high scale & reliability over velocity & flexibility.

Roritharr 3y ago

I am currently looking into building something like this for a client with large (several > 3M LoCs) and old (started in 2001) Java Projects with low coverage. Interesting to read how it would work it you already have good coverage (I assume).

tivert 3y ago

> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test. At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.

kromem 3y ago
Keeping them separated would also improve future training.
chii 3y ago
> you've got someone who can judge if it's the system being stupid or if it's the test. but why couldn't this be done even with the llm generated test cases?
- anoopelias 3y ago
  In that case, I think the point is the difference between what is LLM "assisted" tests (like say Copilot) vs. LLM "owned" tests.
- deleted 3y ago
  [deleted]
christkv 3y ago
I mean if I can use it to generate coverage for a method then prune the tests it will still save me hours. Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.
renewiltord 3y ago
Yeah, but that means you can really live Hyrum's Law.
3y ago

TheChaplain 3y ago

In my experience writing tests is generally an outstanding method to determine code quality. If a test is complicated or coverage is hard to achieve, it's likely your tested code needs improvement.

jillesvangurp 3y ago
Testability of code is indeed a good benchmark for code quality. The things that make it hard to test code are also the things that are associated with low quality code. Something with low coupling, high cohesion, and low complexity should be easy to unit tests.
- Jensson 3y ago
  Competitive programming code is extremely easy to test, but many would argue it isn't high quality.

planetjones 3y ago

From reading the PDF it seems that this ‘merely’ generates tests that will repeatedly pass i.e. that are not flaky. The main purpose is to create a regression test suite by having tests that pin the behaviour of existing code. This isn’t a replacement for developer written tests, which one would hope come with the knowledge of what the functional requirement is. Almost 20 years ago the company I worked for trialled AgitarOne - its promise was automagically generating test cases for Java code that help explore its behaviour. But also Agitar could create passing tests more or less automatically, which you could then use as a regression suite. Personally I never liked it, as it just led to too much stuff and it was something management didn’t really understand - to them if the test coverage had gone up then the quality must have too. I wonder how much better the LLM approach FB talk about here is compared to that though… http://www.agitar.com/solutions/products/agitarone.html http://www.agitar.com/solutions/products/agitarone.html

seanmcdirmid 3y ago
A lot of unit tests generated that way will simply be change detectors (fail when code changes) rather than regression tests (fail when bug is re-introduced). Those are pretty big distinctions, I don’t see LML’s getting here until they can ascertain tear correctness without just assuming good tests pass or depending on an oracle (the prompt will have to include behavior expectations somehow).
- planetjones 3y ago
  That’s what I believe Facebook have created here, so you’re right ‘regression’ is a big word - the tests are more likely detecting change e.g. by asserting the existing behaviour of conditionals previously not executed.
- sixstringtheory 3y ago
  This articulates the problem I’m having right now in an interesting way. I’m fine writing unit tests that validate business logic requirements or bug fixes, but writing tests that validate implementations to the point that they reimplement the same logic is a bit much. I want to figure out how to count the number of times a test has had to change with updated requirements vs how many defects they’ve prevented (vs how much wall clock time / compute resources they’ve consumed in running them).

hubraumhugo 3y ago

At a large insurance company I worked for, management set a target of 80% test coverage across our entire codebase. So people started writing stupid unit tests for getters and setters in Java DTOs to reach the goal. Of course devs also weren't allowed to change the coverage measuring rules in Sonar. As a young dev, it taught me that focusing only on KPIs can sometimes drive behaviors that don't align with the intended goals. A few well-thought out E2E test scenarios would probably have had a better impact on the software quality.

awwaiid 3y ago
True. However we might think about this differently if we flipped it to "our standard is 20% completely untested". Uncoverage communicates the value much better.
farhanhubble 3y ago
True of any metric when it becomes the goal in itself.
oneshtein 3y ago
Is it hard to write ONE test case for ALL getters and setters using reflection?
- reactordev 3y ago
  at that point what are you testing? allocation/deallocation? Nothing is actually happening unless you depend on constructors/destructors when you use reflection to gen a mock type or real value type to inject. Or do you mean just assert true that all getters are not null? I agree that unit testing for the sake of KPI's is the wrong approach and unit testing functionality (as a means of documenting it, proving it still works as intended) is far better.
  - oneshtein 3y ago
    > at that point what are you testing? Memory. > Nothing is actually happening unless you depend on constructors/destructors when you use reflection to gen a mock type or real value type to inject. Just get an object from IoC container, then test that getFoo() == getFoo(setFoo(getFoo())) for every Foo with get and set methods, so you will have 80% of coverage for those getters and setters. For read-only properties, just get value and throw it away. > I agree that unit testing for the sake of KPI's is the wrong approach and unit testing functionality (as a means of documenting it, proving it still works as intended) is far better. Unit tests are for proving correctness of logic in a unit of code. Data structures are not logic. IMHO, such trivial parts of program should be ignored by coverage tool, but topic starter said that they cannot change rules. :-/

curtis3389 3y ago

TestGen-LLM is such a strange creation. I can see how it could be used as a first step in a refactoring or rewrite, but the emphasis on code coverage in the paper seems totally brain-broke. I suppose it'd be great if your org is already brain-broke and demanding high coverage, but TestGen-LLM won't make your project's code better in any way, and it'll increase the friction involved in actually implementing improvements. It'd be much more useful to generate edge-case tests that may or may not be passing, but TestGen-LLM relies on compiler errors and failing tests to filter out LLM garbage. The lack of any examples of the generated tests in the paper makes me suspect that they're the same as the rest of the LLM-generated code I have seen: amateurish.

thomasahle 3y ago
I recently had to refractory a project that had no tests what so ever. Having LLMs automatically generate a first draft of tests was very helpful. Even just to understand what the code was supposed to do.

Lio 3y ago

It’s papers like this that will act as the justification of the next round of FAANG lAIoffs. Regardless of how successful this approach is in the long term.

shardullavekar 3y ago

At unlogged.io, for some time - our primary focus was to auto-generate junit tests. The approach didn't take off for a few reasons: 1. A Lot of generated test code that no devs wanted to maintain. 2. The generated tests didn't simulate real-world scenarios. 3. Code coverage was a vanity metric. Devs worked around to reach their goals with scenarios that didn't matter. We are currently working on offering no-code replay tests that simulate all unique production scenarios and developers can replay locally while mocking external dependencies. Disclaimer: I am a founder at unlogged.io

pearjuice 3y ago

So write unit tests automatically, change code later and then regenerate the unit tests? Now the code has a bug but the unit tests pass. I'm already seeing this today with devs using ChatGPT to quickly get the "test boilerplate" over and over.

afro88 3y ago
You still need a human in the loop. I doubt Meta are letting it run blindly. More just automating part of the process and having a human decide what is and isn't committed.

regularfry 3y ago

I want to go the other way. Let me feed acceptance criteria in, have it generate tests that check them, and only then generate code that passes the tests. You can get close to this with Copilot, sometimes, in a fairly limited way, but why do I feel like nobody is focusing on doing it that way round?

oslac 3y ago

Is there still no type theoretic answer to unit testing? Does not the type or the class generally contain all the necessary information to unit test itself, assuming its a unit? That is, we should not have to even write these "theoretically". Just hit "compiler --unit_test <type>"

deleted 3y ago
[deleted]
afro88 3y ago
What you're describing is more or less fuzzing [1], at the unit level. I can't remember the names, but there are tools around that work like this at runtime (ie you define a test that executes functions from the test library that run tests based on input/output types and other user defined constraints). There's almost always more business logic to what a unit should do than it's types though. Depending on the language, the type system can only encode so much of that logic. Consider the opposite: can't the compiler generate implementations from types and interfaces? In most cases, no. LLMs are filling some of that gap though because they can use some surrounding context to return the high probability implementation (completion) from the interface or type definition. [1] https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing

aussieguy1234 3y ago

Not sure about improving. But for Tunnelmole (https://github.com/robbie-cahill/tunnelmole-client https://github.com/robbie-cahill/tunnelmole-client) I have used GPT-4 to generate unit tests just by showing it a TypeScript module and asking it to create tests.

Alifatisk 3y ago

Meta have been publishing fire content recent years

deleted 3y ago

[deleted]

emmender2 3y ago

Happy to see such a huge team collaborating on a project at any company. Perhaps becoz, it involves LLM and LLMs are hot, and everyone wants a piece of it.

tdiff 3y ago

I wonder what would be the cost of maintaining some huge auto-generated corpus of tests in the future. They need to provide some automated way not only to generate cases, but also to update them.

MeteorMarc 3y ago

So, assign to: LLM when lots of tests are broken after the next refactoring!

westurner 3y ago

"Automated Unit Test Improvement using Large Language Models at Meta" (2024) https://arxiv.org/abs/2402.09171 https://arxiv.org/abs/2402.09171 : > This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. [...] We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement. Coverage-guided unit test improvement might [with LLMs] be efficient too. https://github.com/topics/coverage-guided-fuzzing https://github.com/topics/coverage-guided-fuzzing : - e.g. Google/syzkaller is a coverage-guided syscall fuzzer: https://github.com/google/syzkaller https://github.com/google/syzkaller - Gitlab CI supports coverage-guided fuzzing: https://docs.gitlab.com/ee/user/application_security/coverage_fuzzing/ https://docs.gitlab.com/ee/user/application_security/coverag... - oss-fuzz, osv Additional ways to improve tests: Hypothesis and pynguin generate tests from type annotations. There are various tools to generate type annotations for Python code; > pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram) [3] all do dynamic / runtime PEP-484 type annotation type inference [4] to generate type annotations. https://news.ycombinator.com/item?id=39139198 https://news.ycombinator.com/item?id=39139198 icontract-hypothesis generates tests from icontract DbC Design by Contract type, value, and invariance constraints specified as precondition and postcondition @decorators: https://github.com/mristin/icontract-hypothesis https://github.com/mristin/icontract-hypothesis Nagini and deal-solver attempt to Formally Verify Python code with or without unit tests: https://news.ycombinator.com/item?id=39139198 https://news.ycombinator.com/item?id=39139198 Additional research: "Fuzz target generation using LLMs" (2023) https://google.github.io/oss-fuzz/research/llms/target_generation/ https://google.github.io/oss-fuzz/research/llms/target_gener... https://security.googleblog.com/2023/08/ai-powered-fuzzing-breaking-bug-hunting.html https://security.googleblog.com/2023/08/ai-powered-fuzzing-b... https://hn.algolia.com/?q=AI-Powered+Fuzzing%3A+Breaking+the+Bug+Hunting+Barrier https://hn.algolia.com/?q=AI-Powered+Fuzzing%3A+Breaking+the... OSSF//fuzz-introspector//doc/Features.md: https://github.com/ossf/fuzz-introspector/blob/main/doc/Features.md https://github.com/ossf/fuzz-introspector/blob/main/doc/Feat... https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=Fuzz+target+generation+using+LLMs&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C43&q=Fuz... : - "Large Language Models Based Fuzzing Techniques: A Survey" (2024) https://arxiv.org/abs/2402.00350 https://arxiv.org/abs/2402.00350 : > This survey provides a systematic overview of the approaches that fuse LLMs and fuzzing tests for software testing. In this paper, a statistical analysis and discussion of the literature in three areas, namely LLMs, fuzzing test, and fuzzing test generated based on LLMs, are conducted by summarising the state-of-the-art methods up until 2024

DeanMey 3y ago

[flagged]