6 ms·

Empirical Study of Test Generation with LLM's

40 points by nickpsecurity 2y ago

jitl 2y ago
My cursor workflow for getting tests is to make the test file, import the code under test, and then type cmd-k, “unit tests for <class>”, enter. Add additional cmd-k to prompt for method tests and cases as needed. Pretty basic, would adding more shenanigans get me better results? I try to write doc comments for methods with contracts and it seems cursor/claude does a good job reading and testing those.
- peterldowns 2y ago
  What value are the tests giving you? Asking truly naively, I haven't tried generating tests for code before, and I'm skeptical that it would be useful for me.
  - jitl 2y ago
    The tests provide the same value as usual: early detection of coding mistakes, (somewhat) format specification of contracts, assurance the code does what it's supposed to do and continues to do what it's supposed to do as the codebase evolves over time.
    - peterldowns 2y ago
      Could you share an example test (if it doesn't violate your privacy)? Your answer is too vague for me to understand. I can't square my experience of LLMs (generate braindead code for well-documented situations) with the kinds of tests I write (enforcing domain-specific business logic during execution of specific scenarios) about which LLMs know nothing (not the domain-specific logic, not the scenarios, not my test helpers and mock implementations.)
      - jitl 2y ago
        This is the open-source example I have: https://github.com/justjake/eslint-seatbelt/blob/main/src/SeatbeltFile.test.ts https://github.com/justjake/eslint-seatbelt/blob/main/src/Se... Maybe you can declare this is braindead code for well-documented situations? not sure. My rough draft implementation of this class had bugs, I asked Cursor to write tests - prompt was something like "can you set up tests for SeatbeltFile.ts using node:test framework", it added "test" script to package.json and wrote 90% of the test file, then I fixed the bugs, added another case myself, committed the tests.
        peterldowns 2y ago
        Interesting, thank you very much for sharing. I wouldn't have expected an LLM to write those tests. This makes me want to try again for some of the work I've been doing. Thank you!
- arnvald 2y ago
  I tried that with Cody (from Sourcegraph) using probably o1 model and I struggled. I had a long function with a number of conditions and early returns. At first Cody generated a single test case, then I asked if to consider all possible scenarios in that function (around 10), it generated 5 cases, 3 of which were correct and 2 scenarios were made up (it tried to use nonexistent enum values). In the end I used it to generate some mocks and a few minor things but then I copy pasted the test myself a few times and changed the values manually. Did it save me some time in the end? Possibly, but also it caused a lot of frustration. I still hope it gets better because generating tests should be a great use case for LLMs
fovc 2y ago
I still don’t understand how people are getting value out of AI coders. I’ve tried really hard and the commits produced are just a step up from garbage. Writing code from scratch is generally decent. But after a few rounds of edits the assistant just starts piling in conditionals into existing functions until it’s a rats nest 4 layers deep and 100+ lines long. The other day it got into a loop trying to resolve a type error, where it would make a change, then revert it, then make it again ETA: Sorry forgot about the relevancy in my rant! The one area where I’ve found the AIs helpful is enumerating and then creating test cases
- godelski 2y ago
  I have a simple answer for you: most people write garbage code[00]. I know this because I write garbage code but usually less garbage. A little background... I got my undergrad in physics (where I fell in love with math), spent some years working, and got really interested in coding and especially ML (especially the math). So I went to grad school. Unsurprising I had major impostor syndrome being surrounded by CS people[0], so I spent a huge amount of time trying to fill in the gap. The real problem was that many of my friends were PL people, so they're math heavy and great programmers. But after teaching a bunch of high level CS classes I realized I wasn't behind. After having to fix a lot of autograders made by my peers, I didn't feel so behind. When my lab grew and I got to meet a lot more ML people, I felt ahead, and confused. I realized the problem: I was trying to be a physicist in CS. Trying to understand things at very fundamental levels and using that to build up, not feeling like I "knew" a topic until I knew that chain. I realized people were just saying they knew at a different threshold. Back to ML: Working and researching in ML I've noticed one common flaw. People are ignoring details. I thought HuggingFace would be "my savior" where people would see that their generation outputs weren't nearly the quality you see in papers. But this didn't happen. We cherry picked results and ignored failures. It feels like writing proofs but people only look at the last line (I'd argue this is analogous to code! It's about so much more than the output! The steps are the thing that matters). So there's two camps of ML people now: the hype people and "the skeptics" (interestingly there's a large population of people with physics and math backgrounds here). I put the latter in quotes because we're not trying to stop ML progress. I'd argue we're trying to make it! The argument is we need to recognize flaws so we know what needs to be fixed. This is why Francois Chollet made the claim that GPT has delayed progress towards AGI. Because we are doing the same thing that caused the last AI winter: putting all our eggs in one basket. We've made it hard to pursue other ideas and models because to get published you need to beat benchmarks (good luck doing so out of the gate and without thousands of GPUs). Because we don't look at the limitations in benchmarks. Because we don't even check for God damn information spoilage anymore. Even HumanEval is littered with spoilage, and obviously so... There's tons of uses for LLMs and ML systems. My "rage" (as with many others) is more about over promising. Because we know if you don't fulfill those promises quickly, sentiment turns against you and funding quickly goes away. Just look at how even HN went from extremely positive on AI to a similar dichotomy (though the "skeptics" are probably more skeptical than researchers.[1]). Is playing with fire. Prometheus gave it to man to enlighten themselves but they also burned themselves quite frequently. The answer is: you evaluate in more detail than others. [00] of course it is. LLMs replicate average human code. They're optimizers. They optimize fitting data, not fitting optimal symbolic manipulation. If everyone was far better at code, LLMs would be too. That's how they work [0] boy, us physicists have big egos but CS people give us a run for the money [1] I have no doubt that AGI can be created. I have no doubt we humans can make it. But I highly doubt LLMs will get us there and we need to look in other directions. I'm not saying we shouldn't stop perusing LLMs, I'm saying don't stop the other research from happening. It's not a zero sum game. Most things in the real world are not (but for some god damn reason we always think it is)
hitchstory 2y ago
Using LLMs to generate tests is bizarre. If anything, humans should write the tests and the LLMs should make it pass. Tests lose value the further away from the spec they drift. Writing a test after the code causes drift. Writing with an LLM causes even more drift. It ends up being a ritual to appease the unit testing gods rather than an investment that is supposed to pay dividends.
- Etheryte 2y ago
  Agreed, tacking on an arbitrary set of tests after the fact is a great way to pour glue over the code, nothing more.
- weitendorf 2y ago
  The paper is pretty vacuous IMO but there are at least a few reasons I think LLM testing is pretty nice: * It’s actually easier to do TDD or black box testing with LLMs. Yes, the lazy approach is to feed it a function implementation and tell it to make a unit test. But you can instead feed it the function definition and a description of its behavior (which may be what you used to generate the implementation too!) and have it generate a unit test with no visibility to the spec. * Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable (eg Go table test cases) and LLMs can knock that out super quickly. * Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused. It’s a nice step towards later having actually-good tests, and some projects are so poorly tested and plagued with basic breakages/bugs that it’s worth slowing down feature development to keep things stable. Personally I hate when people try to automate this stuff though, because it does trend towards junk. I find it better to treat writing tests with LLMs tactically, basically the same way you use them to write code.
  - hitchstory 2y ago
    >Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable When people use LLMs to write code and they find if helpful, invariably it is because they are spewing boilerplate. If you dont systematically eliminate boilerplate the codebase eventually turns into an unmaintainable mess. >Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused. Really? Id consider this an antipattern. >I find it better to treat writing tests with LLMs tactically I find the prospect of using them to write production code / tests pretty depressing. The best thing that can be said is that they will create lots of jobs with the mess they make.
nickpsecurity 2y ago
Here’s the EvoSuite that they refer to: https://www.evosuite.org/evosuite/ https://www.evosuite.org/evosuite/
bediger4000 2y ago
Doesn't seem to compare to human generated tests. I'm guessing that comparison is mine too favorable.
- nickpsecurity 2y ago
  I think humans are probably still better (a) *when they write tests and (b) when they know how to. Neither is true for a lot of programmers but might happen if automated cheaply. I also think traditional tools are better for this. The test, generation methods include path-based, combinatorial, concolic, and adaptive fuzzing. Fire and forget tools that do each one, suppressing duplicates, would be helpful. What would be easier to train developers on are contracts (or annotations). Then, add test generation either from those or aided by static analysis that leverages them. Then, software that analyzes the code, annotates what it can, and then asks the user to fill in the holes. The LLM’s could turn their answers into formal contracts or specs. That’s how I see it happening. Maybe add patch generation into that, too. At least two tools that predate LLM’s already find errors and suggest fixes.

40 points by nickpsecurity 2y ago

jitl 2y ago

My cursor workflow for getting tests is to make the test file, import the code under test, and then type cmd-k, “unit tests for <class>”, enter. Add additional cmd-k to prompt for method tests and cases as needed. Pretty basic, would adding more shenanigans get me better results? I try to write doc comments for methods with contracts and it seems cursor/claude does a good job reading and testing those.

peterldowns 2y ago
What value are the tests giving you? Asking truly naively, I haven't tried generating tests for code before, and I'm skeptical that it would be useful for me.
- jitl 2y ago
  The tests provide the same value as usual: early detection of coding mistakes, (somewhat) format specification of contracts, assurance the code does what it's supposed to do and continues to do what it's supposed to do as the codebase evolves over time.
  - peterldowns 2y ago
    Could you share an example test (if it doesn't violate your privacy)? Your answer is too vague for me to understand. I can't square my experience of LLMs (generate braindead code for well-documented situations) with the kinds of tests I write (enforcing domain-specific business logic during execution of specific scenarios) about which LLMs know nothing (not the domain-specific logic, not the scenarios, not my test helpers and mock implementations.)
    - jitl 2y ago
      This is the open-source example I have: https://github.com/justjake/eslint-seatbelt/blob/main/src/SeatbeltFile.test.ts https://github.com/justjake/eslint-seatbelt/blob/main/src/Se... Maybe you can declare this is braindead code for well-documented situations? not sure. My rough draft implementation of this class had bugs, I asked Cursor to write tests - prompt was something like "can you set up tests for SeatbeltFile.ts using node:test framework", it added "test" script to package.json and wrote 90% of the test file, then I fixed the bugs, added another case myself, committed the tests.
      - peterldowns 2y ago
        Interesting, thank you very much for sharing. I wouldn't have expected an LLM to write those tests. This makes me want to try again for some of the work I've been doing. Thank you!
arnvald 2y ago
I tried that with Cody (from Sourcegraph) using probably o1 model and I struggled. I had a long function with a number of conditions and early returns. At first Cody generated a single test case, then I asked if to consider all possible scenarios in that function (around 10), it generated 5 cases, 3 of which were correct and 2 scenarios were made up (it tried to use nonexistent enum values). In the end I used it to generate some mocks and a few minor things but then I copy pasted the test myself a few times and changed the values manually. Did it save me some time in the end? Possibly, but also it caused a lot of frustration. I still hope it gets better because generating tests should be a great use case for LLMs

fovc 2y ago

I still don’t understand how people are getting value out of AI coders. I’ve tried really hard and the commits produced are just a step up from garbage. Writing code from scratch is generally decent. But after a few rounds of edits the assistant just starts piling in conditionals into existing functions until it’s a rats nest 4 layers deep and 100+ lines long. The other day it got into a loop trying to resolve a type error, where it would make a change, then revert it, then make it again ETA: Sorry forgot about the relevancy in my rant! The one area where I’ve found the AIs helpful is enumerating and then creating test cases

godelski 2y ago
I have a simple answer for you: most people write garbage code[00]. I know this because I write garbage code but usually less garbage. A little background... I got my undergrad in physics (where I fell in love with math), spent some years working, and got really interested in coding and especially ML (especially the math). So I went to grad school. Unsurprising I had major impostor syndrome being surrounded by CS people[0], so I spent a huge amount of time trying to fill in the gap. The real problem was that many of my friends were PL people, so they're math heavy and great programmers. But after teaching a bunch of high level CS classes I realized I wasn't behind. After having to fix a lot of autograders made by my peers, I didn't feel so behind. When my lab grew and I got to meet a lot more ML people, I felt ahead, and confused. I realized the problem: I was trying to be a physicist in CS. Trying to understand things at very fundamental levels and using that to build up, not feeling like I "knew" a topic until I knew that chain. I realized people were just saying they knew at a different threshold. Back to ML: Working and researching in ML I've noticed one common flaw. People are ignoring details. I thought HuggingFace would be "my savior" where people would see that their generation outputs weren't nearly the quality you see in papers. But this didn't happen. We cherry picked results and ignored failures. It feels like writing proofs but people only look at the last line (I'd argue this is analogous to code! It's about so much more than the output! The steps are the thing that matters). So there's two camps of ML people now: the hype people and "the skeptics" (interestingly there's a large population of people with physics and math backgrounds here). I put the latter in quotes because we're not trying to stop ML progress. I'd argue we're trying to make it! The argument is we need to recognize flaws so we know what needs to be fixed. This is why Francois Chollet made the claim that GPT has delayed progress towards AGI. Because we are doing the same thing that caused the last AI winter: putting all our eggs in one basket. We've made it hard to pursue other ideas and models because to get published you need to beat benchmarks (good luck doing so out of the gate and without thousands of GPUs). Because we don't look at the limitations in benchmarks. Because we don't even check for God damn information spoilage anymore. Even HumanEval is littered with spoilage, and obviously so... There's tons of uses for LLMs and ML systems. My "rage" (as with many others) is more about over promising. Because we know if you don't fulfill those promises quickly, sentiment turns against you and funding quickly goes away. Just look at how even HN went from extremely positive on AI to a similar dichotomy (though the "skeptics" are probably more skeptical than researchers.[1]). Is playing with fire. Prometheus gave it to man to enlighten themselves but they also burned themselves quite frequently. The answer is: you evaluate in more detail than others. [00] of course it is. LLMs replicate average human code. They're optimizers. They optimize fitting data, not fitting optimal symbolic manipulation. If everyone was far better at code, LLMs would be too. That's how they work [0] boy, us physicists have big egos but CS people give us a run for the money [1] I have no doubt that AGI can be created. I have no doubt we humans can make it. But I highly doubt LLMs will get us there and we need to look in other directions. I'm not saying we shouldn't stop perusing LLMs, I'm saying don't stop the other research from happening. It's not a zero sum game. Most things in the real world are not (but for some god damn reason we always think it is)

hitchstory 2y ago

Using LLMs to generate tests is bizarre. If anything, humans should write the tests and the LLMs should make it pass. Tests lose value the further away from the spec they drift. Writing a test after the code causes drift. Writing with an LLM causes even more drift. It ends up being a ritual to appease the unit testing gods rather than an investment that is supposed to pay dividends.

Etheryte 2y ago
Agreed, tacking on an arbitrary set of tests after the fact is a great way to pour glue over the code, nothing more.
weitendorf 2y ago
The paper is pretty vacuous IMO but there are at least a few reasons I think LLM testing is pretty nice: * It’s actually easier to do TDD or black box testing with LLMs. Yes, the lazy approach is to feed it a function implementation and tell it to make a unit test. But you can instead feed it the function definition and a description of its behavior (which may be what you used to generate the implementation too!) and have it generate a unit test with no visibility to the spec. * Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable (eg Go table test cases) and LLMs can knock that out super quickly. * Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused. It’s a nice step towards later having actually-good tests, and some projects are so poorly tested and plagued with basic breakages/bugs that it’s worth slowing down feature development to keep things stable. Personally I hate when people try to automate this stuff though, because it does trend towards junk. I find it better to treat writing tests with LLMs tactically, basically the same way you use them to write code.
- hitchstory 2y ago
  >Unit tests tend to have a lot of boilerplate sometimes, often not copy-pastable When people use LLMs to write code and they find if helpful, invariably it is because they are spewing boilerplate. If you dont systematically eliminate boilerplate the codebase eventually turns into an unmaintainable mess. >Sometimes you do actually want to add a ton of unit tests even if they’re a little too implementation-focused. Really? Id consider this an antipattern. >I find it better to treat writing tests with LLMs tactically I find the prospect of using them to write production code / tests pretty depressing. The best thing that can be said is that they will create lots of jobs with the mess they make.

nickpsecurity 2y ago

Here’s the EvoSuite that they refer to: https://www.evosuite.org/evosuite/ https://www.evosuite.org/evosuite/

bediger4000 2y ago

Doesn't seem to compare to human generated tests. I'm guessing that comparison is mine too favorable.

nickpsecurity 2y ago
I think humans are probably still better (a) *when they write tests and (b) when they know how to. Neither is true for a lot of programmers but might happen if automated cheaply. I also think traditional tools are better for this. The test, generation methods include path-based, combinatorial, concolic, and adaptive fuzzing. Fire and forget tools that do each one, suppressing duplicates, would be helpful. What would be easier to train developers on are contracts (or annotations). Then, add test generation either from those or aided by static analysis that leverages them. Then, software that analyzes the code, annotates what it can, and then asks the user to fill in the holes. The LLM’s could turn their answers into formal contracts or specs. That’s how I see it happening. Maybe add patch generation into that, too. At least two tools that predate LLM’s already find errors and suggest fixes.