4 ms·

Everyone by now should be writing unit tests using ChatGPT4. I paste in functions / classes I want to write unit tests for. Paste in a sample unit test, and it

by LASR 3y ago

Everyone by now should be writing unit tests using ChatGPT4. I paste in functions / classes I want to write unit tests for. Paste in a sample unit test, and it does a solid job of writing tests for it in the same manner as in the sample. For unit tests, you don't even need the multi-step coverage optimization in this article. You just manually inspect, adjust it etc.

swatcoder 3y ago
If I have a choice between a clever, practiced antagonist to plan and write my tests and an automated system that can fit common testing patterns to my code... I'm always going to get more robust results from the former. But yeah, if you're just working solo on basic stuff and need to protect against off-by-one errors and accidental mutations in later refactoring, it's a great tool. You'd never write duly antagonistic tests for your own code anyway.
interroboink 3y ago
What about people who don't trust sending their code to a 3rd party for processing? (edit: I didn't downvote you, but I do think your claim is over-broad)
- deleted 3y ago
  [deleted]
- biot 3y ago
  I suspect a lot of people overestimate how special their code is. Also, if your code exists in a private repo on GitHub, then you're already trusting the same third party when using GitHub Copilot.
  - dns_snek 3y ago
    This isn't just about IP. In the not so distant future we'll find out about 3 letter agencies using companies like OpenAI to deploy corporate backdoors with ease.
- ric2b 3y ago
  I think those people overestimate how useful it is to get a bunch of random code snippets from some company, that may or may not even reach production unchanged.
  - dns_snek 3y ago
    I think you're underestimating the power such third parties are going to have in the future. There's a 100% likelihood that OpenAI and other LLMs providers are going to cooperate with intelligence agencies looking to deploy their backdoors to businesses across the world. It's the perfect supply chain attack vector with complete deniability.
petesergeant 3y ago
Doesn’t that just bake in any bugs in the original implementation?
- romwell 3y ago
  Of course not! The Power of AI™ can figure out the true intent of the code by looking at the initial (and, potentially, buggy) implementation, and help the programmer by generating edge test cases where the code doesn't produce correct results. The programmer will easily tell those test cases from the ones where the AI did a mistake and generated a flawed test case because the AI doesn't make such silly mistakes; clearly, it's the programmer's code that needs to be corrected. In fact, the AI would do a better job at that, too, which clearly speeds up the development cycle. The correct way to use the tool is to let the AI both generate the test cases and modify the code so that it would pass the tests it generates. After all, if the AI can't figure out what you wanted to do in the first place — how can you? Of course, there's more to it. Whichever problems you run into can be surely attributed to writing the code in an AI-unfriendly way. In the past, we had the adage that the code is read more times than it's written. This is still true, but we need to abandon the old habit habit of having a human reader in mind. Just like you rearrange your furniture to make your house more accessible to the robot vacuum cleaner, you need to write the code with the AI in mind. When you write a Google query or a prompt for ChatGPT (effectively the same thing anyway), you don't write it like you'd talk to a person. You're going to have to write code the same way to be truly effective, and think a little bit like AI to get the most use out of it. That might sound like a lot of work to get code that does what you want. But of course, that's not the case. Just use the AI for this.
- samatman 3y ago
  Oddly enough it doesn't. The secret sauce is: the robot doesn't run the tests. It digests the code and writes tests for it. So it doesn't know if they'll pass or not. And sure enough, some of them don't, because the code was buggy. I've seen some awful code come out of ChatGPT, but never a bad test. Good tests are short, which makes them hard to screw up. It has a reasonable grasp on what an edge case is as well.

by LASR 3y ago

swatcoder 3y ago

If I have a choice between a clever, practiced antagonist to plan and write my tests and an automated system that can fit common testing patterns to my code... I'm always going to get more robust results from the former. But yeah, if you're just working solo on basic stuff and need to protect against off-by-one errors and accidental mutations in later refactoring, it's a great tool. You'd never write duly antagonistic tests for your own code anyway.

interroboink 3y ago

What about people who don't trust sending their code to a 3rd party for processing? (edit: I didn't downvote you, but I do think your claim is over-broad)

deleted 3y ago
[deleted]
biot 3y ago
I suspect a lot of people overestimate how special their code is. Also, if your code exists in a private repo on GitHub, then you're already trusting the same third party when using GitHub Copilot.
- dns_snek 3y ago
  This isn't just about IP. In the not so distant future we'll find out about 3 letter agencies using companies like OpenAI to deploy corporate backdoors with ease.
ric2b 3y ago
I think those people overestimate how useful it is to get a bunch of random code snippets from some company, that may or may not even reach production unchanged.
- dns_snek 3y ago
  I think you're underestimating the power such third parties are going to have in the future. There's a 100% likelihood that OpenAI and other LLMs providers are going to cooperate with intelligence agencies looking to deploy their backdoors to businesses across the world. It's the perfect supply chain attack vector with complete deniability.

petesergeant 3y ago

Doesn’t that just bake in any bugs in the original implementation?

romwell 3y ago
Of course not! The Power of AI™ can figure out the true intent of the code by looking at the initial (and, potentially, buggy) implementation, and help the programmer by generating edge test cases where the code doesn't produce correct results. The programmer will easily tell those test cases from the ones where the AI did a mistake and generated a flawed test case because the AI doesn't make such silly mistakes; clearly, it's the programmer's code that needs to be corrected. In fact, the AI would do a better job at that, too, which clearly speeds up the development cycle. The correct way to use the tool is to let the AI both generate the test cases and modify the code so that it would pass the tests it generates. After all, if the AI can't figure out what you wanted to do in the first place — how can you? Of course, there's more to it. Whichever problems you run into can be surely attributed to writing the code in an AI-unfriendly way. In the past, we had the adage that the code is read more times than it's written. This is still true, but we need to abandon the old habit habit of having a human reader in mind. Just like you rearrange your furniture to make your house more accessible to the robot vacuum cleaner, you need to write the code with the AI in mind. When you write a Google query or a prompt for ChatGPT (effectively the same thing anyway), you don't write it like you'd talk to a person. You're going to have to write code the same way to be truly effective, and think a little bit like AI to get the most use out of it. That might sound like a lot of work to get code that does what you want. But of course, that's not the case. Just use the AI for this.
samatman 3y ago
Oddly enough it doesn't. The secret sauce is: the robot doesn't run the tests. It digests the code and writes tests for it. So it doesn't know if they'll pass or not. And sure enough, some of them don't, because the code was buggy. I've seen some awful code come out of ChatGPT, but never a bad test. Good tests are short, which makes them hard to screw up. It has a reasonable grasp on what an edge case is as well.