6 ms·

I am not an software engineer, but I write code almost daily for personal side projects and automation at work. I write unit and integration tests even for the

I am not an software engineer, but I write code almost daily for personal side projects and automation at work. I write unit and integration tests even for the most toy thing I do (this doesn't come from some kind of fanatic view about testing, but simply because I enjoy writting code and learning/exposing myself to the reasoning and decisions of certain scenarios). I understand the benefits of it, however, I find unit testing specifically tedious and boring as hell. I always wondered if it could be done by some framework that relies on something like Github Copilot: Install the framework, run it pointing to your function/methods file(s), get a file with your tests that you can perhaps adjust here and there and then integrate it in your CI. I understand the challenge due to, perhaps in some cases, the implicit difficulty of coming up with relevant tests for certain parts of your business logic but, do you think it will be possible some day, Am I saying something stupid?

danenania 4y ago
Not at all stupid. I'm sure people are already using ChatGPT to generate unit tests. There are limits, of course, given that (for now) it doesn't have the full context of your code, but it's definitely capable of generating tests for pure functions that are named well and don't require too much outside context. Some projects have tons of these.
- seanmcdirmid 4y ago
  What about chatGPT’s unit tests? Does it even make sense to unit test code that generates a language model?
  - noduerme 4y ago
    code with output that can only be evaluated artistically can't be called right or wrong anyway. As long as it throws no errors, it's a valid language model. I can only imagine how many nulls and undefineds and Infinitys they just throw out and shove under the rug. That's actually the kind of code where not knowing how it works is okay. Encouraged, even.
- danbulant 4y ago
  Github copilot is pretty good at writing tests
- persedes 4y ago
  chatGPT works surprisingly well for python (and am sure other popular languages too). I can dump a code bit and tell it to write the tests + fixtures for me. The tests it wrote actually showed it "understood" (or recognized the pattern of) the code too. For example part of the code renamed some columns for a data frame that was loaded from a csv. The fixture it created for the csv had the correct column names the code was renaming. Unless you're doing TDD or chatGPT is too busy, there's almost no excuse now to not write some unit tests ;)
- davps 4y ago
  Yes, I use it in that way. And if ChatGPT didn't generated the code with pure functions (usually it didn't), you can explicitly ask to generate the code with pure functions. Then ask to generate the tests. Usually I get good tests from ChatGPT when I approach it as an iterative process, requesting multiple improvements to the generated test based on what it gives to me. Note that it doesn't replace the skillsets you need to know to write good test coverage. For example, you can ask to generate integration tests instead of unit tests in case it need context. Providing details on how the testing code should be generated really helps. Also, asking to refactoring the code in preparation to make it testeable, for example, or requesting to convert some functions to actual pure functions, or requesting to refactor a piece of code it generated to a separate function. Then you ask to generate tests for normal and also for boundary conditions. The more specific you get, the chance of getting a good and extensive tests from it is much higher. Having the tests and code generated by ChatGPT really helps to catch the subtle bugs it usually generates on the generated code (fixing it manually), usually I get test coverage that proof the robustness I needed for production code. This approach still needs manual fine tuning of the generated code, I think ChatGPT still struggle to get the context right but in general, when it makes sense to use it, I'm more productive writing tests in this way than manually.
- scruple 4y ago
  I used my Copilot trial to explore writing tests more than writing code. I found that it actually worked well, especially for data and boilerplate heavy tests, like table-driven tests in Go. It was saving me quite literally dozens, if not a hundred or more, of keystrokes. I wrote Go for side projects these days, not for work, so I don't pay for the license, but if I was writing Go again professionally I'd pay for the license for this alone. In my day job, I write Ruby, and it didn't impress me much when I used it with Rspec. I'd says it was saving me maybe a dozen or so keystrokes in total to write a new spec.
- fomine3 4y ago
  I wonder which is good at for GPT finally: generate impl from test or vice versa. I think former, as a noob.
noduerme 4y ago
This may sound weird, but I've written code professionally for 25+ years and I have never written a unit test, I'm not even sure what a "unit test" is or would look like in the context of what I write. My job is to first consider a dozen ways a new feature might be used; then consider how it could be misused; then write a backend portion and UI for it, testing it for usability and potential for error along the way, using everything I've already thought could go wrong; then monitor it once it's deployed to see if anything actually does go wrong. After actually testing it through everything I can think of by hand, function by function, line by line as I write it, it seems totally stupid to write code to test my own code. Not to sound like a shit, but that's what end users are for ;)
- spoils19 4y ago
  That's awesome to hear! I myself do something similar with my own code, but then again, I also trust myself more than any computer ;) About how long would you say that process takes you? Do you repeat it every time a minor change needs to happen to your code? If not, how do you verify the impact of your changes?
  - noduerme 4y ago
    Rough answer, I spend about 50% of my time just thinking about pitfalls before I actually write the code, 10% of the time writing it, and the other 40% testing it to my satisfaction (doing every stupid thing I can think of someone doing to break it) before I deploy. Usually though, in practice, this looks like tracing/logging several times in the middle of each function to make sure I'm getting it right. Where unit tests are supposedly most handy is when unexpected types are passed to functions, leading to unexpected errors. The majority of proofing against that can be done by strict typing and good code documentation. Ultimately the best test is when you ask yourself "wait, what if someone sent this" and you try it to see if you can break your own code. That's just a spidey sense in the back of your head. If you didn't have the specific doubt to begin with, I don't know how you could write a unit test to disprove it anyway. [edit] just a note to any new coders reading this: Spend the time to always read, read, read everything about how to harden the code you write for security purposes. Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.

danenania 4y ago

Not at all stupid. I'm sure people are already using ChatGPT to generate unit tests. There are limits, of course, given that (for now) it doesn't have the full context of your code, but it's definitely capable of generating tests for pure functions that are named well and don't require too much outside context. Some projects have tons of these.

seanmcdirmid 4y ago
What about chatGPT’s unit tests? Does it even make sense to unit test code that generates a language model?
- noduerme 4y ago
  code with output that can only be evaluated artistically can't be called right or wrong anyway. As long as it throws no errors, it's a valid language model. I can only imagine how many nulls and undefineds and Infinitys they just throw out and shove under the rug. That's actually the kind of code where not knowing how it works is okay. Encouraged, even.
danbulant 4y ago
Github copilot is pretty good at writing tests
persedes 4y ago
chatGPT works surprisingly well for python (and am sure other popular languages too). I can dump a code bit and tell it to write the tests + fixtures for me. The tests it wrote actually showed it "understood" (or recognized the pattern of) the code too. For example part of the code renamed some columns for a data frame that was loaded from a csv. The fixture it created for the csv had the correct column names the code was renaming. Unless you're doing TDD or chatGPT is too busy, there's almost no excuse now to not write some unit tests ;)
davps 4y ago
Yes, I use it in that way. And if ChatGPT didn't generated the code with pure functions (usually it didn't), you can explicitly ask to generate the code with pure functions. Then ask to generate the tests. Usually I get good tests from ChatGPT when I approach it as an iterative process, requesting multiple improvements to the generated test based on what it gives to me. Note that it doesn't replace the skillsets you need to know to write good test coverage. For example, you can ask to generate integration tests instead of unit tests in case it need context. Providing details on how the testing code should be generated really helps. Also, asking to refactoring the code in preparation to make it testeable, for example, or requesting to convert some functions to actual pure functions, or requesting to refactor a piece of code it generated to a separate function. Then you ask to generate tests for normal and also for boundary conditions. The more specific you get, the chance of getting a good and extensive tests from it is much higher. Having the tests and code generated by ChatGPT really helps to catch the subtle bugs it usually generates on the generated code (fixing it manually), usually I get test coverage that proof the robustness I needed for production code. This approach still needs manual fine tuning of the generated code, I think ChatGPT still struggle to get the context right but in general, when it makes sense to use it, I'm more productive writing tests in this way than manually.
scruple 4y ago
I used my Copilot trial to explore writing tests more than writing code. I found that it actually worked well, especially for data and boilerplate heavy tests, like table-driven tests in Go. It was saving me quite literally dozens, if not a hundred or more, of keystrokes. I wrote Go for side projects these days, not for work, so I don't pay for the license, but if I was writing Go again professionally I'd pay for the license for this alone. In my day job, I write Ruby, and it didn't impress me much when I used it with Rspec. I'd says it was saving me maybe a dozen or so keystrokes in total to write a new spec.
fomine3 4y ago
I wonder which is good at for GPT finally: generate impl from test or vice versa. I think former, as a noob.

noduerme 4y ago

This may sound weird, but I've written code professionally for 25+ years and I have never written a unit test, I'm not even sure what a "unit test" is or would look like in the context of what I write. My job is to first consider a dozen ways a new feature might be used; then consider how it could be misused; then write a backend portion and UI for it, testing it for usability and potential for error along the way, using everything I've already thought could go wrong; then monitor it once it's deployed to see if anything actually does go wrong. After actually testing it through everything I can think of by hand, function by function, line by line as I write it, it seems totally stupid to write code to test my own code. Not to sound like a shit, but that's what end users are for ;)

spoils19 4y ago
That's awesome to hear! I myself do something similar with my own code, but then again, I also trust myself more than any computer ;) About how long would you say that process takes you? Do you repeat it every time a minor change needs to happen to your code? If not, how do you verify the impact of your changes?
- noduerme 4y ago
  Rough answer, I spend about 50% of my time just thinking about pitfalls before I actually write the code, 10% of the time writing it, and the other 40% testing it to my satisfaction (doing every stupid thing I can think of someone doing to break it) before I deploy. Usually though, in practice, this looks like tracing/logging several times in the middle of each function to make sure I'm getting it right. Where unit tests are supposedly most handy is when unexpected types are passed to functions, leading to unexpected errors. The majority of proofing against that can be done by strict typing and good code documentation. Ultimately the best test is when you ask yourself "wait, what if someone sent this" and you try it to see if you can break your own code. That's just a spidey sense in the back of your head. If you didn't have the specific doubt to begin with, I don't know how you could write a unit test to disprove it anyway. [edit] just a note to any new coders reading this: Spend the time to always read, read, read everything about how to harden the code you write for security purposes. Unit testing will not save you from SQL injection or XSS attacks, so study those and bulletproof your work against them first before you worry about mathematical proof that your database call never results in an error under some odd condition.