4 ms·

I would hire copilot to write tests for me, that’s about it. Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it.

I would hire copilot to write tests for me, that’s about it. Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If it’s a usable test, maybe it will catch a bug. If not, then kill it and let it generate a few more. The expectation is entirely different than producing code. Code needs to be correct, secure, performant, and readable. Failure on any of those fronts can be expensive to disastrous. Nobody can reasonably expect a test suite to catch every bug, even if created by the smartest humans. If a copilot-created test does prevent a bug from shipping it provides immediate value. I could see it coming up with some whacky-but-useful test cases that a sane person might not consider. From a training perspective I would think that assertion descriptions contain more consistent lexical value than the average function signature. It seems like the ambitious data scientists, product marketers, and managers fell in love with a revolutionary idea about AI writing code, and neglected to consult the engineers they are trying to ‘augment’.

squidsurfer 5y ago
Many tests require an understanding of the intent of the code, especially the further from a pure unit test they stray. CoPilot has no understanding of anything and I anticipate will be useless in writing the tests that are actually hard to write.
neatze 5y ago
My time is spend mostly not on writing code, but thinking what program has to do, testing (including writing unit tests) and understanding errors, I always tell people half joking programming is not about writing code, but an ability to debug it; understanding requirements, errors and bugs is hard, writing code and fixing bugs is relatively easy, in general. Maybe Copilot 2 will do exactly this; it will generate tests based of half working code, run them and suggest improvements, that would increase productivity by like ~100%, but to me this sounds to good to be true.
wpietri 5y ago
Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother. For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
- cmeacham98 5y ago
  They didn't say they were going to have Copilot write _all_ the tests. Writing tests for cases you can think of and trying Copilot for the extras doesn't seem like that bad of an idea.
  - debaserab2 5y ago
    Sounds like a good way to pile up your test suite with a bunch useless tests.
    - haggy 5y ago
      Can you imagine the CI build time bloat? I'm getting anxiety just thinking about it
- comicjk 5y ago
  It would be awful to write every test using Copilot, but there is potential there for a certain kind of test. If I'm writing an API, I want fresh eyes on it, not just tests written by the person who understands it most (me). For example, a fresh user might try to apply a common pattern that my API breaks. Copilot might be able to act like such a tester. By writing generic tests, it could mimic developers who haven't understood the API before they start using it (most of them).
  - wpietri 5y ago
    If you can find an example of Copilot coming up with a test you wouldn't have thought of, I'd be very interested to see it. Even if that happened, which I am not expecting, I think the need is much more easily solved via means that are simpler and more effective. E.g., a good tester writing up a list of things they test about APIs: https://www.sisense.com/blog/rest-api-testing-strategy-what-exactly-should-you-test/ https://www.sisense.com/blog/rest-api-testing-strategy-what-...
  - barkingcat 5y ago
    copilot as defined would not be "fresh eyes"... it would be "old tired eyes of every code writer who uploaded stuff to github, not knowing if they made one off errors or mistakes in their code"
IncRnd 5y ago
> Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If Copilot can't write the correct code in the first place, you really shouldn't expect a proper test to be written by Copilot. > Code needs to be correct, secure, performant, and readable. Most tests should also have at least three of those attributes. Nobody actually wants their tests to be incorrect, slow, or impossible to understand or modify.
dkersten 5y ago
From the article: "Oh, and they're both wrong." "Both look plausibly correct at a glance" You would end up with tests that look plausibly correct, but test the wrong results.
PufPufPuf 5y ago
On the contrary, it might be interesting writing tests by hand and using the AI to produce code. If the tests are good enough for humans, they should be good enough for AI, given that the AI doesn't try to be actively malicious.

squidsurfer 5y ago

Many tests require an understanding of the intent of the code, especially the further from a pure unit test they stray. CoPilot has no understanding of anything and I anticipate will be useless in writing the tests that are actually hard to write.

neatze 5y ago

My time is spend mostly not on writing code, but thinking what program has to do, testing (including writing unit tests) and understanding errors, I always tell people half joking programming is not about writing code, but an ability to debug it; understanding requirements, errors and bugs is hard, writing code and fixing bugs is relatively easy, in general. Maybe Copilot 2 will do exactly this; it will generate tests based of half working code, run them and suggest improvements, that would increase productivity by like ~100%, but to me this sounds to good to be true.

wpietri 5y ago

Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother. For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/

cmeacham98 5y ago
They didn't say they were going to have Copilot write _all_ the tests. Writing tests for cases you can think of and trying Copilot for the extras doesn't seem like that bad of an idea.
- debaserab2 5y ago
  Sounds like a good way to pile up your test suite with a bunch useless tests.
  - haggy 5y ago
    Can you imagine the CI build time bloat? I'm getting anxiety just thinking about it
comicjk 5y ago
It would be awful to write every test using Copilot, but there is potential there for a certain kind of test. If I'm writing an API, I want fresh eyes on it, not just tests written by the person who understands it most (me). For example, a fresh user might try to apply a common pattern that my API breaks. Copilot might be able to act like such a tester. By writing generic tests, it could mimic developers who haven't understood the API before they start using it (most of them).
- wpietri 5y ago
  If you can find an example of Copilot coming up with a test you wouldn't have thought of, I'd be very interested to see it. Even if that happened, which I am not expecting, I think the need is much more easily solved via means that are simpler and more effective. E.g., a good tester writing up a list of things they test about APIs: https://www.sisense.com/blog/rest-api-testing-strategy-what-exactly-should-you-test/ https://www.sisense.com/blog/rest-api-testing-strategy-what-...
- barkingcat 5y ago
  copilot as defined would not be "fresh eyes"... it would be "old tired eyes of every code writer who uploaded stuff to github, not knowing if they made one off errors or mistakes in their code"

IncRnd 5y ago

> Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If Copilot can't write the correct code in the first place, you really shouldn't expect a proper test to be written by Copilot. > Code needs to be correct, secure, performant, and readable. Most tests should also have at least three of those attributes. Nobody actually wants their tests to be incorrect, slow, or impossible to understand or modify.

dkersten 5y ago

From the article: "Oh, and they're both wrong." "Both look plausibly correct at a glance" You would end up with tests that look plausibly correct, but test the wrong results.

PufPufPuf 5y ago

On the contrary, it might be interesting writing tests by hand and using the AI to produce code. If the tests are good enough for humans, they should be good enough for AI, given that the AI doesn't try to be actively malicious.