5 ms·

Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the mean

Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother. For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/

cmeacham98 5y ago
They didn't say they were going to have Copilot write _all_ the tests. Writing tests for cases you can think of and trying Copilot for the extras doesn't seem like that bad of an idea.
- debaserab2 5y ago
  Sounds like a good way to pile up your test suite with a bunch useless tests.
  - haggy 5y ago
    Can you imagine the CI build time bloat? I'm getting anxiety just thinking about it
comicjk 5y ago
It would be awful to write every test using Copilot, but there is potential there for a certain kind of test. If I'm writing an API, I want fresh eyes on it, not just tests written by the person who understands it most (me). For example, a fresh user might try to apply a common pattern that my API breaks. Copilot might be able to act like such a tester. By writing generic tests, it could mimic developers who haven't understood the API before they start using it (most of them).
- wpietri 5y ago
  If you can find an example of Copilot coming up with a test you wouldn't have thought of, I'd be very interested to see it. Even if that happened, which I am not expecting, I think the need is much more easily solved via means that are simpler and more effective. E.g., a good tester writing up a list of things they test about APIs: https://www.sisense.com/blog/rest-api-testing-strategy-what-exactly-should-you-test/ https://www.sisense.com/blog/rest-api-testing-strategy-what-...
- barkingcat 5y ago
  copilot as defined would not be "fresh eyes"... it would be "old tired eyes of every code writer who uploaded stuff to github, not knowing if they made one off errors or mistakes in their code"
  - comicjk 5y ago
    I mean fresh eyes with respect to my new API. Having seen a lot of other code is a benefit. I expect most tests that Copilot writes to fail, but I would hope some would fail in interesting ways. For example, off-by-one errors might encourage me to document my indexing convention, or to use a generator rather than indexing.
- ska 5y ago
  > I want fresh eyes on it Crucially, that's not what copilot is.
developer2 5y ago
> one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. I feel like you just described every developer/codebase where mock testing is stupidly enforced. Where every single unit test mocks every single indirect object. 98% of the testing code is just exhaustive setup and teardown of objects not being tested by each test, and then a bunch of conditional checks to ensure that every deeper/indirect method is being called exactly the right number of times with exactly the right arguments and returning exactly the right value. Almost all of the test code is just hacking mock objects. The actual purpose of each test is buried so deep that it's impossible to even understand the business logic being applied. I hate evangelical "mock testers" with a passion.
- wpietri 5y ago
  Yes. I have been doing unit testing for a long time, and I think this is a clear antipattern. It's cargo-cult testing, not actually a serious effort to improve quality and developer productivity.
- spullara 5y ago
  These are absolutely the worst tests I have ever seen. They make iterating on the implementation almost impossible. Why people do this I will never understand.
- lexandstuff 5y ago
  Preach. This is what chasing 100% test coverage looks like.
nullc 5y ago
It would be a mistake to say that the output from GPT3 lacks coherent meaning. It's not that the output is gibberish, it's that it's too easy to mistake it for a human's work. This means that it's easy to mistake it for something that was created with understanding and intention, when in fact the author was nothing more than a random number generator. The same risk exists for copilot. [--GPT3]
- deleted 5y ago
  [deleted]
- TchoBeer 5y ago
  >when in fact the author was nothing more than a random number generator GPT is not "nothing more than a random number generator"
  - nullc 5y ago
    Well, take it up with GPT3 since it wrote that reply. :P Though I don't fully disagree with it, though 'nothing more' is a bit too strong. The author of a GPT3 written comment like the one here where the prompt was pretty much just the thread is pretty much just the RNG. The language model makes the random choice draw from the distribution of plausible texts, and the RNG picks the output. GPT3 could have written your comment-- if only it drew the right random numbers.
    - TchoBeer 5y ago
      >pretty much just the RNG What RNG? It definitely doesn't randomly pick words. If the comment I responded to was written by a bot (is that legal? Can I report that?) then it's indistinguishable from a human written comment.
      - nullc 5y ago
        GPT3 works in compressed representation with symbols that are (sometimes) smaller than complete words but larger than letters. It takes a set of symbols as a context and generates a probability distribution function for the next symbol. Then a random number generator is used to sample from that distribution, and the process is repeated with the selected output added to the context. So its output is random but not uniformly random. Exclusively selecting the most likely symbol produces pathological behavior outside of extremely short output. What caused GPT3 to output its comment rather than yours is a product of its random choices. There is a set of choices it could have made which would have caused it to output your comment. You can see this property employed by the GPT2 text compressor: https://bellard.org/libnc/gpt2tc.html https://bellard.org/libnc/gpt2tc.html to compress text it just writes down the choices, using an entropy coder to represent likely choices with fewer bits. I assume copilot is the same general structure as GPT-- just trained on different data. And yes, the comment you responded to was written entirely by GPT3 (with some number of retries and trims). As it said-- it's "easy to mistake it for a human's work". :) There is nothing illegal about it, but I suppose HN would prefer that there be enough human supervision of bot comments such that they're limited to contexts where they are funny/insightful. :P
ayewo 5y ago
You claim that: > If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. There is a company in this space of generating "plausible tests" for legacy code bases at very large enterprises (think Goldman Sachs, telcos etc) called Diffblue [0]. They raised funding back in 2017 [1] and it seems their biggest value-add is in creating unit tests for legacy Java code bases that often have little to no unit tests. Essentially, these AI generated unit tests help a team "document" all known the behaviors of a legacy code base such that when a change is introduced that violates the behaviors covered by the generated unit tests, the tool can alert the team of the potential presence of a regression. Anyway, they offer a fairly basic browser-based demo of their AI product called Diffblue Cover [2]. Are you aware of them? 0: https://www.diffblue.com/ https://www.diffblue.com/ 1: https://techcrunch.com/2017/06/27/diffblue/ https://techcrunch.com/2017/06/27/diffblue/ 2: https://www.diffblue.com/try-cover-browser/ https://www.diffblue.com/try-cover-browser/
- wpietri 5y ago
  I haven't looked at it. But there are plenty of "magic beans" product targeted at Enterprise companies with legacy code. It's perfectly plausible to me that many of the companies using something generating bad tests wouldn't know the difference, because that's what their code base has already.
- gravypod 5y ago
  Is diffblue AI based or is it just property based testing? I assume that since it's limited to Java that they just decompile the opcodes and find what branches each method has and writes a test that calls each method with all possible permutations that lands down each branch.
  - ayewo 5y ago
    I'm not affiliated to them in any way, so I have no insight into their secret sauce. I shared because their product seems relevant to the discussion.

cmeacham98 5y ago

They didn't say they were going to have Copilot write _all_ the tests. Writing tests for cases you can think of and trying Copilot for the extras doesn't seem like that bad of an idea.

debaserab2 5y ago
Sounds like a good way to pile up your test suite with a bunch useless tests.
- haggy 5y ago
  Can you imagine the CI build time bloat? I'm getting anxiety just thinking about it

comicjk 5y ago

It would be awful to write every test using Copilot, but there is potential there for a certain kind of test. If I'm writing an API, I want fresh eyes on it, not just tests written by the person who understands it most (me). For example, a fresh user might try to apply a common pattern that my API breaks. Copilot might be able to act like such a tester. By writing generic tests, it could mimic developers who haven't understood the API before they start using it (most of them).

wpietri 5y ago
If you can find an example of Copilot coming up with a test you wouldn't have thought of, I'd be very interested to see it. Even if that happened, which I am not expecting, I think the need is much more easily solved via means that are simpler and more effective. E.g., a good tester writing up a list of things they test about APIs: https://www.sisense.com/blog/rest-api-testing-strategy-what-exactly-should-you-test/ https://www.sisense.com/blog/rest-api-testing-strategy-what-...
barkingcat 5y ago
copilot as defined would not be "fresh eyes"... it would be "old tired eyes of every code writer who uploaded stuff to github, not knowing if they made one off errors or mistakes in their code"
- comicjk 5y ago
  I mean fresh eyes with respect to my new API. Having seen a lot of other code is a benefit. I expect most tests that Copilot writes to fail, but I would hope some would fail in interesting ways. For example, off-by-one errors might encourage me to document my indexing convention, or to use a generator rather than indexing.
ska 5y ago
> I want fresh eyes on it Crucially, that's not what copilot is.

developer2 5y ago

> one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. I feel like you just described every developer/codebase where mock testing is stupidly enforced. Where every single unit test mocks every single indirect object. 98% of the testing code is just exhaustive setup and teardown of objects not being tested by each test, and then a bunch of conditional checks to ensure that every deeper/indirect method is being called exactly the right number of times with exactly the right arguments and returning exactly the right value. Almost all of the test code is just hacking mock objects. The actual purpose of each test is buried so deep that it's impossible to even understand the business logic being applied. I hate evangelical "mock testers" with a passion.

wpietri 5y ago
Yes. I have been doing unit testing for a long time, and I think this is a clear antipattern. It's cargo-cult testing, not actually a serious effort to improve quality and developer productivity.
spullara 5y ago
These are absolutely the worst tests I have ever seen. They make iterating on the implementation almost impossible. Why people do this I will never understand.
lexandstuff 5y ago
Preach. This is what chasing 100% test coverage looks like.

nullc 5y ago

It would be a mistake to say that the output from GPT3 lacks coherent meaning. It's not that the output is gibberish, it's that it's too easy to mistake it for a human's work. This means that it's easy to mistake it for something that was created with understanding and intention, when in fact the author was nothing more than a random number generator. The same risk exists for copilot. [--GPT3]

deleted 5y ago
[deleted]
TchoBeer 5y ago
>when in fact the author was nothing more than a random number generator GPT is not "nothing more than a random number generator"
- nullc 5y ago
  Well, take it up with GPT3 since it wrote that reply. :P Though I don't fully disagree with it, though 'nothing more' is a bit too strong. The author of a GPT3 written comment like the one here where the prompt was pretty much just the thread is pretty much just the RNG. The language model makes the random choice draw from the distribution of plausible texts, and the RNG picks the output. GPT3 could have written your comment-- if only it drew the right random numbers.
  - TchoBeer 5y ago
    >pretty much just the RNG What RNG? It definitely doesn't randomly pick words. If the comment I responded to was written by a bot (is that legal? Can I report that?) then it's indistinguishable from a human written comment.
    - nullc 5y ago
      GPT3 works in compressed representation with symbols that are (sometimes) smaller than complete words but larger than letters. It takes a set of symbols as a context and generates a probability distribution function for the next symbol. Then a random number generator is used to sample from that distribution, and the process is repeated with the selected output added to the context. So its output is random but not uniformly random. Exclusively selecting the most likely symbol produces pathological behavior outside of extremely short output. What caused GPT3 to output its comment rather than yours is a product of its random choices. There is a set of choices it could have made which would have caused it to output your comment. You can see this property employed by the GPT2 text compressor: https://bellard.org/libnc/gpt2tc.html https://bellard.org/libnc/gpt2tc.html to compress text it just writes down the choices, using an entropy coder to represent likely choices with fewer bits. I assume copilot is the same general structure as GPT-- just trained on different data. And yes, the comment you responded to was written entirely by GPT3 (with some number of retries and trims). As it said-- it's "easy to mistake it for a human's work". :) There is nothing illegal about it, but I suppose HN would prefer that there be enough human supervision of bot comments such that they're limited to contexts where they are funny/insightful. :P

ayewo 5y ago

You claim that: > If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. There is a company in this space of generating "plausible tests" for legacy code bases at very large enterprises (think Goldman Sachs, telcos etc) called Diffblue [0]. They raised funding back in 2017 [1] and it seems their biggest value-add is in creating unit tests for legacy Java code bases that often have little to no unit tests. Essentially, these AI generated unit tests help a team "document" all known the behaviors of a legacy code base such that when a change is introduced that violates the behaviors covered by the generated unit tests, the tool can alert the team of the potential presence of a regression. Anyway, they offer a fairly basic browser-based demo of their AI product called Diffblue Cover [2]. Are you aware of them? 0: https://www.diffblue.com/ https://www.diffblue.com/ 1: https://techcrunch.com/2017/06/27/diffblue/ https://techcrunch.com/2017/06/27/diffblue/ 2: https://www.diffblue.com/try-cover-browser/ https://www.diffblue.com/try-cover-browser/

wpietri 5y ago
I haven't looked at it. But there are plenty of "magic beans" product targeted at Enterprise companies with legacy code. It's perfectly plausible to me that many of the companies using something generating bad tests wouldn't know the difference, because that's what their code base has already.
gravypod 5y ago
Is diffblue AI based or is it just property based testing? I assume that since it's limited to Java that they just decompile the opcodes and find what branches each method has and writes a test that calls each method with all possible permutations that lands down each branch.
- ayewo 5y ago
  I'm not affiliated to them in any way, so I have no insight into their secret sauce. I shared because their product seems relevant to the discussion.