5 ms·

How well can copilot write unit tests? This seems like an area where it could be really useful and actually improve software development practices.

kqr 5y ago
If you're looking for test case generation there are already mature tools for that. I doubt anything generic could improve on those.
- ericlewis 5y ago
  any suggestions for said tools?
  - prionassembly 5y ago
    Hypothesis for Python. Schemathesis builds on Hypothesis and generates tests from OpenAPI specs.
    - dmitry_dygalo 5y ago
      And from GraphQL APIs as well :)
  - svieira 5y ago
    QuickcCheck-type tools (generators for tests that know about the edge cases of a domain - e. g. for the domain of numbers considering things like 0, the infinities, various almost-and-just-over powers of two, NaN and mantissas for floats, etc.): * QuickCheck: https://hackage.haskell.org/package/QuickCheck https://hackage.haskell.org/package/QuickCheck * Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/ * JUnit QuickCheck: https://github.com/pholser/junit-quickcheck https://github.com/pholser/junit-quickcheck Fuzz testing tools (tools which mutate the inputs to a program in order to find interesting / failing states in that program). Generally paired with code coverage: * American Fuzzy Lop (AFL): https://github.com/google/AFL https://github.com/google/AFL * JQF: https://github.com/rohanpadhye/JQF https://github.com/rohanpadhye/JQF Mutation / Fault based test tools (review your existing unit coverage and try to introduce changes to your _production_ code that none of your tests catch) * PITest: https://pitest.org/ https://pitest.org/
manquer 5y ago
Writing tests for the sake of coverage is already practically useless which is what a lot of orgs do, This could maybe generate such tests. However it doesn't materially impact quality now, so not much difference if automated. One of the main value props for writing meaningful unit tests, is it helps the developer think differently about the code he is writing tests for, and that improves quality of the code composition.
- Graffur 5y ago
  Why is that useless? Codebases I have worked on that had high code coverage requirements had very little bugs. * It promotes actually looking at the code before considering it done * It promotes refactoring * It helps to prevent breaking changes for stuff that wasn't supposed to change
  - matsemann 5y ago
    I feel the opposite of codebases where having high coverage has been a priority: * The tests doesn't actually test functionality, edge cases etc, just that things doesn't crash in a happy-path. * Any changes to an implementation breaks a test needlessly, because the test tests specifics of the implementation, not correctness. Thus it makes refactoring actually harder, since your test said you broke something, but you probably didn't, and now you have to double the work of writing a new test. * In codebases for dynamic languages, most of what these tests end up catching is stuff a compiler would catch in a statically typed language.
    - drran 5y ago
      > The tests doesn't actually test functionality, edge cases etc, just that things doesn't crash in a happy-path. This is low coverage. > Any changes to an implementation breaks a test needlessly, because the test tests specifics of the implementation, not correctness. This is bad design. > In codebases for dynamic languages, most of what these tests end up catching is stuff a compiler would catch in a statically typed language. So they are not useless.
golergka 5y ago
On my current pet project, it has written almost all of the tests here: https://github.com/golergka/rs-lox/blob/master/src/compiler.rs#L421 https://github.com/golergka/rs-lox/blob/master/src/compiler.... ad here: https://github.com/golergka/rs-lox/blob/master/src/scanner.rs#L303 https://github.com/golergka/rs-lox/blob/master/src/scanner.r... (albeit not the screwed indentation) completely by itself. I didn't even have to write the function names, just a few macros to help it along and a couple of examples to teach it to use it.
francilien 5y ago
Here is one: https://www.ponicode.com/ https://www.ponicode.com/
sixstringtheory 5y ago
I think replies mentioning automatic unit test generation miss the point. To me, the value of copilot helping to write tests is that we, the engineers, come up with the test cases, and copilot helps write the code for that case. I think humans will still be more imaginative in the test cases they can dream up (although I’ve never used an automatic generator, maybe they’re better than I think), but almost all test code is boilerplate, either in the setup or the assertions. If I don’t have to write that repetitious, yet slightly different boilerplate for each test case, that frees me up to design other interesting test cases (as opposed to getting tired of the activity by the time I cover the happy path) or move on to the next bug/feature work.

kqr 5y ago

If you're looking for test case generation there are already mature tools for that. I doubt anything generic could improve on those.

ericlewis 5y ago
any suggestions for said tools?
- prionassembly 5y ago
  Hypothesis for Python. Schemathesis builds on Hypothesis and generates tests from OpenAPI specs.
  - dmitry_dygalo 5y ago
    And from GraphQL APIs as well :)
- svieira 5y ago
  QuickcCheck-type tools (generators for tests that know about the edge cases of a domain - e. g. for the domain of numbers considering things like 0, the infinities, various almost-and-just-over powers of two, NaN and mantissas for floats, etc.): * QuickCheck: https://hackage.haskell.org/package/QuickCheck https://hackage.haskell.org/package/QuickCheck * Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/ * JUnit QuickCheck: https://github.com/pholser/junit-quickcheck https://github.com/pholser/junit-quickcheck Fuzz testing tools (tools which mutate the inputs to a program in order to find interesting / failing states in that program). Generally paired with code coverage: * American Fuzzy Lop (AFL): https://github.com/google/AFL https://github.com/google/AFL * JQF: https://github.com/rohanpadhye/JQF https://github.com/rohanpadhye/JQF Mutation / Fault based test tools (review your existing unit coverage and try to introduce changes to your _production_ code that none of your tests catch) * PITest: https://pitest.org/ https://pitest.org/

manquer 5y ago

Writing tests for the sake of coverage is already practically useless which is what a lot of orgs do, This could maybe generate such tests. However it doesn't materially impact quality now, so not much difference if automated. One of the main value props for writing meaningful unit tests, is it helps the developer think differently about the code he is writing tests for, and that improves quality of the code composition.

Graffur 5y ago
Why is that useless? Codebases I have worked on that had high code coverage requirements had very little bugs. * It promotes actually looking at the code before considering it done * It promotes refactoring * It helps to prevent breaking changes for stuff that wasn't supposed to change
- matsemann 5y ago
  I feel the opposite of codebases where having high coverage has been a priority: * The tests doesn't actually test functionality, edge cases etc, just that things doesn't crash in a happy-path. * Any changes to an implementation breaks a test needlessly, because the test tests specifics of the implementation, not correctness. Thus it makes refactoring actually harder, since your test said you broke something, but you probably didn't, and now you have to double the work of writing a new test. * In codebases for dynamic languages, most of what these tests end up catching is stuff a compiler would catch in a statically typed language.
  - drran 5y ago
    > The tests doesn't actually test functionality, edge cases etc, just that things doesn't crash in a happy-path. This is low coverage. > Any changes to an implementation breaks a test needlessly, because the test tests specifics of the implementation, not correctness. This is bad design. > In codebases for dynamic languages, most of what these tests end up catching is stuff a compiler would catch in a statically typed language. So they are not useless.

golergka 5y ago

On my current pet project, it has written almost all of the tests here: https://github.com/golergka/rs-lox/blob/master/src/compiler.rs#L421 https://github.com/golergka/rs-lox/blob/master/src/compiler.... ad here: https://github.com/golergka/rs-lox/blob/master/src/scanner.rs#L303 https://github.com/golergka/rs-lox/blob/master/src/scanner.r... (albeit not the screwed indentation) completely by itself. I didn't even have to write the function names, just a few macros to help it along and a couple of examples to teach it to use it.

francilien 5y ago

Here is one: https://www.ponicode.com/ https://www.ponicode.com/

sixstringtheory 5y ago

I think replies mentioning automatic unit test generation miss the point. To me, the value of copilot helping to write tests is that we, the engineers, come up with the test cases, and copilot helps write the code for that case. I think humans will still be more imaginative in the test cases they can dream up (although I’ve never used an automatic generator, maybe they’re better than I think), but almost all test code is boilerplate, either in the setup or the assertions. If I don’t have to write that repetitious, yet slightly different boilerplate for each test case, that frees me up to design other interesting test cases (as opposed to getting tired of the activity by the time I cover the happy path) or move on to the next bug/feature work.