11 ms·

99% code coverage (2017)

100 points by fagnerbrack 8y ago

Radle 8y ago
The author suggests to add mutations to the code, like replacing '<' with '<=' in order to "test the tests". He assumes that tests should fail due to such small differences. I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit though and draws parallels to writing tests with variable or random input parameters, I am pretty sure there already was something on this.
- adrianN 8y ago
  You think customers aren't affected if you mistakenly replace a < with a <= in your code?
  - majewsky 8y ago
    while (car.wheels < 4) car.attach(new Wheel);
- nathan_f77 8y ago
  '<' versus '<=' can be very important, and it can cause severe bugs when dealing with things like array indices or financial data. There are some cases where it's not a big deal. '<' and '<=' can be interchangeable if one or both of the values are a rough estimate. One example would be comparing two images and doing something if they are a close match. It might not be too bad if you use `<= 0.995` instead of `< 0.995` when checking the percentage. Some other examples might involve accelerometer data, pressure sensors, GPS boundaries, debounce algorithms, animations, etc. But I think it's just a good habit to write test cases that test those boundary values.
  - progval 8y ago
    Basically, it depends on whether we're dealing with floats or integers.
- rachelmcarmena 8y ago
  Thanks for the feedback. The post only answers "Do we have a good safety net to change this legacy code?". Mutation testing is not a new concept. It's usually used in critical systems: the system is mutated and you measure the time for detecting the mutation and the time for solving the problem.
peterwaller 8y ago
Mutation testing is a neat idea I'd not heard of. Wonder how well it works in practice. Someone's implemented a package for doing it with Go which looks good: https://github.com/zimmski/go-mutesting https://github.com/zimmski/go-mutesting
- jesusmg 8y ago
  Me neither, I liked the concept of mutation testing. (I was using this performing changes manually, without knowing this technique has a nice name). I would appreciate if somebody points out a mutation framework /tools for .net
  - purity_resigns 8y ago
    https://github.com/fscheck/FsCheck https://github.com/fscheck/FsCheck is something I've used very briefly in the past. I did more work with this sort of thing in Scala. In my experience, you usually end up with much more coverage than you want or need.
  - mdpopescu 8y ago
    There's something that used to be called Pex, and now is apparently called IntelliTest [1], but I believe it's only available in the Enterprise edition of Visual Studio. [1] https://docs.microsoft.com/en-us/visualstudio/test/intellitest-manual/getting-started?view=vs-2017 https://docs.microsoft.com/en-us/visualstudio/test/intellite...
- Vinnl 8y ago
  I haven't worked with it extensively, but from what I've seen, it's far beyond what most teams should consider. It's basically a more thorough method of measuring test coverage, pointing out cases that have not been covered by your tests yet. However, the number of tests that it would have you write to reach 100% mutation coverage is not justified by the number of bugs you'll catch, unless the impact of any bug is very high (I suppose at NASA?). In fact, the amount of work required to even check which tests you missed is already not justified. I usually use coverage as a tool to remind me of pieces of code I intended but forgot to test [1], so setting it to an arbitrary percentage is not that useful, in my opinion. If 100% is not feasible usually as well, that makes Mutation Testing in general not worth the effort. Again, take my opinion with a grain of salt. [1] https://vincenttunru.com/100-percent-coverage https://vincenttunru.com/100-percent-coverage
pedro1976 8y ago
I had a similar experience, you always get what you measure. IMO the best a approach is to first integrate the test coverage in the code reviews, cause there is no hard rule [0] and second write property-based tests [1]. [0] http://www.se-radio.net/2018/05/se-radio-episode-324-marc-hoffmann-on-code-test-coverage-analysis-and-tools/ http://www.se-radio.net/2018/05/se-radio-episode-324-marc-ho... [1] Sample framework for JavaScript https://github.com/jsverify/jsverify https://github.com/jsverify/jsverify
- sethammons 8y ago
  I like the idea of property based testing, but have found that it obfuscated the test code, making it much harder to read (especially for new teammates). This was in Go, so maybe it had a lot more cruft around it than it would in something more functional or js.
- Vinnl 8y ago
  I prefer just setting coverage requirements to 100%, and give developers the freedom to mark code as ignored for coverage (e.g. /* istanbul ignore next */ for many Javascript applications). That way, the annotation is something that can be brought up during code reviews in case the reviewer does not agree with that not being covered, and it doesn't depend on the reviewer having to remember to run or look at a separate coverage report. (I wrote more about this here: https://vincenttunru.com/100-percent-coverage/ https://vincenttunru.com/100-percent-coverage/)
cryptica 8y ago
The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project. If you build square wheels for example, you may not realize that they're not designed correctly until you attach them to the car and realize that the car doesn't function well with them. It makes no sense to preemtively write unit tests for a component which has a very high likelihood of not being in its final desired state; you're just giving yourself more work to refactor the unit tests over and over; or worse, you're afraid of refactoring the tests and so you lie to yourself thinking that square wheels are fine. Integration tests are by far the most useful tests to have at the beginning and middle stages of the project; integration tests allow you to keep your focus on the real goals of the project and not get stuck on designing the perfect square wheel.
- rypskar 8y ago
  What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
  - twodave 8y ago
    The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.
    - jondubois 8y ago
      Wikipedia disagrees with that definition: "Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method.[2] Unit tests are short code fragments[3] created by programmers or occasionally by white box testers during the development process. It forms the basis for component testing.[4] Ideally, each test case is independent from the others. Substitutes such as method stubs, mock objects,[5] fakes, and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended." https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing
dankohn1 8y ago
Does anyone have experience with https://github.com/stryker-mutator/stryker https://github.com/stryker-mutator/stryker for code mutation testing?
sethammons 8y ago
I hadn't realized this mutation testing existed as automated tooling. I'll be looking more into it. Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.
- josteink 8y ago
  > Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed. That's completely backwards compared to how you should be doing things. You either write the test: 1. to verify the existence of a bug by reproducing it, or 2. to formalize the spec for yet-to-be implemented code/feature. And then you make the test green. Retroactively writing tests for working code, only to sabotage the code... Seems like an odd way of doing things.
  - sethammons 8y ago
    You don't always write a test first. A bug? Sure, write a test that shows the bug, fix it, make it green. A very well defined something? Sure, TDD. Many times, you will write code (or come across pre-existing code) and tests come last.
    - TravelAndFood 8y ago
      Yep, TDD is quite hard when requirements are vague/changing... the reqs changed? OK now I have to rewrite my test AND the code that passes it. Vague/changing reqs is of course a separate problem.
  - paulriddle 8y ago
    Some people feel more comfortable writing code without tests first, gradually shaping final design of the interface. Especially if the tests involve a lot of stubbing and mocking of things like Redis and Sphinx Search, or messing with crypto tokens, parsing HTML, freezing time, setting up global config attributes like support emails, interaction between 2 different databases, etc. Tests are code as well and oftentimes might feel heavier than production code. You can of course say that there is a way things should be no matter what but that might lead to negative emotions, toxicity and complaining instead of getting stuff done.
iamleppert 8y ago
Who is going to test that the mutators are doing the right kind of mutations? lol I’ve worked on code that has 1000’s of tests and is still a bug ridden hell.
- blueyed 8y ago
  Those tools also have tests.. :)
_pmf_ 8y ago
You can do that, or you can let priority of business requirements drive your testing efforts. For each project, I have a certain set of hard requirements that will never be used in the field. We deliver these half-assed and fix bugs if 2 years later someone uses the feature by accident.
snidane 8y ago
I always achieve 100% relevant code coverage. I just set up a test with one input which tests my main(), which then obviously calls all the relevant parts of the code. Or am I missing something in the definition of code coverage? I don't see anyone using exhaustive-input tests, which is usually impossible anyway, or splitting up each branch of conditionals into separate functions so as to make them unit testable. I only see people splitting up the code into arbitrary function blocks, having at least one unit test for such function and then declaring all lines of code of that function as test-covered.
simplecomplex 8y ago
Use a language that doesn’t allow incorrect code and you get 100% code coverage without writing tests. Smart compilers and precise language (ala Haskell) obviate the need for writing unit tests.
blueyed 8y ago
Some tools for mutation testing in Python: - https://github.com/boxed/mutmut https://github.com/boxed/mutmut - https://github.com/sixty-north/cosmic-ray https://github.com/sixty-north/cosmic-ray
luord 8y ago
I can't imagine, as a developer, writing tests without assertions. Any developer who does that is actively making the situation worse for the team and the application. Conversely, I've worked in teams whose managers encouraged the developers to forgo tests for the sake of delivering faster. Never again.

100 points by fagnerbrack 8y ago

The author suggests to add mutations to the code, like replacing '<' with '<=' in order to "test the tests". He assumes that tests should fail due to such small differences. I think this might apply when you write your own algorithms and want to test them. But if you are, like most of us, working on Business Logic, then you are probably writing the wrong tests. We usually want to know whether a workflow or a customer story works as intended and as such we should write more Integration tests. The idea itself has merit though and draws parallels to writing tests with variable or random input parameters, I am pretty sure there already was something on this.

adrianN 8y ago
You think customers aren't affected if you mistakenly replace a < with a <= in your code?
- majewsky 8y ago
  while (car.wheels < 4) car.attach(new Wheel);
nathan_f77 8y ago
'<' versus '<=' can be very important, and it can cause severe bugs when dealing with things like array indices or financial data. There are some cases where it's not a big deal. '<' and '<=' can be interchangeable if one or both of the values are a rough estimate. One example would be comparing two images and doing something if they are a close match. It might not be too bad if you use `<= 0.995` instead of `< 0.995` when checking the percentage. Some other examples might involve accelerometer data, pressure sensors, GPS boundaries, debounce algorithms, animations, etc. But I think it's just a good habit to write test cases that test those boundary values.
- progval 8y ago
  Basically, it depends on whether we're dealing with floats or integers.
rachelmcarmena 8y ago
Thanks for the feedback. The post only answers "Do we have a good safety net to change this legacy code?". Mutation testing is not a new concept. It's usually used in critical systems: the system is mutated and you measure the time for detecting the mutation and the time for solving the problem.

peterwaller 8y ago

Mutation testing is a neat idea I'd not heard of. Wonder how well it works in practice. Someone's implemented a package for doing it with Go which looks good: https://github.com/zimmski/go-mutesting https://github.com/zimmski/go-mutesting

jesusmg 8y ago
Me neither, I liked the concept of mutation testing. (I was using this performing changes manually, without knowing this technique has a nice name). I would appreciate if somebody points out a mutation framework /tools for .net
- purity_resigns 8y ago
  https://github.com/fscheck/FsCheck https://github.com/fscheck/FsCheck is something I've used very briefly in the past. I did more work with this sort of thing in Scala. In my experience, you usually end up with much more coverage than you want or need.
- mdpopescu 8y ago
  There's something that used to be called Pex, and now is apparently called IntelliTest [1], but I believe it's only available in the Enterprise edition of Visual Studio. [1] https://docs.microsoft.com/en-us/visualstudio/test/intellitest-manual/getting-started?view=vs-2017 https://docs.microsoft.com/en-us/visualstudio/test/intellite...
Vinnl 8y ago
I haven't worked with it extensively, but from what I've seen, it's far beyond what most teams should consider. It's basically a more thorough method of measuring test coverage, pointing out cases that have not been covered by your tests yet. However, the number of tests that it would have you write to reach 100% mutation coverage is not justified by the number of bugs you'll catch, unless the impact of any bug is very high (I suppose at NASA?). In fact, the amount of work required to even check which tests you missed is already not justified. I usually use coverage as a tool to remind me of pieces of code I intended but forgot to test [1], so setting it to an arbitrary percentage is not that useful, in my opinion. If 100% is not feasible usually as well, that makes Mutation Testing in general not worth the effort. Again, take my opinion with a grain of salt. [1] https://vincenttunru.com/100-percent-coverage https://vincenttunru.com/100-percent-coverage

pedro1976 8y ago

I had a similar experience, you always get what you measure. IMO the best a approach is to first integrate the test coverage in the code reviews, cause there is no hard rule [0] and second write property-based tests [1]. [0] http://www.se-radio.net/2018/05/se-radio-episode-324-marc-hoffmann-on-code-test-coverage-analysis-and-tools/ http://www.se-radio.net/2018/05/se-radio-episode-324-marc-ho... [1] Sample framework for JavaScript https://github.com/jsverify/jsverify https://github.com/jsverify/jsverify

sethammons 8y ago
I like the idea of property based testing, but have found that it obfuscated the test code, making it much harder to read (especially for new teammates). This was in Go, so maybe it had a lot more cruft around it than it would in something more functional or js.
Vinnl 8y ago
I prefer just setting coverage requirements to 100%, and give developers the freedom to mark code as ignored for coverage (e.g. /* istanbul ignore next */ for many Javascript applications). That way, the annotation is something that can be brought up during code reviews in case the reviewer does not agree with that not being covered, and it doesn't depend on the reviewer having to remember to run or look at a separate coverage report. (I wrote more about this here: https://vincenttunru.com/100-percent-coverage/ https://vincenttunru.com/100-percent-coverage/)

cryptica 8y ago

The industry is obsessed with getting 100% unit test code coverage even though it doesn't mean anything to the project. The purpose of unit tests is to lock down the project's source code once it's essentially completed; to avoid regressions when making minor changes. If you start writing unit tests too early in the project, you're effecively locking down units of code which haven't yet proved themseves to be useful to your project. If you build square wheels for example, you may not realize that they're not designed correctly until you attach them to the car and realize that the car doesn't function well with them. It makes no sense to preemtively write unit tests for a component which has a very high likelihood of not being in its final desired state; you're just giving yourself more work to refactor the unit tests over and over; or worse, you're afraid of refactoring the tests and so you lie to yourself thinking that square wheels are fine. Integration tests are by far the most useful tests to have at the beginning and middle stages of the project; integration tests allow you to keep your focus on the real goals of the project and not get stuck on designing the perfect square wheel.

rypskar 8y ago
What is an unit? Is it the wheel or the car? When you start to see that the car is the unit the difference between an unit test and an integration test gets blurry.
- twodave 8y ago
  The basic rule I go by is whether or not the test is self-contained. For example if you are making a trip to a database or hitting a 3rd party dll then it’s an integration test. I thought this was pretty widely accepted.
  - jondubois 8y ago
    Wikipedia disagrees with that definition: "Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method.[2] Unit tests are short code fragments[3] created by programmers or occasionally by white box testers during the development process. It forms the basis for component testing.[4] Ideally, each test case is independent from the others. Substitutes such as method stubs, mock objects,[5] fakes, and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended." https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing

dankohn1 8y ago

Does anyone have experience with https://github.com/stryker-mutator/stryker https://github.com/stryker-mutator/stryker for code mutation testing?

sethammons 8y ago

I hadn't realized this mutation testing existed as automated tooling. I'll be looking more into it. Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed.

josteink 8y ago
> Traditionally, I've gone with "sabotaging" the code when writing unit tests; altering the code to verify a test goes from red to green or vise versa. Never trust a test that has never failed. That's completely backwards compared to how you should be doing things. You either write the test: 1. to verify the existence of a bug by reproducing it, or 2. to formalize the spec for yet-to-be implemented code/feature. And then you make the test green. Retroactively writing tests for working code, only to sabotage the code... Seems like an odd way of doing things.
- sethammons 8y ago
  You don't always write a test first. A bug? Sure, write a test that shows the bug, fix it, make it green. A very well defined something? Sure, TDD. Many times, you will write code (or come across pre-existing code) and tests come last.
  - TravelAndFood 8y ago
    Yep, TDD is quite hard when requirements are vague/changing... the reqs changed? OK now I have to rewrite my test AND the code that passes it. Vague/changing reqs is of course a separate problem.
- paulriddle 8y ago
  Some people feel more comfortable writing code without tests first, gradually shaping final design of the interface. Especially if the tests involve a lot of stubbing and mocking of things like Redis and Sphinx Search, or messing with crypto tokens, parsing HTML, freezing time, setting up global config attributes like support emails, interaction between 2 different databases, etc. Tests are code as well and oftentimes might feel heavier than production code. You can of course say that there is a way things should be no matter what but that might lead to negative emotions, toxicity and complaining instead of getting stuff done.

iamleppert 8y ago

Who is going to test that the mutators are doing the right kind of mutations? lol I’ve worked on code that has 1000’s of tests and is still a bug ridden hell.

blueyed 8y ago
Those tools also have tests.. :)

_pmf_ 8y ago

You can do that, or you can let priority of business requirements drive your testing efforts. For each project, I have a certain set of hard requirements that will never be used in the field. We deliver these half-assed and fix bugs if 2 years later someone uses the feature by accident.

snidane 8y ago

I always achieve 100% relevant code coverage. I just set up a test with one input which tests my main(), which then obviously calls all the relevant parts of the code. Or am I missing something in the definition of code coverage? I don't see anyone using exhaustive-input tests, which is usually impossible anyway, or splitting up each branch of conditionals into separate functions so as to make them unit testable. I only see people splitting up the code into arbitrary function blocks, having at least one unit test for such function and then declaring all lines of code of that function as test-covered.

simplecomplex 8y ago

Use a language that doesn’t allow incorrect code and you get 100% code coverage without writing tests. Smart compilers and precise language (ala Haskell) obviate the need for writing unit tests.

blueyed 8y ago

Some tools for mutation testing in Python: - https://github.com/boxed/mutmut https://github.com/boxed/mutmut - https://github.com/sixty-north/cosmic-ray https://github.com/sixty-north/cosmic-ray

luord 8y ago

I can't imagine, as a developer, writing tests without assertions. Any developer who does that is actively making the situation worse for the team and the application. Conversely, I've worked in teams whose managers encouraged the developers to forgo tests for the sake of delivering faster. Never again.