4 ms·

That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in

That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in and of themselves and no one wants to measure whether they are counterproductive. Even if one took that initiative, good luck getting anyone to care. Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better. Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.

geoffjentry 5y ago
I've become a fan of using mutation testing to drive unit test coverage. The goal of mutation testing is to ensure there is test coverage around areas with a higher likelihood of being brittle. Work smarter, not harder.
lytefm 5y ago
Worse still, I feel like doing unit testing at the expense of integration testing can create a false sense of security. Unit tests often skip the error cases that are really important for real life scenarios. We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.
- voxic11 5y ago
  I feel the opposite way, with unit tests you can test failure scenarios as easily as any other scenario. So it makes it simple to exhaustively test every code path including failure scenarios that would be very difficult to set up in integration tests.
  - mywittyname 5y ago
    There are bugs that don't appear as such at the unit level. Two objects might individually be in perfectly valid states, but those states can be invalid at a system level. Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?
- johndfsgdgdfg 5y ago
  False sense of security sold by conference indsutrial complex is my biggest issues with unit testing. One time I've seen a junior dev wrote copious amount of unit tests to achieve near 100% code coverage. When this "perfect" code went into prod it almost took the whole system down. Because the junior dev was joining close to a dozen tables and each one of the tables has at least a million rows. If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.
  - rat9988 5y ago
    I'm pretty sure if he didn't write the unit tests it would have equally failed. There is something to blame here, but not the unit tests.
  - sharken 5y ago
    No doubt the junior dev would benefit a lot more from only getting up to 60-70% code coverage. The extra time could then be used to learn basic Db performance, which is hugely important when tables have millions or billions of rows. Stuff like indexes and isolation levels should be taught to all junior devs, it's just too risky not to.
whakim 5y ago
> Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. For sure there's diminishing returns on test coverage and I share your frustration with the myopic focus on coverage but there's no way it's anywhere close to 25%. Perhaps 25% is sufficient (or way more than enough) for new, small projects using languages with good compilers and type checkers. But it's no where near enough even under those ideal conditions when you have several generations of developers working on big multi-year projects. Tests do reduce bugs. That's why automated testing was invented in the first place. Number of bugs vs company revenue isn't ignored because it'd make people look bad, but because it's an incredibly difficult metric to calculate.

geoffjentry 5y ago

I've become a fan of using mutation testing to drive unit test coverage. The goal of mutation testing is to ensure there is test coverage around areas with a higher likelihood of being brittle. Work smarter, not harder.

lytefm 5y ago

Worse still, I feel like doing unit testing at the expense of integration testing can create a false sense of security. Unit tests often skip the error cases that are really important for real life scenarios. We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.

voxic11 5y ago
I feel the opposite way, with unit tests you can test failure scenarios as easily as any other scenario. So it makes it simple to exhaustively test every code path including failure scenarios that would be very difficult to set up in integration tests.
- mywittyname 5y ago
  There are bugs that don't appear as such at the unit level. Two objects might individually be in perfectly valid states, but those states can be invalid at a system level. Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?
johndfsgdgdfg 5y ago
False sense of security sold by conference indsutrial complex is my biggest issues with unit testing. One time I've seen a junior dev wrote copious amount of unit tests to achieve near 100% code coverage. When this "perfect" code went into prod it almost took the whole system down. Because the junior dev was joining close to a dozen tables and each one of the tables has at least a million rows. If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.
- rat9988 5y ago
  I'm pretty sure if he didn't write the unit tests it would have equally failed. There is something to blame here, but not the unit tests.
- sharken 5y ago
  No doubt the junior dev would benefit a lot more from only getting up to 60-70% code coverage. The extra time could then be used to learn basic Db performance, which is hugely important when tables have millions or billions of rows. Stuff like indexes and isolation levels should be taught to all junior devs, it's just too risky not to.

whakim 5y ago

> Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. For sure there's diminishing returns on test coverage and I share your frustration with the myopic focus on coverage but there's no way it's anywhere close to 25%. Perhaps 25% is sufficient (or way more than enough) for new, small projects using languages with good compilers and type checkers. But it's no where near enough even under those ideal conditions when you have several generations of developers working on big multi-year projects. Tests do reduce bugs. That's why automated testing was invented in the first place. Number of bugs vs company revenue isn't ignored because it'd make people look bad, but because it's an incredibly difficult metric to calculate.