6 ms·

Team I'm on obsesses with 100% unit test coverage which means that it takes 3-5x as long to ship a feature while engineers write tests that are: 1) Almost alwa

Team I'm on obsesses with 100% unit test coverage which means that it takes 3-5x as long to ship a feature while engineers write tests that are: 1) Almost always better to be part of integration test 2) Offer no real ROI because the framework usually handles most of these issues gracefully to begin with, so its just a few hundred lines of cute code trying to mock out shit (or better yet getting multiple people in a zoom and wasting their time trying to test 4 lines of code - WHICH THE FKIN LIBRARY/FRAMEWORK ALREADY HANDLES) 3) Complete garbage. They write the code first and then add tests. That's fine if these were integration tests, but they run their little commands to print out a report that says 92% test coverage. Oh no No nO nO NO No meme-1.gif.jpeg. We have incidents regularly compared to my previous team, which was granted all Senior/Staff engineers and we didn't spend our time chasing dumbass metrics. If you want 100% unit test coverage, just do TDD and not this half-ass yolo cowboy approach. Or just write simple, single responsibility code so it can be tested easily. But nah, you know how it is. These people are annoying as hell to work with, and unfortunately I work at a bigger company so this kind of BS is normal as people try to chase promotions instead of actually shipping quality products. I personally wouldn't hire people like this, nor keep them around this long.

jrockway 5y ago
100% test coverage is really the starting point for testing. Once all the lines of code you've typed in actually run before they're delivered to the customer, then you can start adding the real test scenarios!
- hinkley 5y ago
  And how, pray tell, do you discover all of the scenarios that need to be covered? If only there were some sort of coverage tool that could tell you...
ravenstine 5y ago
That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in and of themselves and no one wants to measure whether they are counterproductive. Even if one took that initiative, good luck getting anyone to care. Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better. Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.
- geoffjentry 5y ago
  I've become a fan of using mutation testing to drive unit test coverage. The goal of mutation testing is to ensure there is test coverage around areas with a higher likelihood of being brittle. Work smarter, not harder.
- lytefm 5y ago
  Worse still, I feel like doing unit testing at the expense of integration testing can create a false sense of security. Unit tests often skip the error cases that are really important for real life scenarios. We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.
  - voxic11 5y ago
    I feel the opposite way, with unit tests you can test failure scenarios as easily as any other scenario. So it makes it simple to exhaustively test every code path including failure scenarios that would be very difficult to set up in integration tests.
    - mywittyname 5y ago
      There are bugs that don't appear as such at the unit level. Two objects might individually be in perfectly valid states, but those states can be invalid at a system level. Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?
  - johndfsgdgdfg 5y ago
    False sense of security sold by conference indsutrial complex is my biggest issues with unit testing. One time I've seen a junior dev wrote copious amount of unit tests to achieve near 100% code coverage. When this "perfect" code went into prod it almost took the whole system down. Because the junior dev was joining close to a dozen tables and each one of the tables has at least a million rows. If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.
gizmo686 5y ago
I think this approach comes out of highly dynamic languages like python. If a typo in a variable name doesn't cause any problems until it get evaluated, then you really need to execute every piece of the code to be sure that you don't have incredibly simple errors. If you are using languages with a even a primitive static type system, you can catch a lot of errors at compile time (or when the interprartor loads them), which provides most of the benifits of 100% coverage.
- theonething 5y ago
  for undeclared variables PHP is even worse. It just keeps going and just gives you a notice!
  - rbanffy 5y ago
    PHP has a lot in common with BASIC at that.
- nicoburns 5y ago
  > If a typo in a variable name doesn't cause any problems until it get evaluated Linters even in languages like python will catch that kind of thing.
stouset 5y ago
Also, this kind of mentality frequently results in what I consider negative-value tests. A test suite is supposed to do two things: a) identify bugs by failing on incorrect behavior, and b) enable refactoring by continuing to pass when underlying code doesn't change behavior Focusing on 100% unit test coverage all the time—particularly in code that isn't doing simple, single responsibility stuff—often results (in my experience) in tests that overly mock other parts of the system. The net result is that you end up writing tests that simple check that "the code was written the way it is currently written". Tests like this fail on the first count because they fundamentally cannot detect incorrect behavior, and they fail on the second count because a non-trivial refactor inevitably changes the implementation even if the resulting behavior stays the same. Not that mocks aren't useful in some scenarios, but if you find yourself writing entire test suites based on them you're likely creating tests that are negative value. They won't catch undiscovered bugs, and they'll give you bad information when you're refactoring.
- lichtenberger 5y ago
  That's precisely what I did in the beginning of my career. To test every method and mock all the dependencies. This had the impact, because we usually don't use a dependency injection framework, that I had to add an interface for almost every class, adding factories with interfaces almost everywhere and then write tests which expect certain method calls and to basically debug the tests while writing them until all calls have been mocked. Thus, whenever I changed the production code I had to change the test as well. IMHO in my own projects I tend to write tests which test certain parts of a system as well as some dependencies. So usually I'd say they are integration tests instead of unit tests. But I can refactor stuff and once I break something it's usually detected by one of the tests. Maybe unit tests should test for certain cases border cases, which are hard to test with integration tests.
- seanhunter 5y ago
  I heartily agree with your general sentiment, but think you're not mentioning a very important function of a test suite which is: c) explicitly and accurately document expected behavior including in a variety of edge cases ...this is sort of implicit in your first 2 but I think worth breaking out. Going for high test coverage (in places where I've done TDD-style dev) has really forced me to think through those edge cases and what I want my code to do in all of them. I think negative-value testing comes from the mindset of "how do I write a test which exercises this branch?" but a positive version of that is "I put this branch to handle the case where foo is getting close to the boundary. Let's put in test cases with foo at and close to the boundary and make sure it does what it is supposed to." If the test focuses on behavior then you find the bugs and enable refactoring and you get high coverage because if you can't exercise all the paths probably some of them are dead and can just get deleted.
avl999 5y ago
Does your team work on a codebase that uses a dynamically typed language? If so then the 100% coverage requirement is perfectly reasonable and arguably the only sane way to develop anything more than a tiny project. If a statically typed language is being used then sure, that requirement is counterproductive but if a team uses a dynamically typed language I don't see a way to have any reasonable amount of confidence that the code is even syntactically and semantically valid without 100% coverage. The 100% coverage is just a tax that you need to pay in dynamically typed codebases (hence my preference to use statically typed language for any reasonably sized work related project). For 3) TDD has nothing to with test coverage or aiming for high coverage. The fact that TDD evangelists have tried to couple "good testing" with TDD is a huge scam. I don't care when you write your tests as long as they get written before the code is merged in. TDD is good for parts of the code which are very pure or where you know exactly what you are doing, but if what you are doing is more exploratory in nature or is not pure and has lots of external dependencies TDD is simply not practical for a lot of people.
- cbhl 5y ago
  In my experience, scaling a large project in a dynamically typed language with 100% unit test coverage just gets you production outages with all unit tests passing. A reasonable threshold is probably closer to 70-90% _plus integration tests_. Hopefully you also picked a language where a bigco has created tooling that lets you incrementally add static types -- TypeScript, Closure Compiler, Python typing and/or pytype, etc.
  - lenkite 5y ago
    Lol, this is so true. A project that I had joined (late) some years ago had 98%! code coverage, full sonar/fortify code compliance and was an utter piece of garbage that had regular outages in production. It taught me a couple of lessons: 1. Integration and System Tests are FAR more valuable than unit tests. By several orders of magnitude. 2. Performance Tests are vital. This is actually more true in a cloud world. There is only so much hardware you can throw at a problem before your costs are un-feasible.
- cush 5y ago
vmception 5y ago
I like this post, because I avoided unit tests for so long lol. After finally doing them, I did notice that I would at least format my code better and this had other benefits. There would be cases I missed, made me start thinking about edge cases further in advance, as well as made it easier for other to read. I think the primary point is made > Or just write simple, single responsibility code so it can be tested easily there it is.
ajmurmann 5y ago
IMO it's a huge problem that TDD and unit testing hey conflated way too often. TDD just means you write the test first and don't write code not required to get the tests to pass. It does NOT tell you what tests you should write. I for one am a huge believer in inside-out TDD. This would have you start with an acceptance test that uses the product as a user would. This may be exercise the feature or reproduce the bug. It's for you to decide what the right abstraction level is. I typically will end up testing edge cases either one level down in integration tests or unit tests, but it depends and there are no hard rules. It's all a trade-off. What's definitely wrong to me is retiring solely on unit tests for your TDD that's a disaster waiting to happen. TDD is about fast feedback loops and quality not about proxy metrics like coverage from unit tests. As others have pointed out, depending on your language you also might need more or fewer tests. When I write Ruby or plain JS, I'll write a lot, a lot of tests on different levels. When I'm writing Rust, I write very few and those that I do write will be focused on complex business logic (granted all Rust code I've written was for toy projects or puzzles where "business" logic might be the wrong term) Edit: we've also corrupted what "unit" is referring to. AFAIK it originally referred to a unit of functionality. Many now take it to refer to a single class.
danielvaughn 5y ago
This describes my experience with unit testing perfectly. What I try to do is just write small functions that feel like they would be well suited for a unit test. I find that makes them much easier to read, and reason about. I don’t even bother writing tests.
rbanffy 5y ago
You are forgetting one important part: unit tests serve as an always updated, always tested documentation of how you use the functionalities of your software. I agree with your point on 3 - you should use the tests to start figuring out how your unit is going to be used. If you write code first, you frequently end up with untestable code that you need to mock the universe to get it running in artificial conditions. I don't recommend writing your tests first unless it's a trivial case, but you should at least have the unit calls in tests before you start coding.
- oxff 5y ago
  This has been my interpretation of unit testing for some years now. They should be renamed to "Examples" or "Documentation Test" because all they really do is when I read them is (i) show how to use the unit (ii) and common error / success cases of that unit. I would never really on them for actually testing the expected property of a public API for example.
  - rbanffy 5y ago
    Depends on how extensive the unit tests are. But, again, you also need to do integration tests for making sure you are not getting anything funny from external sources you rely on.

jrockway 5y ago

100% test coverage is really the starting point for testing. Once all the lines of code you've typed in actually run before they're delivered to the customer, then you can start adding the real test scenarios!

hinkley 5y ago
And how, pray tell, do you discover all of the scenarios that need to be covered? If only there were some sort of coverage tool that could tell you...

ravenstine 5y ago

That's the thing that pisses me off about how testing and the periphery around testing (like coverage) are commonly treated. They're seen as being virtuous in and of themselves and no one wants to measure whether they are counterproductive. Even if one took that initiative, good luck getting anyone to care. Sure, I suppose one could say that any tests is better than no tests, but that also dismisses whether testing can be done better. Code coverage is the worst. I've worked with codebases with various degrees of code coverage, and I've never seen a correlation between coverage and the number of bugs encountered. There may be an asymptotic nature to how much coverage is needed, but I would bet the threshold is low in a general sense. Once you've reached even 25% coverage, it's really dubious whether more coverage than that is actually telling you anything. Of course, we could measure success by number of bugs and company revenue, but no, we can't do that; those could make people look bad.

geoffjentry 5y ago
I've become a fan of using mutation testing to drive unit test coverage. The goal of mutation testing is to ensure there is test coverage around areas with a higher likelihood of being brittle. Work smarter, not harder.
lytefm 5y ago
Worse still, I feel like doing unit testing at the expense of integration testing can create a false sense of security. Unit tests often skip the error cases that are really important for real life scenarios. We've recently introduced a regression test suite for our data pipeline which simply compares all the data before with the data after running the pipeline. This gave us about 65% code coverage and the certainty that we're testing what actually matters for the product instead of how somebody decides to implement a function.
- voxic11 5y ago
  I feel the opposite way, with unit tests you can test failure scenarios as easily as any other scenario. So it makes it simple to exhaustively test every code path including failure scenarios that would be very difficult to set up in integration tests.
  - mywittyname 5y ago
    There are bugs that don't appear as such at the unit level. Two objects might individually be in perfectly valid states, but those states can be invalid at a system level. Which is why there can be a false sense of security with unit testing. When you see 100% code coverage, it implies that you've tested every possible scenario, but with most non-trivial software, you haven't. Did you test what happens when the methods are passed in 100GB of data?
- johndfsgdgdfg 5y ago
  False sense of security sold by conference indsutrial complex is my biggest issues with unit testing. One time I've seen a junior dev wrote copious amount of unit tests to achieve near 100% code coverage. When this "perfect" code went into prod it almost took the whole system down. Because the junior dev was joining close to a dozen tables and each one of the tables has at least a million rows. If the junior dev took the time to understand basic DB performance instead of writing dozens of unit tests, then it wouldn't have happened. I blame the gurus who don't write code for living, but peddle magic solutions like "clean code", unit testing etc. to sell more books and seminars.

gizmo686 5y ago

I think this approach comes out of highly dynamic languages like python. If a typo in a variable name doesn't cause any problems until it get evaluated, then you really need to execute every piece of the code to be sure that you don't have incredibly simple errors. If you are using languages with a even a primitive static type system, you can catch a lot of errors at compile time (or when the interprartor loads them), which provides most of the benifits of 100% coverage.

theonething 5y ago
for undeclared variables PHP is even worse. It just keeps going and just gives you a notice!
- rbanffy 5y ago
  PHP has a lot in common with BASIC at that.
nicoburns 5y ago
> If a typo in a variable name doesn't cause any problems until it get evaluated Linters even in languages like python will catch that kind of thing.

stouset 5y ago

Also, this kind of mentality frequently results in what I consider negative-value tests. A test suite is supposed to do two things: a) identify bugs by failing on incorrect behavior, and b) enable refactoring by continuing to pass when underlying code doesn't change behavior Focusing on 100% unit test coverage all the time—particularly in code that isn't doing simple, single responsibility stuff—often results (in my experience) in tests that overly mock other parts of the system. The net result is that you end up writing tests that simple check that "the code was written the way it is currently written". Tests like this fail on the first count because they fundamentally cannot detect incorrect behavior, and they fail on the second count because a non-trivial refactor inevitably changes the implementation even if the resulting behavior stays the same. Not that mocks aren't useful in some scenarios, but if you find yourself writing entire test suites based on them you're likely creating tests that are negative value. They won't catch undiscovered bugs, and they'll give you bad information when you're refactoring.

lichtenberger 5y ago
That's precisely what I did in the beginning of my career. To test every method and mock all the dependencies. This had the impact, because we usually don't use a dependency injection framework, that I had to add an interface for almost every class, adding factories with interfaces almost everywhere and then write tests which expect certain method calls and to basically debug the tests while writing them until all calls have been mocked. Thus, whenever I changed the production code I had to change the test as well. IMHO in my own projects I tend to write tests which test certain parts of a system as well as some dependencies. So usually I'd say they are integration tests instead of unit tests. But I can refactor stuff and once I break something it's usually detected by one of the tests. Maybe unit tests should test for certain cases border cases, which are hard to test with integration tests.
seanhunter 5y ago
I heartily agree with your general sentiment, but think you're not mentioning a very important function of a test suite which is: c) explicitly and accurately document expected behavior including in a variety of edge cases ...this is sort of implicit in your first 2 but I think worth breaking out. Going for high test coverage (in places where I've done TDD-style dev) has really forced me to think through those edge cases and what I want my code to do in all of them. I think negative-value testing comes from the mindset of "how do I write a test which exercises this branch?" but a positive version of that is "I put this branch to handle the case where foo is getting close to the boundary. Let's put in test cases with foo at and close to the boundary and make sure it does what it is supposed to." If the test focuses on behavior then you find the bugs and enable refactoring and you get high coverage because if you can't exercise all the paths probably some of them are dead and can just get deleted.

avl999 5y ago

Does your team work on a codebase that uses a dynamically typed language? If so then the 100% coverage requirement is perfectly reasonable and arguably the only sane way to develop anything more than a tiny project. If a statically typed language is being used then sure, that requirement is counterproductive but if a team uses a dynamically typed language I don't see a way to have any reasonable amount of confidence that the code is even syntactically and semantically valid without 100% coverage. The 100% coverage is just a tax that you need to pay in dynamically typed codebases (hence my preference to use statically typed language for any reasonably sized work related project). For 3) TDD has nothing to with test coverage or aiming for high coverage. The fact that TDD evangelists have tried to couple "good testing" with TDD is a huge scam. I don't care when you write your tests as long as they get written before the code is merged in. TDD is good for parts of the code which are very pure or where you know exactly what you are doing, but if what you are doing is more exploratory in nature or is not pure and has lots of external dependencies TDD is simply not practical for a lot of people.

cbhl 5y ago
In my experience, scaling a large project in a dynamically typed language with 100% unit test coverage just gets you production outages with all unit tests passing. A reasonable threshold is probably closer to 70-90% _plus integration tests_. Hopefully you also picked a language where a bigco has created tooling that lets you incrementally add static types -- TypeScript, Closure Compiler, Python typing and/or pytype, etc.
- lenkite 5y ago
  Lol, this is so true. A project that I had joined (late) some years ago had 98%! code coverage, full sonar/fortify code compliance and was an utter piece of garbage that had regular outages in production. It taught me a couple of lessons: 1. Integration and System Tests are FAR more valuable than unit tests. By several orders of magnitude. 2. Performance Tests are vital. This is actually more true in a cloud world. There is only so much hardware you can throw at a problem before your costs are un-feasible.
cush 5y ago

vmception 5y ago

I like this post, because I avoided unit tests for so long lol. After finally doing them, I did notice that I would at least format my code better and this had other benefits. There would be cases I missed, made me start thinking about edge cases further in advance, as well as made it easier for other to read. I think the primary point is made > Or just write simple, single responsibility code so it can be tested easily there it is.

ajmurmann 5y ago

IMO it's a huge problem that TDD and unit testing hey conflated way too often. TDD just means you write the test first and don't write code not required to get the tests to pass. It does NOT tell you what tests you should write. I for one am a huge believer in inside-out TDD. This would have you start with an acceptance test that uses the product as a user would. This may be exercise the feature or reproduce the bug. It's for you to decide what the right abstraction level is. I typically will end up testing edge cases either one level down in integration tests or unit tests, but it depends and there are no hard rules. It's all a trade-off. What's definitely wrong to me is retiring solely on unit tests for your TDD that's a disaster waiting to happen. TDD is about fast feedback loops and quality not about proxy metrics like coverage from unit tests. As others have pointed out, depending on your language you also might need more or fewer tests. When I write Ruby or plain JS, I'll write a lot, a lot of tests on different levels. When I'm writing Rust, I write very few and those that I do write will be focused on complex business logic (granted all Rust code I've written was for toy projects or puzzles where "business" logic might be the wrong term) Edit: we've also corrupted what "unit" is referring to. AFAIK it originally referred to a unit of functionality. Many now take it to refer to a single class.

danielvaughn 5y ago

This describes my experience with unit testing perfectly. What I try to do is just write small functions that feel like they would be well suited for a unit test. I find that makes them much easier to read, and reason about. I don’t even bother writing tests.

rbanffy 5y ago

You are forgetting one important part: unit tests serve as an always updated, always tested documentation of how you use the functionalities of your software. I agree with your point on 3 - you should use the tests to start figuring out how your unit is going to be used. If you write code first, you frequently end up with untestable code that you need to mock the universe to get it running in artificial conditions. I don't recommend writing your tests first unless it's a trivial case, but you should at least have the unit calls in tests before you start coding.

oxff 5y ago
This has been my interpretation of unit testing for some years now. They should be renamed to "Examples" or "Documentation Test" because all they really do is when I read them is (i) show how to use the unit (ii) and common error / success cases of that unit. I would never really on them for actually testing the expected property of a public API for example.
- rbanffy 5y ago
  Depends on how extensive the unit tests are. But, again, you also need to do integration tests for making sure you are not getting anything funny from external sources you rely on.