3 ms·

I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. Sure, integration tests "save" you from writing pesky unit

by imiric 3y ago

I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. Sure, integration tests "save" you from writing pesky unit tests, and changing them frequently after every refactor. But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. Unit tests give you that ability. If written correctly, they should be the first to fail (which is a good thing!), and if an integration test fails, it should ideally also be accompanied by at least one unit test failure. This way it immediately pinpoints the root cause. The higher up the stack you test, the harder it is to debug. With E2E tests you're essentially debugging the entire system, which is why we don't exclusively write E2E tests, even though they're very useful. To me the traditional test pyramid is still the best way to think about tests. Tests shouldn't be an afterthought or a chore. Maintaining a comprehensive and effective test suite takes as much hard work as, if not more than, maintaining the application itself, and it should test all layers of the system. But if you do have that, it gives you superpowers to safely and reliably work on any part of the system.

hamandcheese 3y ago
> But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. As long as the error is reproducible, never in my career have I had a hard time locating the source of the error. Bisection does wonders (as a general concept, not specifically referring to git bisect). That said, I have encountered plenty of non-reproducible test failures. Moral of the story: make things reproducible, especially tests. Easier said than done.
mk89 3y ago
> I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. It's very simple: most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it. The irony is that the pieces that really count don't get tested properly, because you unit-test the happy-path and maybe 1 or 2 negative scenarios, and that's it, missing out a bunch of potential regressions. EDIT: This just to say that I don't believe the author of the comment said "don't write unit tests" (I hope not, at least!) but, if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs". Which would be partially true also with properly written unit tests (and they would do so in a fraction of the time you need with integration tests).
- imiric 3y ago
  > most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it So strict rules from management in a company that likely doesn't understand software development, and lazy developers who decide to ignore this by intentionally writing useless tests, lead to thinking that unit tests and coverage are useless? That doesn't track at all. I'd say that the answer is somewhere in the middle. If the company doesn't understand software development, it's the engineer's job to educate them, or find a better place to work at. It's also the engineer's job to educate lazy developers to care about testing and metrics like code coverage. > if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs" And unit tests don't? I would argue that unit tests give you much more of that dopamine, since you see the failures and passes much more quickly, and there should be much more of them overall. Not that we should structure our work towards chasing dopamine hits... I'd say that most of the people who advocate for this position haven't worked with a well tested codebase. Sadly, not all of us have the privilege of working with codebases like SQLite's, which go much beyond 100% line/statement coverage[1]. Is all that work in vain? Are they some crazy dogmatic programmers that like wasting their time? I would say: no. They just put a lot of effort and care in their product, which speaks for itself, and I would think makes working on it much safer, more efficient and pleasant. I would also argue that the current state of our industry, and in turn everything that depends on software, where buggy software is the norm would be much better overall if that kind of effort and care would be put in all software projects. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
  - mk89 3y ago
    > Sadly, not all of us have the privilege of working with codebases like SQLite's, which go much beyond 100% line/statement coverage[1]... Linked SQLite page mentions this: > 100% branch test coverage in an as-deployed configuration Branch test coverage is different from line coverage and in my opinion it should be the only metric used in this context for coverage. 90-95% line coverage is exactly why many unit tests are garbage and why many come up with the argument "I prefer integration tests, unit tests are not that useful".
ric2b 3y ago
If the test fails consistently (as it should) it is usually just a question of using a debugger and stepping through some suspect sections of the code to find the issue. Compared to the amount of time saved by not rewriting unit tests every time you refactor stuff, it's a great trade-off.
JackFr 3y ago
> Sure, integration tests "save" you from writing pesky unit tests, and changing them frequently after every refactor. The DB migration has "ADD COLUMN rateLimit INT" The application class member is annotated with "@Column(name=”ratelimit”, nullable=true)" The failure is at the interface between the app and the DB. What testcontainers does is allow you to write a quasi-unit test (not a truly full-blown integration test, but testing a small piece of functionality) across the boundary of two components. I am not aware of a way to reasonably unit test for this error. That might just be me -- seriously if there is a tried strategy for unit testing things like this I'd love to know it.
- imiric 3y ago
  That's an _integration_ failure. We're talking about testing two entirely different things here. Of course you shouldn't expect to test integration scenarios in unit tests. But you also shouldn't expect to test all unit scenarios in integration tests. This is what the "trophy testing" model advocates for. That somehow unit tests are not needed if you have thorough integration tests, which is entirely false. They test the application at different layers, because they're meant to ensure behavior to different types of users. They're both useful for catching bugs and unexpected behavior, and they're meant to complement each other.

by imiric 3y ago

hamandcheese 3y ago

> But how do you quickly locate the reason that integration test failed? There could be hundreds of moving parts involved, and any one of them malfunctioning, or any unexpected interaction between them, could cause it to fail. The error itself would likely not be clear enough, if it's covered by layers of indirection. As long as the error is reproducible, never in my career have I had a hard time locating the source of the error. Bisection does wonders (as a general concept, not specifically referring to git bisect). That said, I have encountered plenty of non-reproducible test failures. Moral of the story: make things reproducible, especially tests. Easier said than done.

mk89 3y ago

> I've heard this aversion to unit tests a few times in my career, and I'm unable to make sense of it. It's very simple: most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it. The irony is that the pieces that really count don't get tested properly, because you unit-test the happy-path and maybe 1 or 2 negative scenarios, and that's it, missing out a bunch of potential regressions. EDIT: This just to say that I don't believe the author of the comment said "don't write unit tests" (I hope not, at least!) but, if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs". Which would be partially true also with properly written unit tests (and they would do so in a fraction of the time you need with integration tests).

imiric 3y ago
> most of the time people are told by management that they MUST achieve a 80-90-95% of code coverage (with unit tests), which leads to a lot of absolutely worthless tests - tests for the sake of it So strict rules from management in a company that likely doesn't understand software development, and lazy developers who decide to ignore this by intentionally writing useless tests, lead to thinking that unit tests and coverage are useless? That doesn't track at all. I'd say that the answer is somewhere in the middle. If the company doesn't understand software development, it's the engineer's job to educate them, or find a better place to work at. It's also the engineer's job to educate lazy developers to care about testing and metrics like code coverage. > if I can rephrase it, "well, the integration tests give you a better dopamine effect because they actually help you catch bugs" And unit tests don't? I would argue that unit tests give you much more of that dopamine, since you see the failures and passes much more quickly, and there should be much more of them overall. Not that we should structure our work towards chasing dopamine hits... I'd say that most of the people who advocate for this position haven't worked with a well tested codebase. Sadly, not all of us have the privilege of working with codebases like SQLite's, which go much beyond 100% line/statement coverage[1]. Is all that work in vain? Are they some crazy dogmatic programmers that like wasting their time? I would say: no. They just put a lot of effort and care in their product, which speaks for itself, and I would think makes working on it much safer, more efficient and pleasant. I would also argue that the current state of our industry, and in turn everything that depends on software, where buggy software is the norm would be much better overall if that kind of effort and care would be put in all software projects. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- mk89 3y ago
  > Sadly, not all of us have the privilege of working with codebases like SQLite's, which go much beyond 100% line/statement coverage[1]... Linked SQLite page mentions this: > 100% branch test coverage in an as-deployed configuration Branch test coverage is different from line coverage and in my opinion it should be the only metric used in this context for coverage. 90-95% line coverage is exactly why many unit tests are garbage and why many come up with the argument "I prefer integration tests, unit tests are not that useful".

ric2b 3y ago

If the test fails consistently (as it should) it is usually just a question of using a debugger and stepping through some suspect sections of the code to find the issue. Compared to the amount of time saved by not rewriting unit tests every time you refactor stuff, it's a great trade-off.

JackFr 3y ago

> Sure, integration tests "save" you from writing pesky unit tests, and changing them frequently after every refactor. The DB migration has "ADD COLUMN rateLimit INT" The application class member is annotated with "@Column(name=”ratelimit”, nullable=true)" The failure is at the interface between the app and the DB. What testcontainers does is allow you to write a quasi-unit test (not a truly full-blown integration test, but testing a small piece of functionality) across the boundary of two components. I am not aware of a way to reasonably unit test for this error. That might just be me -- seriously if there is a tried strategy for unit testing things like this I'd love to know it.

imiric 3y ago
That's an _integration_ failure. We're talking about testing two entirely different things here. Of course you shouldn't expect to test integration scenarios in unit tests. But you also shouldn't expect to test all unit scenarios in integration tests. This is what the "trophy testing" model advocates for. That somehow unit tests are not needed if you have thorough integration tests, which is entirely false. They test the application at different layers, because they're meant to ensure behavior to different types of users. They're both useful for catching bugs and unexpected behavior, and they're meant to complement each other.