4 ms·

There are entirely separate communities IMO when it comes to automated tests. There are the pushing-the-edge-of-excellence folks, constantly refining methods a

There are entirely separate communities IMO when it comes to automated tests. There are the pushing-the-edge-of-excellence folks, constantly refining methods and approaches, who tend to be passionate about the holistic benefits of testing. And there are the folks who write convoluted tests that, when you dig into them, just confirm that String's .equals() works OK. DAO/database tests which have mocking going on to the point where you're faking a database response of "hello", say, and then checking "expected response == hello". Then yay our test passes!! Personally I feel our energy should somehow be spent collectively on getting the latter group to just not write total garbage, and that the bar for "good enough" is really not that high, after which we get into diminishing-returns territory. And I think, personally, that mocking is responsible for so much confusion in folks that ultimately don't really know what they're trying to accomplish.

XorNot 3y ago
It's all about staying in language though, at least in my opinion. The test suite I don't run is the one that requires the complicated docker compose stack and the local network to look just right. The test suite I do run is the one where I hit "run" in my IDE and then it lets be jump to what failed and debug seconds later. That, IMO, is what mocking is about: making the tests fast and easy to run so they actually are.
- brianmcc 3y ago
  Agree. Mocking is a slippery slope I think. We need to mock, for example, DownstreamAPI01 that our app POSTS to and GETs from, so we can put in accurate and realistic faked responses. We don't strictly need to mock other bits of our own apps - but many people choose to do so. Martin Fowler writes very well about this, and I hadn't even appreciated the "split" between sociable unit tests and solitary tests: https://martinfowler.com/bliki/UnitTest.html https://martinfowler.com/bliki/UnitTest.html I can see the appeal of both sides, but (a) I prefer sociable tests as they tend to test real code more than artificial constructs, and (b) people tend to get into a mess when they try to produce pure solitary tests, leading to the "not really testing your app" example in my earlier comment.
  - StackOverlord 3y ago
    95% of unit tests will bring 5% of value and 5% of unit tests will bring 95% of value. Most unit tests should be handled at the integration level.
- MoreQARespect 3y ago
  The compose stack tests (when engineered right), are the only ones which can reliably reproduce and test almost any bug or feature reliably. In 2018 I could understand the aversion to them given how unbearably slow and flaky they could be but those problems are draining away with improved tooling and faster machines. I think before long they'll be the default - a test that is 4 seconds slower and 30% more realistic will seem like a no brainer.
- time0ut 3y ago
  I have both. 90% of my tests are normal unit tests that use mocks or fakes as needed to test the things I care about. 9% are integration type tests that validate broad behaviors with my app’s direct dependencies in an ephemeral environment managed via docker compose (usually anyway). 1% is a couple end to end tests that validate the whole system at deploy time and beyond.
sethammons 3y ago
I cannot count the number of python and perl tests leveraging mocks that we inherited that after we dug into them, there tests were asserting that 1=1 exactly as you say. "Make the db return hello, insure we returned hello; yay, framework works?". Unit tests should validate error paths are properly exercised and that we handle expected results appropriately. It is wild how often this is not expressed in unit tests.
- TeMPOraL 3y ago
  On the other hand, I've been surprised just how often a simple "1=1 in a roundabout way" sanity test explodes in my face. The code people write is just that shit. And so is, often, my understanding of it. So I always like to keep around tests confirming we can shuffle some data back and forth on the "golden path" - they're good documentation, good at detecting broken refactors, ad well... if you're going to point out that a test effectively checks if String.equals() works OK - sure, but that's an implementation detail, to which the test is oblivious, and you should be too.
TillE 3y ago
Tests are often extremely helpful, but I think the goal of 100% coverage has done much more harm than good, and leads to the kind of trivial tests you describe. TDD usually works well, and helps guide you towards designing a usable, testable API. Don't test getters and setters, test the actual logic you care about.

XorNot 3y ago

It's all about staying in language though, at least in my opinion. The test suite I don't run is the one that requires the complicated docker compose stack and the local network to look just right. The test suite I do run is the one where I hit "run" in my IDE and then it lets be jump to what failed and debug seconds later. That, IMO, is what mocking is about: making the tests fast and easy to run so they actually are.

brianmcc 3y ago
Agree. Mocking is a slippery slope I think. We need to mock, for example, DownstreamAPI01 that our app POSTS to and GETs from, so we can put in accurate and realistic faked responses. We don't strictly need to mock other bits of our own apps - but many people choose to do so. Martin Fowler writes very well about this, and I hadn't even appreciated the "split" between sociable unit tests and solitary tests: https://martinfowler.com/bliki/UnitTest.html https://martinfowler.com/bliki/UnitTest.html I can see the appeal of both sides, but (a) I prefer sociable tests as they tend to test real code more than artificial constructs, and (b) people tend to get into a mess when they try to produce pure solitary tests, leading to the "not really testing your app" example in my earlier comment.
- StackOverlord 3y ago
  95% of unit tests will bring 5% of value and 5% of unit tests will bring 95% of value. Most unit tests should be handled at the integration level.
MoreQARespect 3y ago
The compose stack tests (when engineered right), are the only ones which can reliably reproduce and test almost any bug or feature reliably. In 2018 I could understand the aversion to them given how unbearably slow and flaky they could be but those problems are draining away with improved tooling and faster machines. I think before long they'll be the default - a test that is 4 seconds slower and 30% more realistic will seem like a no brainer.
time0ut 3y ago
I have both. 90% of my tests are normal unit tests that use mocks or fakes as needed to test the things I care about. 9% are integration type tests that validate broad behaviors with my app’s direct dependencies in an ephemeral environment managed via docker compose (usually anyway). 1% is a couple end to end tests that validate the whole system at deploy time and beyond.

sethammons 3y ago

I cannot count the number of python and perl tests leveraging mocks that we inherited that after we dug into them, there tests were asserting that 1=1 exactly as you say. "Make the db return hello, insure we returned hello; yay, framework works?". Unit tests should validate error paths are properly exercised and that we handle expected results appropriately. It is wild how often this is not expressed in unit tests.

TeMPOraL 3y ago
On the other hand, I've been surprised just how often a simple "1=1 in a roundabout way" sanity test explodes in my face. The code people write is just that shit. And so is, often, my understanding of it. So I always like to keep around tests confirming we can shuffle some data back and forth on the "golden path" - they're good documentation, good at detecting broken refactors, ad well... if you're going to point out that a test effectively checks if String.equals() works OK - sure, but that's an implementation detail, to which the test is oblivious, and you should be too.

TillE 3y ago

Tests are often extremely helpful, but I think the goal of 100% coverage has done much more harm than good, and leads to the kind of trivial tests you describe. TDD usually works well, and helps guide you towards designing a usable, testable API. Don't test getters and setters, test the actual logic you care about.