3 ms·

Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive

Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive tests. But on the test philosophy angle, my take on what's happening is just that developers traditionally look for any reason to skip tests. I've seen this in a few different forms. - right now containers make it trivial to run all of your dependencies. That's much easier than creating a mock or a fake, so we do that and don't bother creating a mock/fake. - compiler folks have created great static analysis tools. That's easier than writing a bunch of tests, so we'll just assume static analysis will catch our bugs for us. - <my language>'s types system does a bunch of work type checking, so I don't need tests. Or maybe I just need randomly generated property tests. - no tests can sufficiently emulate our production environment, so tests are noise and we'll work out issues in dev and prod. What I've noticed, though, is that looking across a wide number of software projects is there's a clear difference in quality between projects that have a strong testing discipline and those that convince themselves they don't need tests because of <containers, or types, or whatever else>. Sure it's possible that tests don't cause the quality difference (maybe there's a third factor for example that causes both). And of course if you have limited resources you have to make a decision about which quality assurance steps to cut. But personally I respect a project more if they just say they don't have the bandwidth to test properly so they're just skipping to the integration stage (or whatever) rather than convince themselves that those tests weren't important any way. Because I've seen so many projects that would have been much better with even a small number of unit tests where they only had integration tests.

lmm 3y ago
Sounds like some kind of protestant work ethic mentality: testing should be hard work, the harder writing your tests was the better your soul and the better your system. I've seen plenty of projects that made oodles of mocks and fakes and unit tests and just sucked, outright didn't work at all in a way that would've been obvious if they'd done testcontainers-based integration tests or even just manual testing in prod. I would absolutely trust a project that was written in Haskell and had no tests, or only integration tests, ahead of one that had lots of unit tests. Indeed if anything I'd say the number of mocks/fakes is negatively correlated with the actual project quality.
- sverhagen 3y ago
  Just to add, there's also a (Chicago) school of thought that pushes back against mocks and fakes, so even if you're religiously (to stick with the metaphore) writing unit tests, you might still not invest in mocks and fakes.
  - Mavvie 3y ago
    Can you (or someone else) explain what the alternatives are? How can I write unit tests without mocks or fakes?
    - hyperadvanced 3y ago
      They might mean that rather than using a mock, use a real typed object/instance of a real thing and inject it into the unit that you’re testing. Admittedly, that might meet the definition of a fake/mock once you get down to the level of testing something that needs db access. Another way of interpreting that is that you can use in memory versions of your deps to mirror the interface of your dependency without needing to repeatedly, and possibly haphazardly mock certain functions of your dependency.
sverhagen 3y ago
I think testing is very important. But it's very hard to test everything. (It is not hard to get 100% test coverage by some metric, but that does not mean that all scenarios or even the most useful ones are covered.) So it's an economics game: how can you get the most value for the least amount of money? Or if you want me to rephrase that in a more-positive way: how can you get the most value out of the time that you have available? And I contend that a shift "up" in the pyramid (at which time it looses that shape, hence the "testing trophy") is where the current sweet spot lies. You have to use the tools that you have.

lmm 3y ago

Sounds like some kind of protestant work ethic mentality: testing should be hard work, the harder writing your tests was the better your soul and the better your system. I've seen plenty of projects that made oodles of mocks and fakes and unit tests and just sucked, outright didn't work at all in a way that would've been obvious if they'd done testcontainers-based integration tests or even just manual testing in prod. I would absolutely trust a project that was written in Haskell and had no tests, or only integration tests, ahead of one that had lots of unit tests. Indeed if anything I'd say the number of mocks/fakes is negatively correlated with the actual project quality.

sverhagen 3y ago
Just to add, there's also a (Chicago) school of thought that pushes back against mocks and fakes, so even if you're religiously (to stick with the metaphore) writing unit tests, you might still not invest in mocks and fakes.
- Mavvie 3y ago
  Can you (or someone else) explain what the alternatives are? How can I write unit tests without mocks or fakes?
  - hyperadvanced 3y ago
    They might mean that rather than using a mock, use a real typed object/instance of a real thing and inject it into the unit that you’re testing. Admittedly, that might meet the definition of a fake/mock once you get down to the level of testing something that needs db access. Another way of interpreting that is that you can use in memory versions of your deps to mirror the interface of your dependency without needing to repeatedly, and possibly haphazardly mock certain functions of your dependency.

sverhagen 3y ago

I think testing is very important. But it's very hard to test everything. (It is not hard to get 100% test coverage by some metric, but that does not mean that all scenarios or even the most useful ones are covered.) So it's an economics game: how can you get the most value for the least amount of money? Or if you want me to rephrase that in a more-positive way: how can you get the most value out of the time that you have available? And I contend that a shift "up" in the pyramid (at which time it looses that shape, hence the "testing trophy") is where the current sweet spot lies. You have to use the tools that you have.