3 ms·

1) At least in the Java world, the term "unit testing" is often confused by "things you do in JUnit", which runs both "pure" unit tests and project-level integr

1) At least in the Java world, the term "unit testing" is often confused by "things you do in JUnit", which runs both "pure" unit tests and project-level integration tests, i.e. spinning up an application context (like Spring) and testing against real REST endpoints etc. 2) While unit tests are cheaper and quicker than (project-level) integration tests, they also in many cases don't provide results as good a result and level of confidence, because a lot of run-time aspects (serialization, HTTP responses, database responses, etc.) are not as straightforward to mock. There's been some noise about The Testing Trophy, instead of the Testing Pyramid where, in short, there are still unit tests where it makes sense, but a lot of testing has moved to the (project-level) integration tests. These are slower, but only by so much that the trade-off is often worth it. Whether it's worth it, depends heavily on what you're testing. If it's a CRUD API: I use integration tests. If it's something algorithmic, or string manipulation, etc.: I use unit tests. When I saw the Testing Trophy presented, it came with the asterisk that (project-level) integration testing has gotten easier and cheaper over time, thus allowing a shift in trade-off. Testcontainers is one of the primary reasons why this shift has happened. (And... I respect that it's not for everyone.) Some references: https://kentcdodds.com/blog/the-testing-trophy-and-testing-classifications https://kentcdodds.com/blog/the-testing-trophy-and-testing-c... https://www.youtube.com/watch?v=t-HmXomntCA https://www.youtube.com/watch?v=t-HmXomntCA

ants_everywhere 3y ago
Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive tests. But on the test philosophy angle, my take on what's happening is just that developers traditionally look for any reason to skip tests. I've seen this in a few different forms. - right now containers make it trivial to run all of your dependencies. That's much easier than creating a mock or a fake, so we do that and don't bother creating a mock/fake. - compiler folks have created great static analysis tools. That's easier than writing a bunch of tests, so we'll just assume static analysis will catch our bugs for us. - <my language>'s types system does a bunch of work type checking, so I don't need tests. Or maybe I just need randomly generated property tests. - no tests can sufficiently emulate our production environment, so tests are noise and we'll work out issues in dev and prod. What I've noticed, though, is that looking across a wide number of software projects is there's a clear difference in quality between projects that have a strong testing discipline and those that convince themselves they don't need tests because of <containers, or types, or whatever else>. Sure it's possible that tests don't cause the quality difference (maybe there's a third factor for example that causes both). And of course if you have limited resources you have to make a decision about which quality assurance steps to cut. But personally I respect a project more if they just say they don't have the bandwidth to test properly so they're just skipping to the integration stage (or whatever) rather than convince themselves that those tests weren't important any way. Because I've seen so many projects that would have been much better with even a small number of unit tests where they only had integration tests.
- lmm 3y ago
  Sounds like some kind of protestant work ethic mentality: testing should be hard work, the harder writing your tests was the better your soul and the better your system. I've seen plenty of projects that made oodles of mocks and fakes and unit tests and just sucked, outright didn't work at all in a way that would've been obvious if they'd done testcontainers-based integration tests or even just manual testing in prod. I would absolutely trust a project that was written in Haskell and had no tests, or only integration tests, ahead of one that had lots of unit tests. Indeed if anything I'd say the number of mocks/fakes is negatively correlated with the actual project quality.
  - sverhagen 3y ago
    Just to add, there's also a (Chicago) school of thought that pushes back against mocks and fakes, so even if you're religiously (to stick with the metaphore) writing unit tests, you might still not invest in mocks and fakes.
    - Mavvie 3y ago
      Can you (or someone else) explain what the alternatives are? How can I write unit tests without mocks or fakes?
      - hyperadvanced 3y ago
        They might mean that rather than using a mock, use a real typed object/instance of a real thing and inject it into the unit that you’re testing. Admittedly, that might meet the definition of a fake/mock once you get down to the level of testing something that needs db access. Another way of interpreting that is that you can use in memory versions of your deps to mirror the interface of your dependency without needing to repeatedly, and possibly haphazardly mock certain functions of your dependency.
- sverhagen 3y ago
  I think testing is very important. But it's very hard to test everything. (It is not hard to get 100% test coverage by some metric, but that does not mean that all scenarios or even the most useful ones are covered.) So it's an economics game: how can you get the most value for the least amount of money? Or if you want me to rephrase that in a more-positive way: how can you get the most value out of the time that you have available? And I contend that a shift "up" in the pyramid (at which time it looses that shape, hence the "testing trophy") is where the current sweet spot lies. You have to use the tools that you have.

ants_everywhere 3y ago

Yeah, certainly I see the value of those kinds of tests. And clearly as you say the simpler tests don't provide as realistic a simulation as the more expensive tests. But on the test philosophy angle, my take on what's happening is just that developers traditionally look for any reason to skip tests. I've seen this in a few different forms. - right now containers make it trivial to run all of your dependencies. That's much easier than creating a mock or a fake, so we do that and don't bother creating a mock/fake. - compiler folks have created great static analysis tools. That's easier than writing a bunch of tests, so we'll just assume static analysis will catch our bugs for us. - <my language>'s types system does a bunch of work type checking, so I don't need tests. Or maybe I just need randomly generated property tests. - no tests can sufficiently emulate our production environment, so tests are noise and we'll work out issues in dev and prod. What I've noticed, though, is that looking across a wide number of software projects is there's a clear difference in quality between projects that have a strong testing discipline and those that convince themselves they don't need tests because of <containers, or types, or whatever else>. Sure it's possible that tests don't cause the quality difference (maybe there's a third factor for example that causes both). And of course if you have limited resources you have to make a decision about which quality assurance steps to cut. But personally I respect a project more if they just say they don't have the bandwidth to test properly so they're just skipping to the integration stage (or whatever) rather than convince themselves that those tests weren't important any way. Because I've seen so many projects that would have been much better with even a small number of unit tests where they only had integration tests.

lmm 3y ago
Sounds like some kind of protestant work ethic mentality: testing should be hard work, the harder writing your tests was the better your soul and the better your system. I've seen plenty of projects that made oodles of mocks and fakes and unit tests and just sucked, outright didn't work at all in a way that would've been obvious if they'd done testcontainers-based integration tests or even just manual testing in prod. I would absolutely trust a project that was written in Haskell and had no tests, or only integration tests, ahead of one that had lots of unit tests. Indeed if anything I'd say the number of mocks/fakes is negatively correlated with the actual project quality.
- sverhagen 3y ago
  Just to add, there's also a (Chicago) school of thought that pushes back against mocks and fakes, so even if you're religiously (to stick with the metaphore) writing unit tests, you might still not invest in mocks and fakes.
  - Mavvie 3y ago
    Can you (or someone else) explain what the alternatives are? How can I write unit tests without mocks or fakes?
    - hyperadvanced 3y ago
      They might mean that rather than using a mock, use a real typed object/instance of a real thing and inject it into the unit that you’re testing. Admittedly, that might meet the definition of a fake/mock once you get down to the level of testing something that needs db access. Another way of interpreting that is that you can use in memory versions of your deps to mirror the interface of your dependency without needing to repeatedly, and possibly haphazardly mock certain functions of your dependency.
sverhagen 3y ago
I think testing is very important. But it's very hard to test everything. (It is not hard to get 100% test coverage by some metric, but that does not mean that all scenarios or even the most useful ones are covered.) So it's an economics game: how can you get the most value for the least amount of money? Or if you want me to rephrase that in a more-positive way: how can you get the most value out of the time that you have available? And I contend that a shift "up" in the pyramid (at which time it looses that shape, hence the "testing trophy") is where the current sweet spot lies. You have to use the tools that you have.