3 ms·

> Most of the systems I build use a database on which all logic depends, and often a network connection. Why do you believe this is relevant? Unit tests target

> Most of the systems I build use a database on which all logic depends, and often a network connection. Why do you believe this is relevant? Unit tests target units of code, and check if the unit of code verifies specific invariants for specific combinations and ranges of input values. The concept of a network or a database does not exist in unit tests. In fact, you are expected to abstract away these aspects. > I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. This scenario is totally unrealistic if you knew what you are doing. The fact alone that you're talking about spinning up a database in the context of unit tests already raises red flags. Even if somehow you're talking about integration and end-to-end tests, none of the perceived problems you're mentioning are even a challenge, let alone a problem. > Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. That's perfectly fine. It's besides the point though, and completely ignores the whole point of unit tests, which is to have a bunch of fast tests that check if your code continues to do the right thing between changes. Unit tests also compel developers to think through how they create/update code to prevent them from introducing handled scenarios. There are many reasons why the test pyramid is a thing, and why the unit test layer includes far more tests than any other layer.

simonw 2y ago
Tests that mock databases are bad tests. This is a hill I am happy to die on. Spinning up a fresh database instance for each testing run is trivially easy in every testing framework worth its salt.
rcxdude 2y ago
This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. The integration and interaction are where you get by far the most value out of testing (it's where most errors will occur: units are usually small enough to fit in your head while working on them, but the system as a whole it not), and for that you should abstract out the bare minimum needed to control your test environment and get acceptable performance, not mock absolutely everything by default. Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).
- motorest 2y ago
  > This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. That's perfectly fine. It's also the basis of the whole test pyramid concept. One of the main reasons why you have some problems caught in integration tests instead of unit tests is that developers introduced bugs by failing to characterize how some components actually work and thus fail to design both the units of code and the tests that go along with them. This is not a problem caused by unit tests. This is a problem caused by developers failing to design and implement components, which consequently means they failed to add relevant tests. > The integration and interaction are where you get by far the most value out of testing (...) It's perfectly fine to think like that. The test pyramid concept exists for a reason. However, when looking at failures you need to understand their root causes. If a fault is triggered with an integration test but not with a unit test, that tells you two things: - one or more unit of code is broken, - your unit tests fail to cover a very predictable failure mode. Each of these items by themselves reflect a fundamental failure in the way developers implemented their test strategy. > Unit tests are the last tests that should be added to any system (...) This take is fundamentally wrong, to the point that your software design process needs to be completely and fundamentally broken to believe it's reasonable that unit tests can be added post-facto after skipping the whole design process.
  - rcxdude 2y ago
    I mean, I design my code, I'm just uninterested in distorting the design of the code to make it a little bit easier to write the least valuable kinds of tests (you can always unit test some code, it's just a matter of how extreme you want to go in your test harness - I believe it's the test harness that should change to fit the natural design of the code, not the other way around). >If a fault is triggered with an integration test but not with a unit test, that tells you two things: In most cases the interaction between units is far less predictable than the behavior of a given unit. And it's often not obvious which unit is 'broken'. Usually the units are doing what they were intended to do, it's the combination of behaviors that are broken (which yes, means 'the design is broken' - that's usually the harder part than writing the code!). No amount of unit tests will catch that. The integration tests will catch problems units, in contrast (admittedly, not 'accidentally working' cases, but I find it's rare anyone else cares about those).

simonw 2y ago

Tests that mock databases are bad tests. This is a hill I am happy to die on. Spinning up a fresh database instance for each testing run is trivially easy in every testing framework worth its salt.

rcxdude 2y ago

This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. The integration and interaction are where you get by far the most value out of testing (it's where most errors will occur: units are usually small enough to fit in your head while working on them, but the system as a whole it not), and for that you should abstract out the bare minimum needed to control your test environment and get acceptable performance, not mock absolutely everything by default. Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).

motorest 2y ago
> This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. That's perfectly fine. It's also the basis of the whole test pyramid concept. One of the main reasons why you have some problems caught in integration tests instead of unit tests is that developers introduced bugs by failing to characterize how some components actually work and thus fail to design both the units of code and the tests that go along with them. This is not a problem caused by unit tests. This is a problem caused by developers failing to design and implement components, which consequently means they failed to add relevant tests. > The integration and interaction are where you get by far the most value out of testing (...) It's perfectly fine to think like that. The test pyramid concept exists for a reason. However, when looking at failures you need to understand their root causes. If a fault is triggered with an integration test but not with a unit test, that tells you two things: - one or more unit of code is broken, - your unit tests fail to cover a very predictable failure mode. Each of these items by themselves reflect a fundamental failure in the way developers implemented their test strategy. > Unit tests are the last tests that should be added to any system (...) This take is fundamentally wrong, to the point that your software design process needs to be completely and fundamentally broken to believe it's reasonable that unit tests can be added post-facto after skipping the whole design process.
- rcxdude 2y ago
  I mean, I design my code, I'm just uninterested in distorting the design of the code to make it a little bit easier to write the least valuable kinds of tests (you can always unit test some code, it's just a matter of how extreme you want to go in your test harness - I believe it's the test harness that should change to fit the natural design of the code, not the other way around). >If a fault is triggered with an integration test but not with a unit test, that tells you two things: In most cases the interaction between units is far less predictable than the behavior of a given unit. And it's often not obvious which unit is 'broken'. Usually the units are doing what they were intended to do, it's the combination of behaviors that are broken (which yes, means 'the design is broken' - that's usually the harder part than writing the code!). No amount of unit tests will catch that. The integration tests will catch problems units, in contrast (admittedly, not 'accidentally working' cases, but I find it's rare anyone else cares about those).