5 ms·

You claim it's dogmatism, yet do the same thing in reverse. (: Unit and integration tests test different layers of the system, and one isn't inherently better

by imiric 1y ago

You claim it's dogmatism, yet do the same thing in reverse. (: Unit and integration tests test different layers of the system, and one isn't inherently better or more useful than the other. They complement each other to cover behavior that is impossible to test otherwise. You can't test low-level functionality in integration tests, just as you can't test high-level functionality in unit tests. There's nothing dogmatic about that statement. If you disagree with it, that's your prerogative, but it's also my opinion that it is a mistake. It is a harmful mentality that makes code bases risky to change, and regressions more likely. So feel free to adopt it in your personal projects if you wish, but don't be surprised if you get push back on it when working in a team. Unless your teammates think the same, in which case, good luck to you all.

tsimionescu 1y ago
The problem with this line of argument is that, in general, high level behavior (covered by integratuon tests) is dependent on low level behavior. So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. So, integration tests also tell you if your component works at a low level, not just a high level. The converse is not true, however. It's perfectly possible for individual components to "work" well, but to not do the right thing from a high level perspective. Say, one component provides a good fast quicksort function, but the other component requires a stable sort to work properly - each is OK in isolation, but you need an integration test to figure out the mistake. Unit tests are typically good scaffolding. They allow you to test bits of your infrastructure as you're building it but before it's ready for integration into the larger project. But they give you realtively little assurance at the project level, and are not worth it unless you're pretty sure you're building the right thing in the first place.
- imiric 1y ago
  > So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. No, that is not guaranteed. Integration and E2E tests can only cover certain code paths, because they depend on the input and output from other systems (frontend, databases, etc.). This I/O might be crafted in ways that never trigger a failure scenario or expose a bug within the lower-level components. This doesn't mean that the issue doesn't exist—it just means that you're not seeing it. Furthermore, the fact that, by their nature, integration and E2E tests are often more expensive to setup and run, there will be fewer of them, which means they will not have full coverage of the underlying components. Another issue is that often these tests, particularly E2E and acceptance tests, are written only with a happy path in mind, and ignore the myriad of input that might trigger a failure in the real world. Another problem with your argument is that it ignores that tests have different audiences. E2E and acceptance tests are written for the end user; integration tests are written for system integrators and operators; and unit tests are written for users of the API, which includes the author and other programmers. If you disregard one set of tests, you are disregarding that audience. To a programmer and maintainer of the software, E2E and acceptance tests have little value. They might not use the software at all. What they do care about is that the function, method, object, module, or package, does what says on the tin; that it returns the correct output when given a specific input; that it's performant, efficient, well documented, and so on. These users matter because they are the ones who will maintain the software in the long run. So thinking that unit tests are useless because they're a chore to maintain is a very shortsighted mentality. Instead, it's more beneficial to see them as guardrails that make your future work easier, by giving you the confidence that you're not inadvertently breaking an API contract whenever you make a change, even when all higher-level tests remain green across the board.
  - troupo 1y ago
    > This I/O might be crafted in ways that never trigger a failure scenario or expose a bug within the lower-level components. You mean just like unit tests where every useful interaction between units is mocked out of existence? > Furthermore, the fact that, by their nature, integration and E2E tests are often more expensive to setup and run, there will be fewer of them And that's the main issue: people pretend that only unit tests matter, and as a result all other forms of testing are an afterthought. Every test harness and library is geared towards unit testing, and unit testing only.
    - imiric 1y ago
      > You mean just like unit tests where every useful interaction between units is mocked out of existence? Sure, that is a risk. But not all unit tests require mocking or stubbing. There may be plenty of pure functions that are worth testing. Writing good tests requires care and effort, like any other code, regardless of the test type. > And that's the main issue: people pretend that only unit tests matter, and as a result all other forms of testing are an afterthought. Huh? Who is saying this? The argument is coming from the other side with the claim that unit tests don't matter. Everyone arguing against this is saying that, no, all tests matter. (Let's not devolve into politics... :)) The idea of the test pyramid has nothing to do with one type of test being more important than another. It's simply a matter of practicality and utility. Higher-level tests can cover much more code than lower-level ones. In projects that keep track of code coverage, it's not unheard of for a few E2E and integration tests to cover a large percentage of the code base, e.g. >50% of lines or statements. This doesn't mean that these tests are more valuable. It simply means that they have a larger reach by their nature. These tests also require more boilerplate to setup, external system dependencies, they take more time to run, and so on. It is often impractical to rely on them during development, since they slow down the write-test loop. Instead, running the full unit test suite and a select couple of integration and E2E tests can serve as a quick sanity check, while the entire test suite runs in CI. Conversely, achieving >50% of line or statement coverage with unit tests alone also doesn't mean that the software works as it should when it interacts with other systems, or the end user. So, again, all test types are important and useful in their own way, and help ensure that the software doesn't regress.
- integralid 1y ago
  > So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too In the ideal world maybe. But It's very hard to test edge cases of a sorting algorithm with integration test. In general my experience is that algorithms and some complex but pure functions are worth writing unit tests for. CRUD app boilerplate is not.
  - MoreQARespect 1y ago
    Ive never in my life written a test for a sorting algorithm nor, im sure, will i ever need to. The bias most developers have towards integration tests reflects the fact that even though we're often interviewed on it, it's quite rare that most developers actually have to write complex algorithms. It's one of the ironies of the profession.
    - yakshaving_jgt 1y ago
      I write parsers all the time. Why wouldn’t you test parsers in isolation?
      - simianwords 1y ago
        Sure but not everyone is working at this level. Dogmatically writing unit tests where they don’t bring much value is something that happens all the time and needs to stop. No one actually evaluates whether unit tests are needed. Unit tests at least in my experience, are needed sparingly - in specific places that encompass slightly complicated well contained logic.
        yakshaving_jgt 1y ago
        I think parsing happens in more places than people might think. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
        1y ago
- EliRivers 1y ago
  So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. I have 100% seen bugs that cancel each other out; code that's just plain wrong at the lower level, coming together by chance to work at the higher level such that one or more integration tests pass. When one piece of that lower level code then gets fixed, either deliberately or because of a library update or hardware improvement or some other change that should have nothing to do with the functionality, and the top level integration tests starts failing, it can be so painful to figure it out. I've also seen bugs that cancel either other out to make one integration test pass, but don't cancel each other out such that other integration tests fail. That can be a mindmelt; surely if THIS test works, then ALL THIS low level code must be correct, but simultaneously if THAT test fails, then ALL THIS low level code is NOT correct. At which point, people start wishing they had lower level tests.

by imiric 1y ago

tsimionescu 1y ago

The problem with this line of argument is that, in general, high level behavior (covered by integratuon tests) is dependent on low level behavior. So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. So, integration tests also tell you if your component works at a low level, not just a high level. The converse is not true, however. It's perfectly possible for individual components to "work" well, but to not do the right thing from a high level perspective. Say, one component provides a good fast quicksort function, but the other component requires a stable sort to work properly - each is OK in isolation, but you need an integration test to figure out the mistake. Unit tests are typically good scaffolding. They allow you to test bits of your infrastructure as you're building it but before it's ready for integration into the larger project. But they give you realtively little assurance at the project level, and are not worth it unless you're pretty sure you're building the right thing in the first place.

imiric 1y ago
> So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. No, that is not guaranteed. Integration and E2E tests can only cover certain code paths, because they depend on the input and output from other systems (frontend, databases, etc.). This I/O might be crafted in ways that never trigger a failure scenario or expose a bug within the lower-level components. This doesn't mean that the issue doesn't exist—it just means that you're not seeing it. Furthermore, the fact that, by their nature, integration and E2E tests are often more expensive to setup and run, there will be fewer of them, which means they will not have full coverage of the underlying components. Another issue is that often these tests, particularly E2E and acceptance tests, are written only with a happy path in mind, and ignore the myriad of input that might trigger a failure in the real world. Another problem with your argument is that it ignores that tests have different audiences. E2E and acceptance tests are written for the end user; integration tests are written for system integrators and operators; and unit tests are written for users of the API, which includes the author and other programmers. If you disregard one set of tests, you are disregarding that audience. To a programmer and maintainer of the software, E2E and acceptance tests have little value. They might not use the software at all. What they do care about is that the function, method, object, module, or package, does what says on the tin; that it returns the correct output when given a specific input; that it's performant, efficient, well documented, and so on. These users matter because they are the ones who will maintain the software in the long run. So thinking that unit tests are useless because they're a chore to maintain is a very shortsighted mentality. Instead, it's more beneficial to see them as guardrails that make your future work easier, by giving you the confidence that you're not inadvertently breaking an API contract whenever you make a change, even when all higher-level tests remain green across the board.
- troupo 1y ago
  > This I/O might be crafted in ways that never trigger a failure scenario or expose a bug within the lower-level components. You mean just like unit tests where every useful interaction between units is mocked out of existence? > Furthermore, the fact that, by their nature, integration and E2E tests are often more expensive to setup and run, there will be fewer of them And that's the main issue: people pretend that only unit tests matter, and as a result all other forms of testing are an afterthought. Every test harness and library is geared towards unit testing, and unit testing only.
  - imiric 1y ago
    > You mean just like unit tests where every useful interaction between units is mocked out of existence? Sure, that is a risk. But not all unit tests require mocking or stubbing. There may be plenty of pure functions that are worth testing. Writing good tests requires care and effort, like any other code, regardless of the test type. > And that's the main issue: people pretend that only unit tests matter, and as a result all other forms of testing are an afterthought. Huh? Who is saying this? The argument is coming from the other side with the claim that unit tests don't matter. Everyone arguing against this is saying that, no, all tests matter. (Let's not devolve into politics... :)) The idea of the test pyramid has nothing to do with one type of test being more important than another. It's simply a matter of practicality and utility. Higher-level tests can cover much more code than lower-level ones. In projects that keep track of code coverage, it's not unheard of for a few E2E and integration tests to cover a large percentage of the code base, e.g. >50% of lines or statements. This doesn't mean that these tests are more valuable. It simply means that they have a larger reach by their nature. These tests also require more boilerplate to setup, external system dependencies, they take more time to run, and so on. It is often impractical to rely on them during development, since they slow down the write-test loop. Instead, running the full unit test suite and a select couple of integration and E2E tests can serve as a quick sanity check, while the entire test suite runs in CI. Conversely, achieving >50% of line or statement coverage with unit tests alone also doesn't mean that the software works as it should when it interacts with other systems, or the end user. So, again, all test types are important and useful in their own way, and help ensure that the software doesn't regress.
integralid 1y ago
> So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too In the ideal world maybe. But It's very hard to test edge cases of a sorting algorithm with integration test. In general my experience is that algorithms and some complex but pure functions are worth writing unit tests for. CRUD app boilerplate is not.
- MoreQARespect 1y ago
  Ive never in my life written a test for a sorting algorithm nor, im sure, will i ever need to. The bias most developers have towards integration tests reflects the fact that even though we're often interviewed on it, it's quite rare that most developers actually have to write complex algorithms. It's one of the ironies of the profession.
  - yakshaving_jgt 1y ago
    I write parsers all the time. Why wouldn’t you test parsers in isolation?
    - simianwords 1y ago
      Sure but not everyone is working at this level. Dogmatically writing unit tests where they don’t bring much value is something that happens all the time and needs to stop. No one actually evaluates whether unit tests are needed. Unit tests at least in my experience, are needed sparingly - in specific places that encompass slightly complicated well contained logic.
      - yakshaving_jgt 1y ago
        I think parsing happens in more places than people might think. https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
        1y ago
EliRivers 1y ago
So if your code is ascertained to work at the high level, you also know that it must be working at the lower level too. I have 100% seen bugs that cancel each other out; code that's just plain wrong at the lower level, coming together by chance to work at the higher level such that one or more integration tests pass. When one piece of that lower level code then gets fixed, either deliberately or because of a library update or hardware improvement or some other change that should have nothing to do with the functionality, and the top level integration tests starts failing, it can be so painful to figure it out. I've also seen bugs that cancel either other out to make one integration test pass, but don't cancel each other out such that other integration tests fail. That can be a mindmelt; surely if THIS test works, then ALL THIS low level code must be correct, but simultaneously if THAT test fails, then ALL THIS low level code is NOT correct. At which point, people start wishing they had lower level tests.