4 ms·

I find integration tests more useful for documentation because they show you how the system works at a higher level which is precisely what new developers need

I find integration tests more useful for documentation because they show you how the system works at a higher level which is precisely what new developers need to understand. If the code itself is written properly then the individual classes and objects used throughout should be high cohesion and easy to read without needing any comments or tests. There are always cases where unit tests are important though like the parts of the code that deal with money. But there is no benefit in adding unit tests if you're building a standard chat system for example; integration tests are sufficient in that case. Also getting 100% test coverage is a complete waste of time if you don't actually test the right kinds of input and output parameters.

hmottestad 9y ago
Integration tests are much more brittle in my experience. A simple test for doing frontend/backend integrations tests with web driver can easily break due to race conditions or changes in other parts of the system. Such tests are great for detecting that something is wrong, but it can be very hard to figure out what's wrong and where things went wrong. Had a lot of "fun" rewriting a bunch of integration tests for angular, because the framework used a lot of Promises which caused race conditions. Those tests were much harder to write than unit tests and were much easier to mess up.
- jondubois 9y ago
  They're not necessarily brittle but I know what you mean. When the integration tests reach across multiple services then things tend to get brittle but this can be fixed by stubbing out the external services. Stubs are not just useful for unit tests. Service stubs are often much easier to write than module stubs because services often communicate via string based protocols which are easy to simulate.
- crdoconnor 9y ago
  >Integration tests are much more brittle in my experience. In my experience brittleness in tests is caused by bugs in tests or bugs in code. Most of these things can be fixed, e.g. * Add an order by to select statements that otherwise returns data in an indeterminate order. * Fix the interpreter/database version you are using to run your code rather than relying upon the version the package manager has installed. * Stub out code that gets date/time/random numbers. * Stubbing out external APIs which go down on Tuesdays. * Fixing the bugs in your code. * Abandoning the code you rely upon which is buggy (I'm just sayin', angular might not be the ideal framework). Avoiding writing these tests because you know that "they'll be too buggy" doesn't exactly send out a great signal. >Such tests are great for detecting that something is wrong, but it can be very hard to figure out what's wrong and where things went wrong. This is alleviated by integrating debugging tools into your test environments.
zxcmx 9y ago
I think a part of the difficulty is in the language of coverage. "100% test coverage" usually means statement coverage due to the nature of popular tooling. If you deeply care about correctness you are usually more interested in path coverage which is more tied to things like heap shape or state space... (which depending on your system design might encompass stuff in your database). Also yeah, technically the exceptions which can be thrown at each point in your software if you work in a language with those. Unfortunately there's not much tooling for that so it's not visible in the same way as the more naive metrics. Anway my point is that you can waste time pushing for "100% statement coverage" measurable by naive tools but still not exercise important paths, which I think is your point.
- z3t4 9y ago
  I want to clarify that the whole point of unit tests is to have 100% coverage, that all code paths are exercised by the tests. If you do unit testing and have code that is not covered it's either a bug or dead code that should be removed.
- deleted 9y ago
  [deleted]
rectang 9y ago
> But there is no benefit in adding unit tests if you're building a standard chat system for example Disagree. Finding the right amount of unit testing is an optimization problem. Some amount is nearly always useful. For starters: Integration tests will not necessarily exercise a full range of inputs for the low-level functions. For a chat system, I would think hard about how to test low-level functions with a wide range of Unicode input. A chat system must survive in the harsh environment of the public internet. It needs to be hardened against unexpected user input, either incidental or malicious.
- crdoconnor 9y ago
  >For starters: Integration tests will not necessarily exercise a full range of inputs for the low-level functions. For a chat system, I would think hard about how to test low-level functions with a wide range of Unicode input. I don't see any problem at all with writing integration tests that exercise a chat system with a wide range of unicode input. I'd view it as irresponsible not to, in fact.
  - rectang 9y ago
    Low-level functions may be reused in combinations which limit what gets fed to them by higher-level tests. Perhaps there are bugs in some text processing widget, which only happen on small blocks -- and small blocks only get fed to low-level functions on disconnection from the service, which would be too costly to repeat exhaustively in integration tests. The most reliable way to address this kind of problem is to match the tests to the function. Look at what each function does, anticipate edge cases, contemplate the core purpose, then write what you believe are an appropriate number of unit tests. You can choose not to prioritize the problems which are revealed by unit testing but not integration testing, but you can't say that such problems don't exist.
    - crdoconnor 9y ago
      >on disconnection from the service, which would be too costly to repeat exhaustively in integration tests. Um, disconnection from the service is the exact kind of scenario I'd want to repeat exhaustively in an integration test.

hmottestad 9y ago

Integration tests are much more brittle in my experience. A simple test for doing frontend/backend integrations tests with web driver can easily break due to race conditions or changes in other parts of the system. Such tests are great for detecting that something is wrong, but it can be very hard to figure out what's wrong and where things went wrong. Had a lot of "fun" rewriting a bunch of integration tests for angular, because the framework used a lot of Promises which caused race conditions. Those tests were much harder to write than unit tests and were much easier to mess up.

jondubois 9y ago
They're not necessarily brittle but I know what you mean. When the integration tests reach across multiple services then things tend to get brittle but this can be fixed by stubbing out the external services. Stubs are not just useful for unit tests. Service stubs are often much easier to write than module stubs because services often communicate via string based protocols which are easy to simulate.
crdoconnor 9y ago
>Integration tests are much more brittle in my experience. In my experience brittleness in tests is caused by bugs in tests or bugs in code. Most of these things can be fixed, e.g. * Add an order by to select statements that otherwise returns data in an indeterminate order. * Fix the interpreter/database version you are using to run your code rather than relying upon the version the package manager has installed. * Stub out code that gets date/time/random numbers. * Stubbing out external APIs which go down on Tuesdays. * Fixing the bugs in your code. * Abandoning the code you rely upon which is buggy (I'm just sayin', angular might not be the ideal framework). Avoiding writing these tests because you know that "they'll be too buggy" doesn't exactly send out a great signal. >Such tests are great for detecting that something is wrong, but it can be very hard to figure out what's wrong and where things went wrong. This is alleviated by integrating debugging tools into your test environments.

zxcmx 9y ago

I think a part of the difficulty is in the language of coverage. "100% test coverage" usually means statement coverage due to the nature of popular tooling. If you deeply care about correctness you are usually more interested in path coverage which is more tied to things like heap shape or state space... (which depending on your system design might encompass stuff in your database). Also yeah, technically the exceptions which can be thrown at each point in your software if you work in a language with those. Unfortunately there's not much tooling for that so it's not visible in the same way as the more naive metrics. Anway my point is that you can waste time pushing for "100% statement coverage" measurable by naive tools but still not exercise important paths, which I think is your point.

z3t4 9y ago
I want to clarify that the whole point of unit tests is to have 100% coverage, that all code paths are exercised by the tests. If you do unit testing and have code that is not covered it's either a bug or dead code that should be removed.
deleted 9y ago
[deleted]

rectang 9y ago

> But there is no benefit in adding unit tests if you're building a standard chat system for example Disagree. Finding the right amount of unit testing is an optimization problem. Some amount is nearly always useful. For starters: Integration tests will not necessarily exercise a full range of inputs for the low-level functions. For a chat system, I would think hard about how to test low-level functions with a wide range of Unicode input. A chat system must survive in the harsh environment of the public internet. It needs to be hardened against unexpected user input, either incidental or malicious.

crdoconnor 9y ago
>For starters: Integration tests will not necessarily exercise a full range of inputs for the low-level functions. For a chat system, I would think hard about how to test low-level functions with a wide range of Unicode input. I don't see any problem at all with writing integration tests that exercise a chat system with a wide range of unicode input. I'd view it as irresponsible not to, in fact.
- rectang 9y ago
  Low-level functions may be reused in combinations which limit what gets fed to them by higher-level tests. Perhaps there are bugs in some text processing widget, which only happen on small blocks -- and small blocks only get fed to low-level functions on disconnection from the service, which would be too costly to repeat exhaustively in integration tests. The most reliable way to address this kind of problem is to match the tests to the function. Look at what each function does, anticipate edge cases, contemplate the core purpose, then write what you believe are an appropriate number of unit tests. You can choose not to prioritize the problems which are revealed by unit testing but not integration testing, but you can't say that such problems don't exist.
  - crdoconnor 9y ago
    >on disconnection from the service, which would be too costly to repeat exhaustively in integration tests. Um, disconnection from the service is the exact kind of scenario I'd want to repeat exhaustively in an integration test.