3 ms·

Here's an article from Google on their move away from more end to end and integration tests, and towards more unit tests. I think it justifies it pretty well:

Here's an article from Google on their move away from more end to end and integration tests, and towards more unit tests. I think it justifies it pretty well: http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html http://googletesting.blogspot.com/2015/04/just-say-no-to-mor...

dllthomas 11y ago
That's a really poor article, and doesn't really speak to integration tests in the first place. I'll follow up on the article's deficiencies around the argument they were trying to make when I have a bit more time free.
- dllthomas 11y ago
  Expanding: As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported. The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well. To the weaknesses of the article with regard to its critique of end-to-end tests: > Let's assume the team already has some fantastic test infrastructure in place. Every night: Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests. > To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to: > A failing test does not directly benefit the user. A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied! > End-to-end tests were flaky at times. Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests. Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests. It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.
crdoconnor 11y ago
>When the tests do run, flaky tests tend to be a fact of life. Flaky tests are indicative of bugs in your tests. Google is literally advocating treating your bugs as a fact of life, not fixing them, and avoiding harder engineering challenges because they are harder. This is the worst possible advice.

dllthomas 11y ago

That's a really poor article, and doesn't really speak to integration tests in the first place. I'll follow up on the article's deficiencies around the argument they were trying to make when I have a bit more time free.

dllthomas 11y ago
Expanding: As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported. The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well. To the weaknesses of the article with regard to its critique of end-to-end tests: > Let's assume the team already has some fantastic test infrastructure in place. Every night: Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests. > To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to: > A failing test does not directly benefit the user. A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied! > End-to-end tests were flaky at times. Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests. Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests. It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.

crdoconnor 11y ago

>When the tests do run, flaky tests tend to be a fact of life. Flaky tests are indicative of bugs in your tests. Google is literally advocating treating your bugs as a fact of life, not fixing them, and avoiding harder engineering challenges because they are harder. This is the worst possible advice.