8 ms·

I see unit tests recommended all the time, but in practice I've found them significantly less helpful than integration and acceptance tests. Maybe its just a no

by jameslk 11y ago

I see unit tests recommended all the time, but in practice I've found them significantly less helpful than integration and acceptance tests. Maybe its just a nomenclature thing? It's obvious if you change a "unit" of code, its tests are going to fail. That's not terribly helpful while refactoring. What's more useful is to find out what else breaks and that's where the other tests come in. I'm not saying that unit tests aren't useful. They definitely help for example if you're using a dynamic programming language or if you're practicing TDD. I would just emphasize a higher priority for integration and acceptance tests.

phillipwills 11y ago
I try to keep my unit tests focused on the api of the unit I'm developing. If I'm refactoring and keeping the api the same, I shouldn't have to change any tests. Since I started writing my tests this way, I've felt I have to change my tests far less... Sometimes I add more tests, when a bug is found and needs fixing, but missed tests are better than wrong tests IMHO.
- jameslk 11y ago
  That's always been difficult for me if a unit test is written as purely as possible (i.e. isolating the unit from its dependencies). If I have to mock dependencies, create stubs and spies, I start to have to make assumptions about how the unit works beyond just its API, which makes the test itself more brittle. How do you approach this situation?
  - rapala 11y ago
    All the dependencies of a unit are part of its API. You may not need to give them at every call, but you do need to give them atleast once. Whether the unit calls MissileLauncher::launch or not is in my opinion part of the external behavior of the unit.
  - phillipwills 11y ago
    I feel like I shouldn't be testing that the dependencies' api's didn't change, that's for integration tests, which don't need to run as often as unit tests. Therefore, they can be slower and more complex. Mock or spy everything external and nothing internal. It can feel a little dangerous, but as long as the integration and e2e tests are kept up and run on build, i think I'm covered. I've definitely been bitten before, but I've been bitten more by over complicated and long running tests.
edwinnathaniel 11y ago
Unit-Tests can be helpful when working with Java as they can give you a semi REPL-like environment (especially if your IDE integrates well with JUnit). I find both Integration and Acceptance Tests could potentially hide many bad internal designs (tight coupling, etc). I also find that not many places can speed up their full integration/acceptance tests suites; suddenly build becomes way too painful, be it local dev build or CI-triggered-through-commit builds. Note: My experience with super-slow integration tests choking build is with a large JavaScript project with 100 developers (at least). I implemented integration-tests for a Java-heavy based project before but limited only to test the database component of the project (e.g.: Repository, JPQL, JPA), the ability to test business logics that are tightly coupled with ORM quickly in isolated manner is a blessing. I guess YMMV.
zabil 11y ago
Depends on how you author the unit tests. They must be testing "behavior". If you are refactoring to clean up code, you'll want to verify the behavior and unit tests gives you that quick feedback.
duncanawoods 11y ago
This response has been trotted out a few times on HN. IMHO its terribly mistaken. An analogy is that unit tests are like a brick layer's spirit-level and plumb-line. As you build the wall, you ensure the pieces are put together correctly and head off mistakes early. The "only test at a high-level" is like saying don't confirm the bricks you are laying are correct, instead bring in an inspector to check the whole building is up to code once we have finished. If we do bring in the inspector she takes one look and says the wall have to come down because of faults right at the bottom or because we forgot the damp course or we never get that far because we couldn't fit the roof because all the walls bowed out. I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases and that they are not in a position to be advising others.
- pka 11y ago
  The situation you are describing is best handled by generative tests (i.e. Quickcheck). And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time. Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with. Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?
  - duncanawoods 11y ago
    I'm not a Haskell programmer but how would you deal with a case like e.g. using a set of complex regexes for parsing some unusual date-time formats. I would want to unit test the hell out of them covering all the variations and exceptions I could think of. I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
    - pka 11y ago
      In this particular case I would use some Quickcheck-like generative test library and give it a description of how to generate random dates (i.e. start by generating a day, then a month, then a combination of both etc) and then leave to it the generation of all test data. The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite. On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense). Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help. However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...) (I know this is a very short answer but I'm typing on a phone)
dcw303 11y ago
I treat each unit test as an individual assertion on the logic of the program. I find using a <methodName>_<Conditions>_Expectations syntax documents the intent very well. E.g: testCalculateGrade_scoreOver85_returnsAnA() You're right that refactoring the code can break tests. That's when you have to ask if the original assertion still stands. It's OK to change them if your understand of the problem has changed also.
itsmemattchung 11y ago
I recommend watching this talk: https://www.youtube.com/watch?v=yTkzNHF6rMs https://www.youtube.com/watch?v=yTkzNHF6rMs Both, unit and integration tests, are imperative. Integration tests point in the general direction of where an issue might be but unit tests are more precise.
crpatino 11y ago
In my experience, if you try to do without unit tests, your integration tests will grow to fill in the gap. This will slow you down in a two-fold way: 1. As your integration tests will both more numerous and complex to cover corner case scenarios, the time it takes to execute the suite will grow too. And since the integration test is more opaque too, this price has to be paid by both you and all the developers that touch something that might have used your module. 2. The time at which you receive the feedback of the test is already too late. You are no longer doing experimental changes, you are trying to push this unit of work into the code base and be done with it. Either you will be tempted to gloss over any divergence you find (quality price) or you will have to execute some relevant subset of your integration testsuite during your coding/OODA loop (execution speed price).
- crdoconnor 11y ago
  I haven't found this to be a problem at all. Integration tests follow a power law. If you segregate your tests into "smoke" and "non-smoke", run the former before merging code and the latter continuously and randomly and in parallel, you'll get notification of 95% of regressions within 5 minutes and the other 5% within a few hours. Choosing unit tests because of speed is nearly always a premature optimization. IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode. Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).
  - Bahamut 11y ago
    The main benefit of unit tests is the speed one can run them at, and the feedback. Unit tests make it vastly faster to figure out the culprit in a broken sector of code almost all the time, since it narrows down what code you have to check for any logic errors. It also so happens that unit tests tend to encourage better writing of code into reuseable units. Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play. A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.
njharman 11y ago
> It's obvious if you change a "unit" of code, its tests are going to fail. No it's not. Not at all. The whole point is you can change the guts of a unit around and know you haven't broken things cause the unit tests still pass. Because the unit test is testing the "contract" (explicit or implied) the unit has made with other units about it's inputs and outputs. Unit test should fail (and require updating) only if the unit's INTERFACE changes. But, if your unit tests are failing everytime you touch your code then 1) your code is not cleanly isolated and independent and/or 2) you have poor unittests.
- kazinator 11y ago
  If nothing in an application depends on some aspects of the behavior of a unit interface, what is the point in testing it? You only need that sort of test in a programming language or public library, because you're publishing the behavior to unknown numbers of users, and it has to work, as documented. If they find too many things that don't work according to the documentation, they will correctly suspect that you aren't testing, and lose confidence. But if some API is internal to an application, and no code depends on some aspect of its interface contract, then that aspect effectively doesn't exist. It may happen that someone discovers this aspect in the documentation and tries using it. At that time it is either found broken, or found working. If it is found broken, it can be fixed. Or else it can be striken from the documentation (and another way can be found in the application code to meet its requirements). If the aspect is found correct, or fixed to reflect the documentation, and is then used in the application, then that use constitutes a test. The feature is tested indirectly, through some regression test case which targets something in the application, which depends on that aspect of the API contract lower down. If you design units with all kinds of detailed aspects in their interface contracts which end up not being used at all, and test all of them, it can be argued that you're wasting time: you're basically working on your library design hobby, at the expense of the project, which just serves as a vehicle for it.
  - prodigal_erik 11y ago
    Unit tests can verify assumptions that the implementation is internally relying on. E.g., say a greedy algorithm relies on a collection being sorted, even though the public interface does not return results in any particular order. If I make a mistake such that an edge case leaves the collection not sorted correctly, a test of the public API can't tell unless it happens to trigger the edge case whose importance might not be obvious.
clay_to_n 11y ago
Here's an article from Google on their move away from more end to end and integration tests, and towards more unit tests. I think it justifies it pretty well: http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html http://googletesting.blogspot.com/2015/04/just-say-no-to-mor...
- dllthomas 11y ago
  That's a really poor article, and doesn't really speak to integration tests in the first place. I'll follow up on the article's deficiencies around the argument they were trying to make when I have a bit more time free.
  - dllthomas 11y ago
    Expanding: As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported. The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well. To the weaknesses of the article with regard to its critique of end-to-end tests: > Let's assume the team already has some fantastic test infrastructure in place. Every night: Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests. > To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to: > A failing test does not directly benefit the user. A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied! > End-to-end tests were flaky at times. Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests. Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests. It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.
karmajunkie 11y ago
They serve different roles in my projects. Unit tests are intended to A) define and ensure adherence to a contract, a role which can also be partially (though not fully) fulfilled by strong and/or static typing in some languages; and B) as a harness to drive development while it is ongoing, which is a transient function, meaning those tests can be removed or consolidated after a development cycle. Integration and functional tests, on the other hand, are long-lived tests intended to ensure that all the pieces stay glued together. They allow the middle parts of code to squiggle quite a bit, as long as they get to the same place at the end of the line. They allow for safe refactoring of modular units whose entire scope lies beneath the boundary of the test. And when the functionality of the application or library changes, they should change as well, but this should be an order or two of magnitude less frequent than the unit test level. As a rule of thumb, I put integration tests around any piece of functionality that "makes me money"—less literally, around any bit that, if it were broken, would prevent things like user signups, users paying the correct amount, or critical functionality that is indispensible to a user of the software. Likewise, I usually write unit tests when i'm writing new modules so that I don't have to keep reloading a browser or setting up complex data schemes while they're in development. Sometimes I remove those, especially for code that is low-churn (which is a totally under-appreciated metric to look at when you're figuring out where you should focus coverage, IMHO.) Both types are important necessary. And not all of them are destined to live forever.
stcredzero 11y ago
Maybe its just a nomenclature thing? It's obvious if you change a "unit" of code, its tests are going to fail. Here's the thing about "agile" and Unit Tests. The current generation of agile was re-discovered in Smalltalk, after the Smalltalk Refactoring Browser had arrived on the scene. Not only could you refactor at warp speed and high confidence (1) and with unlimited undo, since Smalltalk SUnit test were just ordinary Smalltalk code, Unit tests often got refactored right along with the code. If you're in an environment where you have to painstakingly refactor, then also separately update your Unit Tests, then it's going to feel like you're doing a whole lot more work -- because you are! Given that very few environments have the requisite properties, TDD is going to feel like a slog on most stacks, and people will naturally gravitate to something lighter, like integration/acceptance/smoke tests. (1) - Yes, the prepackaged Refactorings in the original Smalltalk RB were guaranteed to be correct. There were sophisticated runtime checks, and they'd refuse to fire off if correctness wasn't completely certain. You could fool them -- because dynamic language -- but you had to try pretty hard to mess up like that. Then the rest of the shop got really annoyed at you!

by jameslk 11y ago

phillipwills 11y ago

I try to keep my unit tests focused on the api of the unit I'm developing. If I'm refactoring and keeping the api the same, I shouldn't have to change any tests. Since I started writing my tests this way, I've felt I have to change my tests far less... Sometimes I add more tests, when a bug is found and needs fixing, but missed tests are better than wrong tests IMHO.

jameslk 11y ago
That's always been difficult for me if a unit test is written as purely as possible (i.e. isolating the unit from its dependencies). If I have to mock dependencies, create stubs and spies, I start to have to make assumptions about how the unit works beyond just its API, which makes the test itself more brittle. How do you approach this situation?
- rapala 11y ago
  All the dependencies of a unit are part of its API. You may not need to give them at every call, but you do need to give them atleast once. Whether the unit calls MissileLauncher::launch or not is in my opinion part of the external behavior of the unit.
- phillipwills 11y ago
  I feel like I shouldn't be testing that the dependencies' api's didn't change, that's for integration tests, which don't need to run as often as unit tests. Therefore, they can be slower and more complex. Mock or spy everything external and nothing internal. It can feel a little dangerous, but as long as the integration and e2e tests are kept up and run on build, i think I'm covered. I've definitely been bitten before, but I've been bitten more by over complicated and long running tests.

edwinnathaniel 11y ago

Unit-Tests can be helpful when working with Java as they can give you a semi REPL-like environment (especially if your IDE integrates well with JUnit). I find both Integration and Acceptance Tests could potentially hide many bad internal designs (tight coupling, etc). I also find that not many places can speed up their full integration/acceptance tests suites; suddenly build becomes way too painful, be it local dev build or CI-triggered-through-commit builds. Note: My experience with super-slow integration tests choking build is with a large JavaScript project with 100 developers (at least). I implemented integration-tests for a Java-heavy based project before but limited only to test the database component of the project (e.g.: Repository, JPQL, JPA), the ability to test business logics that are tightly coupled with ORM quickly in isolated manner is a blessing. I guess YMMV.

zabil 11y ago

Depends on how you author the unit tests. They must be testing "behavior". If you are refactoring to clean up code, you'll want to verify the behavior and unit tests gives you that quick feedback.

duncanawoods 11y ago

This response has been trotted out a few times on HN. IMHO its terribly mistaken. An analogy is that unit tests are like a brick layer's spirit-level and plumb-line. As you build the wall, you ensure the pieces are put together correctly and head off mistakes early. The "only test at a high-level" is like saying don't confirm the bricks you are laying are correct, instead bring in an inspector to check the whole building is up to code once we have finished. If we do bring in the inspector she takes one look and says the wall have to come down because of faults right at the bottom or because we forgot the damp course or we never get that far because we couldn't fit the roof because all the walls bowed out. I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases and that they are not in a position to be advising others.

pka 11y ago
The situation you are describing is best handled by generative tests (i.e. Quickcheck). And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time. Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with. Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?
- duncanawoods 11y ago
  I'm not a Haskell programmer but how would you deal with a case like e.g. using a set of complex regexes for parsing some unusual date-time formats. I would want to unit test the hell out of them covering all the variations and exceptions I could think of. I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
  - pka 11y ago
    In this particular case I would use some Quickcheck-like generative test library and give it a description of how to generate random dates (i.e. start by generating a day, then a month, then a combination of both etc) and then leave to it the generation of all test data. The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite. On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense). Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help. However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...) (I know this is a very short answer but I'm typing on a phone)

dcw303 11y ago

I treat each unit test as an individual assertion on the logic of the program. I find using a <methodName>_<Conditions>_Expectations syntax documents the intent very well. E.g: testCalculateGrade_scoreOver85_returnsAnA() You're right that refactoring the code can break tests. That's when you have to ask if the original assertion still stands. It's OK to change them if your understand of the problem has changed also.

itsmemattchung 11y ago

I recommend watching this talk: https://www.youtube.com/watch?v=yTkzNHF6rMs https://www.youtube.com/watch?v=yTkzNHF6rMs Both, unit and integration tests, are imperative. Integration tests point in the general direction of where an issue might be but unit tests are more precise.

crpatino 11y ago

In my experience, if you try to do without unit tests, your integration tests will grow to fill in the gap. This will slow you down in a two-fold way: 1. As your integration tests will both more numerous and complex to cover corner case scenarios, the time it takes to execute the suite will grow too. And since the integration test is more opaque too, this price has to be paid by both you and all the developers that touch something that might have used your module. 2. The time at which you receive the feedback of the test is already too late. You are no longer doing experimental changes, you are trying to push this unit of work into the code base and be done with it. Either you will be tempted to gloss over any divergence you find (quality price) or you will have to execute some relevant subset of your integration testsuite during your coding/OODA loop (execution speed price).

crdoconnor 11y ago
I haven't found this to be a problem at all. Integration tests follow a power law. If you segregate your tests into "smoke" and "non-smoke", run the former before merging code and the latter continuously and randomly and in parallel, you'll get notification of 95% of regressions within 5 minutes and the other 5% within a few hours. Choosing unit tests because of speed is nearly always a premature optimization. IME everywhere that I worked with that relied exclusively on unit tests for automated regression testing didn't trust them enough to do releases without some manual user testing, so the project fell back into waterfall mode. Integration tests tested a lot more realistically and gave more freedom to refactor technical debt (since changing the code didn't require changing the test as often).
- Bahamut 11y ago
  The main benefit of unit tests is the speed one can run them at, and the feedback. Unit tests make it vastly faster to figure out the culprit in a broken sector of code almost all the time, since it narrows down what code you have to check for any logic errors. It also so happens that unit tests tend to encourage better writing of code into reuseable units. Unit tests cannot catch everything though, and for stuff like how multiple units are wired together, that's where integration tests come into play. A good codebase has both types - even combined they won't catch everything, but they should capture almost all business logic issues.

njharman 11y ago

> It's obvious if you change a "unit" of code, its tests are going to fail. No it's not. Not at all. The whole point is you can change the guts of a unit around and know you haven't broken things cause the unit tests still pass. Because the unit test is testing the "contract" (explicit or implied) the unit has made with other units about it's inputs and outputs. Unit test should fail (and require updating) only if the unit's INTERFACE changes. But, if your unit tests are failing everytime you touch your code then 1) your code is not cleanly isolated and independent and/or 2) you have poor unittests.

kazinator 11y ago
If nothing in an application depends on some aspects of the behavior of a unit interface, what is the point in testing it? You only need that sort of test in a programming language or public library, because you're publishing the behavior to unknown numbers of users, and it has to work, as documented. If they find too many things that don't work according to the documentation, they will correctly suspect that you aren't testing, and lose confidence. But if some API is internal to an application, and no code depends on some aspect of its interface contract, then that aspect effectively doesn't exist. It may happen that someone discovers this aspect in the documentation and tries using it. At that time it is either found broken, or found working. If it is found broken, it can be fixed. Or else it can be striken from the documentation (and another way can be found in the application code to meet its requirements). If the aspect is found correct, or fixed to reflect the documentation, and is then used in the application, then that use constitutes a test. The feature is tested indirectly, through some regression test case which targets something in the application, which depends on that aspect of the API contract lower down. If you design units with all kinds of detailed aspects in their interface contracts which end up not being used at all, and test all of them, it can be argued that you're wasting time: you're basically working on your library design hobby, at the expense of the project, which just serves as a vehicle for it.
- prodigal_erik 11y ago
  Unit tests can verify assumptions that the implementation is internally relying on. E.g., say a greedy algorithm relies on a collection being sorted, even though the public interface does not return results in any particular order. If I make a mistake such that an edge case leaves the collection not sorted correctly, a test of the public API can't tell unless it happens to trigger the edge case whose importance might not be obvious.

clay_to_n 11y ago

Here's an article from Google on their move away from more end to end and integration tests, and towards more unit tests. I think it justifies it pretty well: http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html http://googletesting.blogspot.com/2015/04/just-say-no-to-mor...

dllthomas 11y ago
That's a really poor article, and doesn't really speak to integration tests in the first place. I'll follow up on the article's deficiencies around the argument they were trying to make when I have a bit more time free.
- dllthomas 11y ago
  Expanding: As mentioned, the article is directed primarily at end-to-end tests. It concludes speaking favorably of some integration tests, but says that as a guess, you want 70% unit tests, 20% integration tests, and 10% end-to-end tests. It does not actually directly address why you mightn't want more integration tests than this. I'm not at all confident that that's wrong, but I suspect that the precise ratios are context specific, and in any case it's not well supported. The strongest point the article makes against end-to-end tests is that it can be difficult to pin down the exact cause of the error. In my experience this is certainly the case, and is less true of integration tests, and still less true of unit tests, when those tests catch the defect in question. It is less true still of types, when the type checker is sufficient and the software is designed to leverage it well. To the weaknesses of the article with regard to its critique of end-to-end tests: > Let's assume the team already has some fantastic test infrastructure in place. Every night: Hold on! What we're examining now is the difference between low frequency feedback and high frequency feedback. Depending on context, we may be able to run end-to-end tests every commit. And again depending on context, unit tests may be slow to run - we can always throw more generative tests at any SUT, and exhaustive tests may give us a high degree of confidence in exchange for a large amount of compute. The chief difference here is not the scope covered by the tests. > To maintain a high bar for product quality, they also require that at least 90% of their end-to-end tests pass before features are considered complete. It may be desirable to also set such a bar, in order to keep down tech debt, but every known bug should be assessed for severity - which leads to: > A failing test does not directly benefit the user. A failing test deemed critical, which prevents a release and keeps the software from destroying hours of a user's work certainly benefits the user, even before any fix is applied! > End-to-end tests were flaky at times. Flaky tests are obviously a problem. The article simply asserts that this is a problem for end-to-end tests, and doesn't address why it's not (apparently) a problem for unit tests. I'll try to do so on its behalf, here, as well as indicating where it breaks down: as end-to-end tests involve more complex interactions of more components, we may see a flaky test if any component is variable or if any interaction between components is variable, and in a large enough system we'll see some variability even in the infrastructure it runs on. So I agree that we are more likely to see flaky tests in larger scale tests. Any test should be designed to be as repeatable as we can make it, and that's usually easier for unit tests. That said, some units may contain race conditions. Where we can prove their absence, we possibly should. But where we can't (and perhaps even where we can) we should still be designing tests to expose these bugs. Those tests will inherently be flaky, so we need some approach to dealing with flaky tests. It's worth observing that a test that fails one time in a thousand, run ten thousand times, is almost certain (99.995%) to fail at least once. John Hughes talks about doing something like this to successfully test for race conditions. A smart test system could run multifold those tests that are known to be flaky. Further, if we discover a test to be flaky, we could use this same strategy as we walk back the history to discover where the error was introduced.

karmajunkie 11y ago

They serve different roles in my projects. Unit tests are intended to A) define and ensure adherence to a contract, a role which can also be partially (though not fully) fulfilled by strong and/or static typing in some languages; and B) as a harness to drive development while it is ongoing, which is a transient function, meaning those tests can be removed or consolidated after a development cycle. Integration and functional tests, on the other hand, are long-lived tests intended to ensure that all the pieces stay glued together. They allow the middle parts of code to squiggle quite a bit, as long as they get to the same place at the end of the line. They allow for safe refactoring of modular units whose entire scope lies beneath the boundary of the test. And when the functionality of the application or library changes, they should change as well, but this should be an order or two of magnitude less frequent than the unit test level. As a rule of thumb, I put integration tests around any piece of functionality that "makes me money"—less literally, around any bit that, if it were broken, would prevent things like user signups, users paying the correct amount, or critical functionality that is indispensible to a user of the software. Likewise, I usually write unit tests when i'm writing new modules so that I don't have to keep reloading a browser or setting up complex data schemes while they're in development. Sometimes I remove those, especially for code that is low-churn (which is a totally under-appreciated metric to look at when you're figuring out where you should focus coverage, IMHO.) Both types are important necessary. And not all of them are destined to live forever.

stcredzero 11y ago

Maybe its just a nomenclature thing? It's obvious if you change a "unit" of code, its tests are going to fail. Here's the thing about "agile" and Unit Tests. The current generation of agile was re-discovered in Smalltalk, after the Smalltalk Refactoring Browser had arrived on the scene. Not only could you refactor at warp speed and high confidence (1) and with unlimited undo, since Smalltalk SUnit test were just ordinary Smalltalk code, Unit tests often got refactored right along with the code. If you're in an environment where you have to painstakingly refactor, then also separately update your Unit Tests, then it's going to feel like you're doing a whole lot more work -- because you are! Given that very few environments have the requisite properties, TDD is going to feel like a slog on most stacks, and people will naturally gravitate to something lighter, like integration/acceptance/smoke tests. (1) - Yes, the prepackaged Refactorings in the original Smalltalk RB were guaranteed to be correct. There were sophisticated runtime checks, and they'd refuse to fire off if correctness wasn't completely certain. You could fool them -- because dynamic language -- but you had to try pretty hard to mess up like that. Then the rest of the shop got really annoyed at you!