5 ms·

Today I Got Burnt by Isolated Tests

22 points by abyx 15y ago

sleight42 15y ago
Hi again, Aviv! This is exactly why I tend to be more of a "statist"l preferring black box tests. Yes, my unit tests are arguably a form of integration tests themselves. However, I can usually rest assured that my tests prove the behavior that they exercise. The downside is that my tests run more slowly than isolated units under test.
jameskilton 15y ago
"Generally, I prefer the GOOS school of TDD which includes isolating my classes as much as possible, putting mocks and stubs everywhere." [emphasis added] One word: Overmocking. Mocks should be the last thing you add to any test suite and they should only be used to mock out external services or slow systems. If you find yourself needing a lot of mocks for simple tests, then your design is bad and needs to be refactored. Ryan Davis explains things best when he gets into minitest/mock (for Ruby): http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-t-matter http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-... I've been bit by this before, multiple times. I've ripped hundreds of lines of mocking code out of tests in the code I routinely work on (not originally written by me) and it has given me much more faith in my tests.
- benjiweber 15y ago
  > Mocks should be the last thing you add to any test suite and they should only be used to mock out external services or slow systems. It really depends what you're trying to achieve from your tests. I find a combination of all of: 1. Mockist tests that mock /all/ dependencies 2. Tests that stub out interactions with other layers (e.g. test the entire model) 3. Full integration tests that test the whole system with real dependencies (real db, real services) works quite well. The first set help you design and document your implementation through mock specifications. The second allows you to assert the intended behaviour of the code without tight coupling to the implementation. You're less likely to break these when refactoring. The third is the most likely to find real bugs in my experience. They're also generally more difficult to write and take longer to run, but valuable. Mock-everything tests aren't good at finding defects but they're not without value.
  - prateekdayal 15y ago
    I have seen that over time, when you refactor code and tests that use mocks, its a good idea to take out the mocks and replace them with actual objects. Mocks are important early on because you are just discovering the interfaces but over time, they can be replaced by real objects (this obviously can't be done for third party interactions). In the end, there is no substitute for integration tests. We make it a point to write integration tests for atleast all happy paths.
- aaronbrethorst 15y ago
  I like using the VCR gem with remote services that my Rails apps call: https://github.com/myronmarston/vcr https://github.com/myronmarston/vcr
wccrawford 15y ago
Sounds like the problem wasn't isolation, but failure to actually test the class. Once they had tests written for it (because they were adding a new feature) it became glaringly obvious that it wasn't working.
ilcavero 15y ago
"Isolated tests go much better hand in hand with a few integration tests" I would say that it is the other way around and that integration tests go much better hand in hand with a few isolated tests, though I've never been a fan of the TDD dogma
boyter 15y ago
tl;dr Make sure you use integration tests (or a smoke test) as mocking away the entire environment will hide environmental and system errors when you actually run things.
- bguthrie 15y ago
  I actually don't think that's a fair summary. The original article pointed out at least two other problems: - the size of the commit made it difficult to review it for potential errors, and - the feature, once committed, was not fully exercised, or as the article puts it, completely followed-through.
  - boyter 15y ago
    Sorry should have mentioned I was trying to boil down what you can learn from it. In which case a smoke test would highlight this issue pretty quickly.
- Yhippa 15y ago
  Also if you have time try to use as much real-world data or a real-world scenario as possible. I've gotten burned when I've done unit and integration tests and still that one guy slips through that we never would have thought of but of course shows up come production time.
  - boyter 15y ago
    Agreed. I tend to do the following to test any application that requires a high level of quality. 1. Unit tests - for regression testing and to enforce a clean design 2. Integration testing against real data - To check that there are no obvious bugs caused by bad interfaces or data 3. Smoke tests - To ensure that the core functionality works as expected They also get updated in terms of frequency in that order. Smoke tests take the most effort to make, but they deliver the most value so long as they have supporting unit and integration tests which catch obvious errors.

22 points by abyx 15y ago

Hi again, Aviv! This is exactly why I tend to be more of a "statist"l preferring black box tests. Yes, my unit tests are arguably a form of integration tests themselves. However, I can usually rest assured that my tests prove the behavior that they exercise. The downside is that my tests run more slowly than isolated units under test.

jameskilton 15y ago

"Generally, I prefer the GOOS school of TDD which includes isolating my classes as much as possible, putting mocks and stubs everywhere." [emphasis added] One word: Overmocking. Mocks should be the last thing you add to any test suite and they should only be used to mock out external services or slow systems. If you find yourself needing a lot of mocks for simple tests, then your design is bad and needs to be refactored. Ryan Davis explains things best when he gets into minitest/mock (for Ruby): http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-t-matter http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-... I've been bit by this before, multiple times. I've ripped hundreds of lines of mocking code out of tests in the code I routinely work on (not originally written by me) and it has given me much more faith in my tests.

benjiweber 15y ago
> Mocks should be the last thing you add to any test suite and they should only be used to mock out external services or slow systems. It really depends what you're trying to achieve from your tests. I find a combination of all of: 1. Mockist tests that mock /all/ dependencies 2. Tests that stub out interactions with other layers (e.g. test the entire model) 3. Full integration tests that test the whole system with real dependencies (real db, real services) works quite well. The first set help you design and document your implementation through mock specifications. The second allows you to assert the intended behaviour of the code without tight coupling to the implementation. You're less likely to break these when refactoring. The third is the most likely to find real bugs in my experience. They're also generally more difficult to write and take longer to run, but valuable. Mock-everything tests aren't good at finding defects but they're not without value.
- prateekdayal 15y ago
  I have seen that over time, when you refactor code and tests that use mocks, its a good idea to take out the mocks and replace them with actual objects. Mocks are important early on because you are just discovering the interfaces but over time, they can be replaced by real objects (this obviously can't be done for third party interactions). In the end, there is no substitute for integration tests. We make it a point to write integration tests for atleast all happy paths.
aaronbrethorst 15y ago
I like using the VCR gem with remote services that my Rails apps call: https://github.com/myronmarston/vcr https://github.com/myronmarston/vcr

wccrawford 15y ago

Sounds like the problem wasn't isolation, but failure to actually test the class. Once they had tests written for it (because they were adding a new feature) it became glaringly obvious that it wasn't working.

ilcavero 15y ago

"Isolated tests go much better hand in hand with a few integration tests" I would say that it is the other way around and that integration tests go much better hand in hand with a few isolated tests, though I've never been a fan of the TDD dogma

boyter 15y ago

tl;dr Make sure you use integration tests (or a smoke test) as mocking away the entire environment will hide environmental and system errors when you actually run things.

bguthrie 15y ago
I actually don't think that's a fair summary. The original article pointed out at least two other problems: - the size of the commit made it difficult to review it for potential errors, and - the feature, once committed, was not fully exercised, or as the article puts it, completely followed-through.
- boyter 15y ago
  Sorry should have mentioned I was trying to boil down what you can learn from it. In which case a smoke test would highlight this issue pretty quickly.
Yhippa 15y ago
Also if you have time try to use as much real-world data or a real-world scenario as possible. I've gotten burned when I've done unit and integration tests and still that one guy slips through that we never would have thought of but of course shows up come production time.
- boyter 15y ago
  Agreed. I tend to do the following to test any application that requires a high level of quality. 1. Unit tests - for regression testing and to enforce a clean design 2. Integration testing against real data - To check that there are no obvious bugs caused by bad interfaces or data 3. Smoke tests - To ensure that the core functionality works as expected They also get updated in terms of frequency in that order. Smoke tests take the most effort to make, but they deliver the most value so long as they have supporting unit and integration tests which catch obvious errors.