4 ms·

I've found that the higher level the test (i.e. unit -> integration -> functional) the better they catch things, but the harder it is to figure out what broke.

I've found that the higher level the test (i.e. unit -> integration -> functional) the better they catch things, but the harder it is to figure out what broke. For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. The other issue is we had a lot of bugs around the integration of data. For instance if the DB class returned back an empty array vs null vs empty string. When we did the unit test we mocked it to return what we thought it should, which may not be the correct one. Integration tests better caught this, functional tests even more so. It seems a lot of shops don't use tests at all, now working at places that have done both, I'd rather error on the side of fewer tests, and do it smart instead of hitting a metric and assuming your code falls into line.

wr1472 15y ago
Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.
- eaurouge 15y ago
  There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break. If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.
  - wr1472 15y ago
    Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix. Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation. Remember you are testing what it does, not how it does it.
    - jsdalton 15y ago
      > Remember you are testing what it does, not how it does it. While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite. You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html http://martinfowler.com/articles/mocksArentStubs.html
      - wr1472 15y ago
        I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#CouplingTestsToImplementations http://martinfowler.com/articles/mocksArentStubs.html#Coupli...). > Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing. Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises. Of course YMMV.
- keithnoizu 15y ago
  Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.
jrockway 15y ago
For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. This just means your tests are bad. Write good tests and you won't have this problem. Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".
- RandallBrown 15y ago
  It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.
capsule_toy 15y ago
For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.

wr1472 15y ago

Sounds like your test were quite brittle, if refactoring application code causes tests to break. A good mental rule I use is: "test the what not the how". It also a good reason to program (and test) to interfaces than to the implementation, although you have to be careful with this as it is all too easy to go OTT in your use of design patterns. You have to know how to strike the right balance.

eaurouge 15y ago
There are various kinds of tests, from unit tests to behavior driven testing. Refactoring application code may cause tests, especially unit tests, to break. If you're writing unit tests, by definition, you are testing the implementation - that's the point. It seems you're referring to behavioral (or black box) tests, which are very different.
- wr1472 15y ago
  Agreed refactoring method signatures will cause tests to break, but they should be relatively trivial to fix. Black box tests are not limited to behavioral or system integration testing. you can black box at the unit level. In fact if you are doing proper TDD you will write the tests first, which help to define the interface of the class under test, make sure the test fails to ensure you are testing for something, and then finally write the implementation to make the test pass. Such an approach implies it is black box as you've defined your interface before your implementation. Remember you are testing what it does, not how it does it.
  - jsdalton 15y ago
    > Remember you are testing what it does, not how it does it. While I tend to agree with you (I strongly prefer black-box testing over white-box testing), this is a hotly debated topic and many people feel quite strongly the opposite. You can Google around and find many advocates for white box unit tests. Martin Fowler's article from several years ago I think does an excellent job of presenting a balanced look at both sides: http://martinfowler.com/articles/mocksArentStubs.html http://martinfowler.com/articles/mocksArentStubs.html
    - wr1472 15y ago
      I've not come across the article before. The section on coupling tests to implementation I guess is the most pertinent part (http://martinfowler.com/articles/mocksArentStubs.html#CouplingTestsToImplementations http://martinfowler.com/articles/mocksArentStubs.html#Coupli...). > Coupling to the implementation also interferes with refactoring, since implementation changes are much more likely to break tests than with classic testing. Fowler acknowledges two different styles of tests - state/behaviour also can be termed as classic/mockist. He doesn't really advocate one over the other. I do mock when I have to, but only when there is no other way to setup my test. It is certainly not the default. I find that taking this approach does not make my tests brittle, and makes them robust enough to handle the majority of refactoring exercises. Of course YMMV.
keithnoizu 15y ago
Exactly. You want to verify requirements not implementation details.Requirements change less frequently than implementation details.

jrockway 15y ago

For instance we had a rule at a place I worked, where we needed to have 80% unit test coverage. And what is described in this article happened. Anytime we'd make a small refactor, we'd have to go update 3-4 test libraries. And if it was a big one, then you have a lot of tests to update. And since some were complex (or since there were a lot of them) people didn't dig into that much. When people finally did it sometimes showed the test wasn't even testing what it was supposed to anymore. This just means your tests are bad. Write good tests and you won't have this problem. Integration and functional tests are good, but only as a quick sanity check. The nitty gritty details need to be handled by good unit tests; the functional tests only tell you things like "did the RPC protocol change" or "does the database cache actually cache something".

RandallBrown 15y ago
It is really easy to write bad tests. That doesn't mean that you shouldn't write any, but just "writing good tests" can be really hard.

capsule_toy 15y ago

For me, unit tests are to assure that if I refactor the code, the places where the code is being used won't be affected by my changes. If I have to rewrite the tests, I've lost this assurance.