9 ms·

This response has been trotted out a few times on HN. IMHO its terribly mistaken. An analogy is that unit tests are like a brick layer's spirit-level and plumb

This response has been trotted out a few times on HN. IMHO its terribly mistaken. An analogy is that unit tests are like a brick layer's spirit-level and plumb-line. As you build the wall, you ensure the pieces are put together correctly and head off mistakes early. The "only test at a high-level" is like saying don't confirm the bricks you are laying are correct, instead bring in an inspector to check the whole building is up to code once we have finished. If we do bring in the inspector she takes one look and says the wall have to come down because of faults right at the bottom or because we forgot the damp course or we never get that far because we couldn't fit the roof because all the walls bowed out. I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases and that they are not in a position to be advising others.

pka 11y ago
The situation you are describing is best handled by generative tests (i.e. Quickcheck). And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time. Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with. Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?
- duncanawoods 11y ago
  I'm not a Haskell programmer but how would you deal with a case like e.g. using a set of complex regexes for parsing some unusual date-time formats. I would want to unit test the hell out of them covering all the variations and exceptions I could think of. I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
  - pka 11y ago
    In this particular case I would use some Quickcheck-like generative test library and give it a description of how to generate random dates (i.e. start by generating a day, then a month, then a combination of both etc) and then leave to it the generation of all test data. The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite. On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense). Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help. However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...) (I know this is a very short answer but I'm typing on a phone)
    - mattnewton 11y ago
      You can make a finite state machine to model your expectations on random input, but then your test code is almost as complex as the code :) I've found quickcheck super useful as a way to fuzz user input or find edge cases in unit tests I didn't see, but I still write all my integration tests by hand because they are so hard to formalize.
mehwoot 11y ago
and head off mistakes early. What? On the codebase I work on, integration tests run every single commit, just like unit tests. They would both fail at the same time. Except the integration tests are much less brittle.
- quanticle 11y ago
  Yes, but I can run unit tests while I'm coding. Waiting for a check-in is far too late. If I've broken something, it's more helpful to let me know about it ten seconds later, rather than hours later, when I've pushed my change and broken the build.
  - dllthomas 11y ago
    Some integration tests can perfectly well be run on similar terms.
- glenjamin 11y ago
  My unit test suite runs in ~500ms. I run it every single time I save a file. In fact, it's set up with a filesystem watcher to do this automatically and announce the result to me via growl. The faster your test suite, the more often you can run it, and the faster you get feedback on your code. You can combine this with a healthy integration test suite, and only run the unit tests. Validation rules is a good example, if I want to test that my user input validation rules are correct, I don't need to do this through the entire application frontend - I can just check those rules in isolation.
  - crdoconnor 11y ago
    >The faster your test suite, the more often you can run it, and the faster you get feedback on your code. In all likelihood, the faster it is, the more tightly coupled it will be. You're simply exchanging computing power for brain power (refactoring tests when you refactor code), which is nearly always a bad trade off.
    - 11y ago
crdoconnor 11y ago
>I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases That's funny. I suspect that people who advance the pro-unit test view do so haven't had sufficient experience with large complex code bases. Experience with large complex code bases with a spaghetti problem is the exact thing that put me off unit tests. Unit tests are by their nature very tightly coupled to the underlying implementation, which is why they cause so many problems. It's kind of like pouring concrete over a bunch of code bricks: if those bricks are in the wrong place and need refactoring, by pouring unit tests over them means you're solidifying the broken architecture and making it even harder to fix. IME there basically don't exist large projects without technical debt problems either, so arguing that unit tests work well when the architecture is good is kind of a moot point.
- AnimalMuppet 11y ago
  First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test. That doesn't help you if you are adding tests to an existing project that's a mess, though. > Unit tests are by their nature very tightly coupled to the underlying implementation No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. It shouldn't care about the internal behavior of the "unit". That's the theory. They're only supposed to be coupled to the interface, not to the implementation.
  - crdoconnor 11y ago
    >First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test. I've found this to be one of the biggest myths of unit test driven development. Developers who have a tendency to write spaghetti do so anyhow. When it becomes too hard to write tests they simply don't write tests or they write 'pretend' tests. It's perfectly possible to write spaghetti code and write a badly written test that notionally covers it. Good coders code well. Bad coders don't. TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation. >That doesn't help you if you are adding tests to an existing project that's a mess, though. Harms you deeply, in fact. >No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. i.e. it couples itself to a unit, which is an underlying implementation detail. If it's coupling itself to an externally facing interface it's an integration test or an end to end test. >It shouldn't care about the internal behavior of the "unit". No, but then again, since "units" are supposed to be the smallest blocks of code that comprise your code base, it's caring very much about how all of the blocks talk to one another, which is an implementation detail (for your project). Perhaps the most crucial implementation detail, in fact. How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt. Pouring test cement in those cracks means that any technical debt accrued in those API contracts becomes impossible to refactor without breaking the tests. In practice this makes that refactoring more difficult than if it had no tests at all due to the massive psychological inhibition against deleting tests or turning the build red and the added problem that real failures look the same as changed API contracts.

pka 11y ago

The situation you are describing is best handled by generative tests (i.e. Quickcheck). And this is basically the only time I would write an unit test - when I can not capture most invariants in the types. Everything else is basically double the work and a waste of time. Integration tests are always a good idea, but hard to get "conceptually" right. When you get to the level of testing multiple components through the whole stack most of the time IO is involved and this gets hairy fast, not to mention that one mostly tests the 3 codepaths they can come up with. Maybe a DSL/swappable interpreter approach is one way do that? Or make everything pure again with FRP?

duncanawoods 11y ago
I'm not a Haskell programmer but how would you deal with a case like e.g. using a set of complex regexes for parsing some unusual date-time formats. I would want to unit test the hell out of them covering all the variations and exceptions I could think of. I can't see how exhaustively testing a small unit like this with high-level tests would be practical in terms of setup effort and run-time.
- pka 11y ago
  In this particular case I would use some Quickcheck-like generative test library and give it a description of how to generate random dates (i.e. start by generating a day, then a month, then a combination of both etc) and then leave to it the generation of all test data. The amount of edge cases Quickcheck catches is beyond the most comprehensove unit test suite. On a higher level though I would use parser combinators. If you have a grammar specification you can then directly transform it into a parser and thus would only need to test that you transformed it correctly (i.e. prove the proof as opposed to prove the implementation if that makes sense). Basically - try to capture most of your invariants in the types and then you know your code is correct. The invariants may be wrong though, in which case integration testing may help. However, I still have no idea how to test programs with complex state and IO in a way that isn't basically "script the 5 behaviors I can come up with and pray for the best", in which case I mostly don't even bother (i.e. UI etc...) (I know this is a very short answer but I'm typing on a phone)
  - mattnewton 11y ago
    You can make a finite state machine to model your expectations on random input, but then your test code is almost as complex as the code :) I've found quickcheck super useful as a way to fuzz user input or find edge cases in unit tests I didn't see, but I still write all my integration tests by hand because they are so hard to formalize.

mehwoot 11y ago

and head off mistakes early. What? On the codebase I work on, integration tests run every single commit, just like unit tests. They would both fail at the same time. Except the integration tests are much less brittle.

quanticle 11y ago
Yes, but I can run unit tests while I'm coding. Waiting for a check-in is far too late. If I've broken something, it's more helpful to let me know about it ten seconds later, rather than hours later, when I've pushed my change and broken the build.
- dllthomas 11y ago
  Some integration tests can perfectly well be run on similar terms.
glenjamin 11y ago
My unit test suite runs in ~500ms. I run it every single time I save a file. In fact, it's set up with a filesystem watcher to do this automatically and announce the result to me via growl. The faster your test suite, the more often you can run it, and the faster you get feedback on your code. You can combine this with a healthy integration test suite, and only run the unit tests. Validation rules is a good example, if I want to test that my user input validation rules are correct, I don't need to do this through the entire application frontend - I can just check those rules in isolation.
- crdoconnor 11y ago
  >The faster your test suite, the more often you can run it, and the faster you get feedback on your code. In all likelihood, the faster it is, the more tightly coupled it will be. You're simply exchanging computing power for brain power (refactoring tests when you refactor code), which is nearly always a bad trade off.
  - 11y ago

crdoconnor 11y ago

>I fear that those who advance anti-unit-test views only do so because they lack experience with large complex code-bases That's funny. I suspect that people who advance the pro-unit test view do so haven't had sufficient experience with large complex code bases. Experience with large complex code bases with a spaghetti problem is the exact thing that put me off unit tests. Unit tests are by their nature very tightly coupled to the underlying implementation, which is why they cause so many problems. It's kind of like pouring concrete over a bunch of code bricks: if those bricks are in the wrong place and need refactoring, by pouring unit tests over them means you're solidifying the broken architecture and making it even harder to fix. IME there basically don't exist large projects without technical debt problems either, so arguing that unit tests work well when the architecture is good is kind of a moot point.

AnimalMuppet 11y ago
First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test. That doesn't help you if you are adding tests to an existing project that's a mess, though. > Unit tests are by their nature very tightly coupled to the underlying implementation No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. It shouldn't care about the internal behavior of the "unit". That's the theory. They're only supposed to be coupled to the interface, not to the implementation.
- crdoconnor 11y ago
  >First, if you write the unit tests as you're writing the code, they kind of push you to not write spaghetti, because it's hard to test. I've found this to be one of the biggest myths of unit test driven development. Developers who have a tendency to write spaghetti do so anyhow. When it becomes too hard to write tests they simply don't write tests or they write 'pretend' tests. It's perfectly possible to write spaghetti code and write a badly written test that notionally covers it. Good coders code well. Bad coders don't. TDD doesn't help here and if there is any relationship between TDD and good design, I'm certain it's correlation not causation. >That doesn't help you if you are adding tests to an existing project that's a mess, though. Harms you deeply, in fact. >No, not "by their nature". Unit tests are supposed to test that a "unit" has the intended external behavior. i.e. it couples itself to a unit, which is an underlying implementation detail. If it's coupling itself to an externally facing interface it's an integration test or an end to end test. >It shouldn't care about the internal behavior of the "unit". No, but then again, since "units" are supposed to be the smallest blocks of code that comprise your code base, it's caring very much about how all of the blocks talk to one another, which is an implementation detail (for your project). Perhaps the most crucial implementation detail, in fact. How those units talk to one another (their API contracts) is basically the most critical aspect of your project's architecture and the single biggest source of unavoidable technical debt. Pouring test cement in those cracks means that any technical debt accrued in those API contracts becomes impossible to refactor without breaking the tests. In practice this makes that refactoring more difficult than if it had no tests at all due to the massive psychological inhibition against deleting tests or turning the build red and the added problem that real failures look the same as changed API contracts.