8 ms·

"consider instead writing higher level tests that each each test more of the code." I really like the way they keep repeating this as the answer to so many que

"consider instead writing higher level tests that each each test more of the code." I really like the way they keep repeating this as the answer to so many questions. To me it reads as a softly softly approach to weaning people off what appears to be a mania for micro level unit testing, driven by people like Uncle Bob.

ajmurmann 9y ago
I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.
- discreteevent 9y ago
  The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a few lines of code. It's all very neat and satisfying but for me it's just a form of fooling myself. I can see this if I honestly measure the net effect of the these kinds of tests in the large. The only thing is that they might sometimes help me locate a problem a few seconds faster, but that's all.
  - cle 9y ago
    Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.
    - u801e 9y ago
      > I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details I think this comes from "unit tests" where they have assertions such as assertThisOtherSpecificMethodNameWasCalledWithTheseSpecificallyNamedParameters If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. This would also eliminate the need for mocking since the methods that are tested have no side effects. For testing dependencies, an environment where a test instance of the database, API, or other external service is needed. Then the code could be tested against an actual implementation rather than a mock of it.
      - yeukhon 9y ago
        > If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. I am not really following. You can write a seperate function that handles database query, then call this function in your code under test, or one of your argument in the code under test is the databse object handler (this allows you to abstract from a specific db handler). But if you don’t mock out the handler with a “dummy handler”, you will be making real databse call. That is not UT. You will be doing functional testing. So you can only abstract so much to get your code testable.
        lmm 9y ago
        Rather than having 3 different business logic methods that call out to the database, you have each construct a command object and then have a separate service that calls out to the database based on these commands. You can then test the 3 business logic methods by testing that they construct the correct commands based on their parameters.
        falsedan 9y ago
  - querulous 9y ago
    what's wrong with just deleting these now-irrelevant tests?
    - fnord123 9y ago
      Nothing. Integration tests are your hypothesis/experiment. Unit tests are your lab setup.
    - discreteevent 9y ago
      You can delete them but the fact that they are irrelevant now even though you didn't change the behaviour of the software means that it was a waste of time writing them in the first place. Secondly it decreases your confidence in the refactor. In a classic refactor, nothing should break. This is because a good set of tests are a way of documenting and verifying the design/behaviour of the system. Finally, often in a codebase with micro tests, there will be an expectation that you replace one set of micro tests with another.
  - ajmurmann 9y ago
    I've seen this with out differently on different code bases. Particularly on Rails code bases I've seen some of the symptoms you describe. On more recent Java projects I've seen the opposite where code that needed to change actually just ended up getting swapped out. The "only one reason to change"/open-closed dream come true. Old classes and unit tests just got thrown out. On those acceptance tests were the pain point when behavior was changed on purpose. I'm still not sure how to get the good outcome every time. Maybe it's just well designed interfaces ‍️
  - rimliu 9y ago
    I recommend https://www.sandimetz.com/99bottles/ https://www.sandimetz.com/99bottles/ to everyone. Also her other book, Practical Object Oriented Design in Ruby. They both communicate the essential points very clearly. Also, after reading those books I got a better idea why OOP has a bad reputation, and why people are so bad at writing tests. Somehow these skills are presumed to be easy and already there when in reality they need practice, a lot of it.
- crdoconnor 9y ago
  >design feedback that integration or acceptance style tests won't give you. That "feedback" is the pain you feel when you have to maintain one form of tight coupling (ordinary bad code) combined with another (micro-level unit tests). Higher level tests won't cause you that 'guiding pain' because they aren't as tightly coupled to your implementation. IMO you don't need to write tens of thousands of lines of unit test code to spot that you're building tight coupling in. It's something you can spot simply by knowing what to look for. Moreover, if you make a design mistake or introduce tech debt lower down the stack, having a panoply of low level unit tests means that refactoring will break those tests even if you haven't introduced any bugs. IME that causes a dangerous cementing effect on bad code. I think if you've got a module which is self contained, reusable and tightly scoped, it's worth surrounding with a lower level test. But, if you don't, you're doing more harm than good by surrounding a module with tests.
- twic 9y ago
  > You really need a healthy testing pyramid. I don't think you do. I think a testing pyramid is a sign of an unhealthy approach and an unhealthy codebase. I think what you need is a testhenge. For those who aren't familiar with henges, here's a typical (although poorly maintained) one: http://www.english-heritage.org.uk/remote/www.english-heritage.org.uk/content/properties/stonehenge/portico/2670999/stonehenge-sunrise http://www.english-heritage.org.uk/remote/www.english-herita... Rather than being massive at the bottom and then getting thinner as it goes up, there is a continuous band around the top, covering everything, supported by uprights that dive deeper down, just where needed. That is, you need a layer of high-level tests for everything (for a web application, some mix of browser tests or controller-level integration tests), along with component-level integration tests and isolated unit tests for the bits of the system that are particularly risky (complex, error-prone, critical, new, old, etc).
newfoundglory 9y ago
I don't like it, it reminds me of the time a colleague wrote 2 separate pieces of new functionality and wrote one single end to end test that used both of them, as the "most efficient" way to test it.
- philipodonnell 9y ago
  Honest question: can you expand on why this is bad? Doesn't efficiency outweigh respecting feature boundaries in tests?
  - Ace17 9y ago
    > can you expand on why this is bad? Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. > Doesn't efficiency outweigh respecting feature boundaries in tests? In the end, what we're trying to minimize here is feature development time. This of course depends on the feedback loop duration (build + run tests), which is why test efficiency is important ; but there are other factors. Feature development depends on the time it takes to locate the error when a test fails ; a test signalling that "something is wrong somewhere" is not very usefull (in some cases, it can be, if it runs extremly fast - because you can generally see the error with "git diff"). However, infinitely precise tests are often undesirable, because they tend to have extremely rigid expectations about the behavior/API of the code under test, discouraging refactoring, and thus slowing down the development process. Let's keep in mind that "testing = freezing". More precisely, you're freezing what your tests depend upon (by adding friction to future modifications). So be careful what you freeze: the initial intent of testing is to increase code flexibility. If you can't modify anything without breaking a test, you're probably missing their benefit.
    - arkh 9y ago
      > Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. Tests are not a debug tool. Tests are here to tell you when you broke something. When this happen you can get your debugging toolbox out: stack traces, profilers and things like GDB. And then follow the steps your test script did.
seanwilson 9y ago
The argument I get against this is that when a higher level test fails, it's harder to locate the lines of code that broke the test unlike when you have lots of unit tests. I don't find this convincing though. Most of the time it's going to be pretty obvious looking at what lines of code have changed since the last working commit. Lots of projects get by without any tests at all as well. I'm not saying to skip testing completely but it comes as a cost and you need to be practical at weighing up how much time to put into it vs how much time you're going to save. Writing unit tests for everything takes time.
- blowski 9y ago
  There is no silver bullet. Personally, I let a combination of complexity and importance guide my tests. The more likely it is that a piece of code will break, and the more business damage it will do if it does break, the more tests I wrap around it. For self-contained algorithms that have a lot of branches or complex cases, I use more unit tests. When the complexity is in the interaction with other code, I write more high-level tests. When the system is simple but critical, I write more smoke tests. If I’ve got simple code that’s unlikely to break and it doesn’t matter if it does break, I might have no tests at all.
  - ed_blackburn 9y ago
    100% In critical areas I would suggest parameterised tests are worth the effort, especially in conjunction with generators. Property-based testing in FP for example, or just test data generators that'll generate a good range.
    - blowski 9y ago
      I'm not sure what's involved there. Do you have more detail, like a good blog post, to read about how to do that?
      - dvlsg 9y ago
        Not OP, but I particularly enjoy the fsharpforfunandprofit article about it. https://fsharpforfunandprofit.com/posts/property-based-testing/ https://fsharpforfunandprofit.com/posts/property-based-testi...
Ace17 9y ago
> ... micro level unit testing, driven by people like Uncle Bob. "The structure of your tests should not be a mirror of the structure of your code. The fact that you have a class named X should not automatically imply that you have a test class named XTest." "The structure of the tests must not reflect the structure of the production code, because that much coupling makes the system fragile and obstructs refactoring. Rather, the structure of the tests must be independently designed so as to minimize the coupling to the production code." http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContravariance.html http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava...
- ancarda 9y ago
  I really wish Uncle Bob would provide code examples with his blog posts and videos. I struggle to imagine what my tests would look like if I followed this. Possibly as my tests are so tightly coupled right now that refactoring is actually not possible in some cases. Does anyone know of explations of this with a more hands-on approach, or is this simply a collection of ideas that can’t really be shown?
  - hdi 9y ago
    I've been wondering about that for a while. So far, I've not been able to find any examples of test architecture that follows those principles online or in my day job. I've also been too lazy to do a side project and explore a more decoupled and scalable test suite. Maybe I should get off my arse and finally do it.
    - twic 9y ago
      If you're writing a web application, i've had a reasonably happy time with tests at the controller level, or perhaps right below. You want an interface where values go in and out that correspond to fairly user-level ideas, but which are still tractable for testing.
  - ed_blackburn 9y ago
    Practical advice? Consider making every type internal. Then open them up to what is strictly required to be public. Now you are testing the behaviour of your application through its public surface. This reduces the brittleness of your tests because you can change the internal implementation without rewriting your tests. You have higher assurance. It'll also force your hand to enforce invariants and place guard clauses in the right places rather than "everywhere". If you follow Cockburn's Ports and Adapters approach too, then you can substitute adapters like persistent state, buses, HTTP clients for appropriate in-memory equivalents.
Vinnl 9y ago
On the one hand this is true, but as someone who does prefer keeping unit tests relatively small in order to be able to pinpoint issues easier and not have tests break as often without it being clear why, I think the article does strike a nice balance of only recommending it where it really makes sense, rather than simply stimulating you to write as high-level tests as possible.

ajmurmann 9y ago

I'm not sure I agree with the "instead" in his recommendation. "micro level unit testing" comes with its own benefits, like faster feedback and also design feedback that integration or acceptance style tests won't give you. I like to start out with a high level acceptance tests and then write unit tests for every change to involved classes. You really need a healthy testing pyramid.

discreteevent 9y ago
The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a few lines of code. It's all very neat and satisfying but for me it's just a form of fooling myself. I can see this if I honestly measure the net effect of the these kinds of tests in the large. The only thing is that they might sometimes help me locate a problem a few seconds faster, but that's all.
- cle 9y ago
  Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.
  - u801e 9y ago
    > I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details I think this comes from "unit tests" where they have assertions such as assertThisOtherSpecificMethodNameWasCalledWithTheseSpecificallyNamedParameters If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. This would also eliminate the need for mocking since the methods that are tested have no side effects. For testing dependencies, an environment where a test instance of the database, API, or other external service is needed. Then the code could be tested against an actual implementation rather than a mock of it.
    - yeukhon 9y ago
      > If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. I am not really following. You can write a seperate function that handles database query, then call this function in your code under test, or one of your argument in the code under test is the databse object handler (this allows you to abstract from a specific db handler). But if you don’t mock out the handler with a “dummy handler”, you will be making real databse call. That is not UT. You will be doing functional testing. So you can only abstract so much to get your code testable.
      - lmm 9y ago
        Rather than having 3 different business logic methods that call out to the database, you have each construct a command object and then have a separate service that calls out to the database based on these commands. You can then test the 3 business logic methods by testing that they construct the correct commands based on their parameters.
        falsedan 9y ago
- querulous 9y ago
  what's wrong with just deleting these now-irrelevant tests?
  - fnord123 9y ago
    Nothing. Integration tests are your hypothesis/experiment. Unit tests are your lab setup.
  - discreteevent 9y ago
    You can delete them but the fact that they are irrelevant now even though you didn't change the behaviour of the software means that it was a waste of time writing them in the first place. Secondly it decreases your confidence in the refactor. In a classic refactor, nothing should break. This is because a good set of tests are a way of documenting and verifying the design/behaviour of the system. Finally, often in a codebase with micro tests, there will be an expectation that you replace one set of micro tests with another.
- ajmurmann 9y ago
  I've seen this with out differently on different code bases. Particularly on Rails code bases I've seen some of the symptoms you describe. On more recent Java projects I've seen the opposite where code that needed to change actually just ended up getting swapped out. The "only one reason to change"/open-closed dream come true. Old classes and unit tests just got thrown out. On those acceptance tests were the pain point when behavior was changed on purpose. I'm still not sure how to get the good outcome every time. Maybe it's just well designed interfaces ‍️
- rimliu 9y ago
  I recommend https://www.sandimetz.com/99bottles/ https://www.sandimetz.com/99bottles/ to everyone. Also her other book, Practical Object Oriented Design in Ruby. They both communicate the essential points very clearly. Also, after reading those books I got a better idea why OOP has a bad reputation, and why people are so bad at writing tests. Somehow these skills are presumed to be easy and already there when in reality they need practice, a lot of it.
crdoconnor 9y ago
>design feedback that integration or acceptance style tests won't give you. That "feedback" is the pain you feel when you have to maintain one form of tight coupling (ordinary bad code) combined with another (micro-level unit tests). Higher level tests won't cause you that 'guiding pain' because they aren't as tightly coupled to your implementation. IMO you don't need to write tens of thousands of lines of unit test code to spot that you're building tight coupling in. It's something you can spot simply by knowing what to look for. Moreover, if you make a design mistake or introduce tech debt lower down the stack, having a panoply of low level unit tests means that refactoring will break those tests even if you haven't introduced any bugs. IME that causes a dangerous cementing effect on bad code. I think if you've got a module which is self contained, reusable and tightly scoped, it's worth surrounding with a lower level test. But, if you don't, you're doing more harm than good by surrounding a module with tests.
twic 9y ago
> You really need a healthy testing pyramid. I don't think you do. I think a testing pyramid is a sign of an unhealthy approach and an unhealthy codebase. I think what you need is a testhenge. For those who aren't familiar with henges, here's a typical (although poorly maintained) one: http://www.english-heritage.org.uk/remote/www.english-heritage.org.uk/content/properties/stonehenge/portico/2670999/stonehenge-sunrise http://www.english-heritage.org.uk/remote/www.english-herita... Rather than being massive at the bottom and then getting thinner as it goes up, there is a continuous band around the top, covering everything, supported by uprights that dive deeper down, just where needed. That is, you need a layer of high-level tests for everything (for a web application, some mix of browser tests or controller-level integration tests), along with component-level integration tests and isolated unit tests for the bits of the system that are particularly risky (complex, error-prone, critical, new, old, etc).

newfoundglory 9y ago

I don't like it, it reminds me of the time a colleague wrote 2 separate pieces of new functionality and wrote one single end to end test that used both of them, as the "most efficient" way to test it.

philipodonnell 9y ago
Honest question: can you expand on why this is bad? Doesn't efficiency outweigh respecting feature boundaries in tests?
- Ace17 9y ago
  > can you expand on why this is bad? Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. > Doesn't efficiency outweigh respecting feature boundaries in tests? In the end, what we're trying to minimize here is feature development time. This of course depends on the feedback loop duration (build + run tests), which is why test efficiency is important ; but there are other factors. Feature development depends on the time it takes to locate the error when a test fails ; a test signalling that "something is wrong somewhere" is not very usefull (in some cases, it can be, if it runs extremly fast - because you can generally see the error with "git diff"). However, infinitely precise tests are often undesirable, because they tend to have extremely rigid expectations about the behavior/API of the code under test, discouraging refactoring, and thus slowing down the development process. Let's keep in mind that "testing = freezing". More precisely, you're freezing what your tests depend upon (by adding friction to future modifications). So be careful what you freeze: the initial intent of testing is to increase code flexibility. If you can't modify anything without breaking a test, you're probably missing their benefit.
  - arkh 9y ago
    > Because most of the time, it reduces "test precision". Which means when such a test detects an issue, there's now a bigger area of the code where the issue could be located. Tests are not a debug tool. Tests are here to tell you when you broke something. When this happen you can get your debugging toolbox out: stack traces, profilers and things like GDB. And then follow the steps your test script did.

seanwilson 9y ago

The argument I get against this is that when a higher level test fails, it's harder to locate the lines of code that broke the test unlike when you have lots of unit tests. I don't find this convincing though. Most of the time it's going to be pretty obvious looking at what lines of code have changed since the last working commit. Lots of projects get by without any tests at all as well. I'm not saying to skip testing completely but it comes as a cost and you need to be practical at weighing up how much time to put into it vs how much time you're going to save. Writing unit tests for everything takes time.

blowski 9y ago
There is no silver bullet. Personally, I let a combination of complexity and importance guide my tests. The more likely it is that a piece of code will break, and the more business damage it will do if it does break, the more tests I wrap around it. For self-contained algorithms that have a lot of branches or complex cases, I use more unit tests. When the complexity is in the interaction with other code, I write more high-level tests. When the system is simple but critical, I write more smoke tests. If I’ve got simple code that’s unlikely to break and it doesn’t matter if it does break, I might have no tests at all.
- ed_blackburn 9y ago
  100% In critical areas I would suggest parameterised tests are worth the effort, especially in conjunction with generators. Property-based testing in FP for example, or just test data generators that'll generate a good range.
  - blowski 9y ago
    I'm not sure what's involved there. Do you have more detail, like a good blog post, to read about how to do that?
    - dvlsg 9y ago
      Not OP, but I particularly enjoy the fsharpforfunandprofit article about it. https://fsharpforfunandprofit.com/posts/property-based-testing/ https://fsharpforfunandprofit.com/posts/property-based-testi...

Ace17 9y ago

> ... micro level unit testing, driven by people like Uncle Bob. "The structure of your tests should not be a mirror of the structure of your code. The fact that you have a class named X should not automatically imply that you have a test class named XTest." "The structure of the tests must not reflect the structure of the production code, because that much coupling makes the system fragile and obstructs refactoring. Rather, the structure of the tests must be independently designed so as to minimize the coupling to the production code." http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContravariance.html http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava...

ancarda 9y ago
I really wish Uncle Bob would provide code examples with his blog posts and videos. I struggle to imagine what my tests would look like if I followed this. Possibly as my tests are so tightly coupled right now that refactoring is actually not possible in some cases. Does anyone know of explations of this with a more hands-on approach, or is this simply a collection of ideas that can’t really be shown?
- hdi 9y ago
  I've been wondering about that for a while. So far, I've not been able to find any examples of test architecture that follows those principles online or in my day job. I've also been too lazy to do a side project and explore a more decoupled and scalable test suite. Maybe I should get off my arse and finally do it.
  - twic 9y ago
    If you're writing a web application, i've had a reasonably happy time with tests at the controller level, or perhaps right below. You want an interface where values go in and out that correspond to fairly user-level ideas, but which are still tractable for testing.
- ed_blackburn 9y ago
  Practical advice? Consider making every type internal. Then open them up to what is strictly required to be public. Now you are testing the behaviour of your application through its public surface. This reduces the brittleness of your tests because you can change the internal implementation without rewriting your tests. You have higher assurance. It'll also force your hand to enforce invariants and place guard clauses in the right places rather than "everywhere". If you follow Cockburn's Ports and Adapters approach too, then you can substitute adapters like persistent state, buses, HTTP clients for appropriate in-memory equivalents.

Vinnl 9y ago

On the one hand this is true, but as someone who does prefer keeping unit tests relatively small in order to be able to pinpoint issues easier and not have tests break as often without it being clear why, I think the article does strike a nice balance of only recommending it where it really makes sense, rather than simply stimulating you to write as high-level tests as possible.