3 ms·

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to re

The problem is that, in my observation over a number of codebases with many developers: A lot of unit tests end up testing the implementation. When you go to refactor, these tests break which slows you down. To me it seems nearly impossible not to end up with these kinds of net negative tests when you follow a micro approach. I mean the approach outlined in books and blogs where you write a test and then write only a few lines of code. It's all very neat and satisfying but for me it's just a form of fooling myself. I can see this if I honestly measure the net effect of the these kinds of tests in the large. The only thing is that they might sometimes help me locate a problem a few seconds faster, but that's all.

cle 9y ago
Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.
- u801e 9y ago
  > I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details I think this comes from "unit tests" where they have assertions such as assertThisOtherSpecificMethodNameWasCalledWithTheseSpecificallyNamedParameters If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. This would also eliminate the need for mocking since the methods that are tested have no side effects. For testing dependencies, an environment where a test instance of the database, API, or other external service is needed. Then the code could be tested against an actual implementation rather than a mock of it.
  - yeukhon 9y ago
    > If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. I am not really following. You can write a seperate function that handles database query, then call this function in your code under test, or one of your argument in the code under test is the databse object handler (this allows you to abstract from a specific db handler). But if you don’t mock out the handler with a “dummy handler”, you will be making real databse call. That is not UT. You will be doing functional testing. So you can only abstract so much to get your code testable.
    - lmm 9y ago
      Rather than having 3 different business logic methods that call out to the database, you have each construct a command object and then have a separate service that calls out to the database based on these commands. You can then test the 3 business logic methods by testing that they construct the correct commands based on their parameters.
      - falsedan 9y ago
        Yeah, exactly. To put it another way: get your business logic components to return data structures (a list of DB actions or an updated dict), and your core controller to turn those data structures into DB calls. Then your tests simply say: here’s a certain environment, what action do you think should be performed? If you think this sounds like mocking, you’re right (it is like mocking, it isn’t mocking). It has the bonus of making it easier to inspect the logic & expose it to the user ("hey I’m about to do these things, does that sound ok to you?").
      - yeukhon 9y ago
        Thank you. Would you mind to elaborate a little more with perhaps a little pseudo-code snippet?
        lmm 9y ago
- darioush 9y ago
  Another way they prevent refactoring is by creating the work for you to implement the same type / number of tests on your new code so your code review doesn't look like it's deleting 100 tests.
querulous 9y ago
what's wrong with just deleting these now-irrelevant tests?
- fnord123 9y ago
  Nothing. Integration tests are your hypothesis/experiment. Unit tests are your lab setup.
- discreteevent 9y ago
  You can delete them but the fact that they are irrelevant now even though you didn't change the behaviour of the software means that it was a waste of time writing them in the first place. Secondly it decreases your confidence in the refactor. In a classic refactor, nothing should break. This is because a good set of tests are a way of documenting and verifying the design/behaviour of the system. Finally, often in a codebase with micro tests, there will be an expectation that you replace one set of micro tests with another.
ajmurmann 9y ago
I've seen this with out differently on different code bases. Particularly on Rails code bases I've seen some of the symptoms you describe. On more recent Java projects I've seen the opposite where code that needed to change actually just ended up getting swapped out. The "only one reason to change"/open-closed dream come true. Old classes and unit tests just got thrown out. On those acceptance tests were the pain point when behavior was changed on purpose. I'm still not sure how to get the good outcome every time. Maybe it's just well designed interfaces ‍️
rimliu 9y ago
I recommend https://www.sandimetz.com/99bottles/ https://www.sandimetz.com/99bottles/ to everyone. Also her other book, Practical Object Oriented Design in Ruby. They both communicate the essential points very clearly. Also, after reading those books I got a better idea why OOP has a bad reputation, and why people are so bad at writing tests. Somehow these skills are presumed to be easy and already there when in reality they need practice, a lot of it.

cle 9y ago

Agree. I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details--added to satisfy a minimum code coverage metric. UTs were supposed to save us from the pain of refactoring, but in many cases they just trade one time-sink for another, and the cost equation remains the same (not to refactor, because it's too time consuming). In this same vein, I propose another question: How similar does the test look to the production code? These kinds of super-granular, brittle tests have a way of looking very similar to the code they are testing (e.g. super strict mocking of every dependency almost looks exactly the same as the code under test!). That style of testing is basically writing the code twice, and verifying that the second copy does indeed equal the first. I'd rather just write the damn code twice, and save myself from having to figure out all the minute differences between testing libraries that framework jockies bikeshed over.

u801e 9y ago
> I am so sick of having to code in extremely brittle codebases that are impossible to refactor because of the sheer volume of UTs that are tightly coupled to very specific implementation details I think this comes from "unit tests" where they have assertions such as assertThisOtherSpecificMethodNameWasCalledWithTheseSpecificallyNamedParameters If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. This would also eliminate the need for mocking since the methods that are tested have no side effects. For testing dependencies, an environment where a test instance of the database, API, or other external service is needed. Then the code could be tested against an actual implementation rather than a mock of it.
- yeukhon 9y ago
  > If more methods were written to not call out to disk or some external service (e.g., a database or API) and we had a main method that handled the external communication, then the unit tests would be much less fragile. I am not really following. You can write a seperate function that handles database query, then call this function in your code under test, or one of your argument in the code under test is the databse object handler (this allows you to abstract from a specific db handler). But if you don’t mock out the handler with a “dummy handler”, you will be making real databse call. That is not UT. You will be doing functional testing. So you can only abstract so much to get your code testable.
  - lmm 9y ago
    Rather than having 3 different business logic methods that call out to the database, you have each construct a command object and then have a separate service that calls out to the database based on these commands. You can then test the 3 business logic methods by testing that they construct the correct commands based on their parameters.
    - falsedan 9y ago
      Yeah, exactly. To put it another way: get your business logic components to return data structures (a list of DB actions or an updated dict), and your core controller to turn those data structures into DB calls. Then your tests simply say: here’s a certain environment, what action do you think should be performed? If you think this sounds like mocking, you’re right (it is like mocking, it isn’t mocking). It has the bonus of making it easier to inspect the logic & expose it to the user ("hey I’m about to do these things, does that sound ok to you?").
    - yeukhon 9y ago
      Thank you. Would you mind to elaborate a little more with perhaps a little pseudo-code snippet?
      - lmm 9y ago
darioush 9y ago
Another way they prevent refactoring is by creating the work for you to implement the same type / number of tests on your new code so your code review doesn't look like it's deleting 100 tests.

querulous 9y ago

what's wrong with just deleting these now-irrelevant tests?

fnord123 9y ago
Nothing. Integration tests are your hypothesis/experiment. Unit tests are your lab setup.
discreteevent 9y ago
You can delete them but the fact that they are irrelevant now even though you didn't change the behaviour of the software means that it was a waste of time writing them in the first place. Secondly it decreases your confidence in the refactor. In a classic refactor, nothing should break. This is because a good set of tests are a way of documenting and verifying the design/behaviour of the system. Finally, often in a codebase with micro tests, there will be an expectation that you replace one set of micro tests with another.

ajmurmann 9y ago

I've seen this with out differently on different code bases. Particularly on Rails code bases I've seen some of the symptoms you describe. On more recent Java projects I've seen the opposite where code that needed to change actually just ended up getting swapped out. The "only one reason to change"/open-closed dream come true. Old classes and unit tests just got thrown out. On those acceptance tests were the pain point when behavior was changed on purpose. I'm still not sure how to get the good outcome every time. Maybe it's just well designed interfaces ‍️

rimliu 9y ago

I recommend https://www.sandimetz.com/99bottles/ https://www.sandimetz.com/99bottles/ to everyone. Also her other book, Practical Object Oriented Design in Ruby. They both communicate the essential points very clearly. Also, after reading those books I got a better idea why OOP has a bad reputation, and why people are so bad at writing tests. Somehow these skills are presumed to be easy and already there when in reality they need practice, a lot of it.