4 ms·

I'm still firmly in the "no" camp on testing private methods. By definition, anything your class does in the real world is done via its public interface; I'm no

I'm still firmly in the "no" camp on testing private methods. By definition, anything your class does in the real world is done via its public interface; I'm not sure why I need to care about what's done under the covers. I've actually moved further and further away from unit testing over the years (after being a pretty big TDD fan for a long time). In terms of bang for buck, integration tests across your public API are the best IMO. You're testing how your API is actually used. The problem historically has been that they're fragile to refactoring and difficult to run, but with the right tooling you can get around that.

turtles3 3y ago
I find the opposite wrt. refactoring - that is to say integration tests are _more_ robust to refactoring than unit tests, and that's one of the reasons I strongly prefer them. Unit tests are deeply coupled to the internal structure of your system - refactoring often implies changing unit tests, which opens the door to bugs where the code and test both change to match each other. As you say, integration tests validate your public API, which from a 'correctness' point of view is really the only thing you care about, not the internal structure of the system. That's why I love integration tests, you can make sweeping refractors without needing to change the tests, because the test will still tell you whether the behaviour of the whole system is correct.
- throwbadubadu 3y ago
  Yes, especially for some unit testing is done like "atomic testing" (test the smallest possible part, if they could they would test individual lines), but turns out that the putting together is another crucial part with often not less complexity and where the corner cases come in. In that sense even not much like the unit vs integration testing difference & debates, because that line is blurry and just depends on your definition of an unit: It may be the single heavy algorithmic function, it may be the whole state machine, it may even be two heavily dependent and cooperating tasks in an embedded RTOS... and you always know you are in trouble if people discuss more what an unit test is and not is (instead of the value whatever thing provides). I "love" btw those code bases rich with tests but 99% of the time not finding a bug, but breaking on the tiniest detail change.. not supporting refactoring like it should be but making it slow and everyone wanting not to refactor because of the crap test rat tail.
- Yodel0914 3y ago
  Integration tests are robust with respect to process flow/business logic, but (can be) fragile with respect to the mechanics of calling the API. Back In The Day that used to be a real issue because the tooling for writing API integration tests was separate from the tooling for writing the API itself. These days (at least the way we do it) the integration test code gets refactored alongside the rest of the code, and compile-time errors catch 90% of issues. That's part 1 of what sold me on integration tests. Part 2 is the ability to simply and efficiently spin up a DB per test, removing the need for mocking the persistence layer.
kaba0 3y ago
I think unit tests can be divided into two camps as well: there are the trivial ones that are very close-coupled to the implementation, like, check if my ad hoc stack implementation will actually add the element and return sensible outputs. Something like this can easily be a private method as well, and I think they make sense to include right next to the function itself. Some languages allow these reality checks right in the documentation itself - and I don’t see who would it hurt. They are closely coupled to the implementation so sure, they will have to be rewritten with it, but that’s completely fine. The usually called unit tests indeed should not directly test implementation details, like private methods, but then we have to find a different name for the former because I do think they are useful sometimes
jasfi 3y ago
Agreed, there isn't unlimited time to perform testing. Sometimes you also know when you're writing tests that will never help find a bug, and that's a waste of time. I always focus on integration testing. If I have time I'll write unit tests, but there needs to be a good reason. I'll unit test an area that has proven to be buggy even with integration tests, but this is rare. Finally, some code is just really complex and/or critical. These are good candidates for unit tests.
- Yodel0914 3y ago
  Agreed, too :) The one place I still use unit tests is around calculation algorithms where you want to push a lot of different data through. You could do it with an integration test, but the test run overhead of say 100 parameterized integration tests is far too high. In these cases the algorithm will still be exercised by an integration test, but the nitty-gritty will be done in a unit test.
JohnBooty 3y ago
First thing I'd say is that there's no "one size fits all" approach. The right answer depends on the language, the project, the requirements, the amount of churn, the timeline, etc. So I 100% believe you when you say you've found success by moving the focus toward implementation tests. =) But in my experiences, I've had major issues with a reliance on integration tests. This is almost exclusively in Rails. - They're slow, because there wasn't an easy way to stub out the slow stuff - If you stub out the slow stuff now you have no coverage for the stubbed stuff, unless you also have unit tests covering the guts of the app - When integration tests fail, it can take a lot of work to investigate the cause, as opposed to granular unit tests
- Yodel0914 3y ago
  Our default approach is to mock and stub as little as possible, but of course there are things we just have to mock, like external service calls. However, we also have UI automation tests that run on a fully deployed and integrated system, which mostly covers that. Slowness is only a problem when it's a problem, and can sometimes be solved with better tooling. I'd feel more confident in a system with 1000 integration tests that take 2 hours to run over one with 10,000 unit tests that take 10 minutes to run. On your last point, this is positive and negative. Yes, it can be harder to find the issue, but IME integration tests actually find the issues that will cause things to break in production, and it's easier to find out why a test is failing than why an error is being logged.
audunw 3y ago
I’m wondering if the software world has something to learn from digital design here (for once), where there’s a huge emphasis on code coverage collection. You really shouldn’t care how a function is tested. That there exists a test for that particular function isn’t particularly useful measure of how well the function is tested. So you need a methodology that tells you which of your functions have been covered by your tests. If it’s not covered you should make a judgement about how to get that coverage. Getting your private methods covered by the public interface is good because it encourages you to write thorough tests for those public methods that covers all the ways the method might be used.
- sfn42 3y ago
  I agree a lot with this, and want to add that if you have a lot of complex private logic in a class it might make sense to extract an abstraction from it (if possible) and make it its own class for which you can then test the public (previously private) methods.
appplication 3y ago
Integration tests are great, but the very real downside is that they tend to be much more complex to write than unit tests, and can in some cases require setup and tear down overhead that makes it impractical to lean on them for the majority of testing logic. Unit tests for each unit of functionality, and a small number of integration tests to validate all works well together.
- Yodel0914 3y ago
  Again, I think it comes down to tooling. Our app is basically a document management system with quite complex business rules. I can write a test that takes a document from inception to completion in about 15 LoC because we've put time into our test framework to provide sensible abstractions over our API. Setup and teardown can be an issue, but we've found a good balance. We spin up a clean DB for each test in less than a second. Running out full suite does take a couple of hours, but that's what CI servers are for.

turtles3 3y ago

I find the opposite wrt. refactoring - that is to say integration tests are _more_ robust to refactoring than unit tests, and that's one of the reasons I strongly prefer them. Unit tests are deeply coupled to the internal structure of your system - refactoring often implies changing unit tests, which opens the door to bugs where the code and test both change to match each other. As you say, integration tests validate your public API, which from a 'correctness' point of view is really the only thing you care about, not the internal structure of the system. That's why I love integration tests, you can make sweeping refractors without needing to change the tests, because the test will still tell you whether the behaviour of the whole system is correct.

throwbadubadu 3y ago
Yes, especially for some unit testing is done like "atomic testing" (test the smallest possible part, if they could they would test individual lines), but turns out that the putting together is another crucial part with often not less complexity and where the corner cases come in. In that sense even not much like the unit vs integration testing difference & debates, because that line is blurry and just depends on your definition of an unit: It may be the single heavy algorithmic function, it may be the whole state machine, it may even be two heavily dependent and cooperating tasks in an embedded RTOS... and you always know you are in trouble if people discuss more what an unit test is and not is (instead of the value whatever thing provides). I "love" btw those code bases rich with tests but 99% of the time not finding a bug, but breaking on the tiniest detail change.. not supporting refactoring like it should be but making it slow and everyone wanting not to refactor because of the crap test rat tail.
Yodel0914 3y ago
Integration tests are robust with respect to process flow/business logic, but (can be) fragile with respect to the mechanics of calling the API. Back In The Day that used to be a real issue because the tooling for writing API integration tests was separate from the tooling for writing the API itself. These days (at least the way we do it) the integration test code gets refactored alongside the rest of the code, and compile-time errors catch 90% of issues. That's part 1 of what sold me on integration tests. Part 2 is the ability to simply and efficiently spin up a DB per test, removing the need for mocking the persistence layer.

kaba0 3y ago

I think unit tests can be divided into two camps as well: there are the trivial ones that are very close-coupled to the implementation, like, check if my ad hoc stack implementation will actually add the element and return sensible outputs. Something like this can easily be a private method as well, and I think they make sense to include right next to the function itself. Some languages allow these reality checks right in the documentation itself - and I don’t see who would it hurt. They are closely coupled to the implementation so sure, they will have to be rewritten with it, but that’s completely fine. The usually called unit tests indeed should not directly test implementation details, like private methods, but then we have to find a different name for the former because I do think they are useful sometimes

jasfi 3y ago

Agreed, there isn't unlimited time to perform testing. Sometimes you also know when you're writing tests that will never help find a bug, and that's a waste of time. I always focus on integration testing. If I have time I'll write unit tests, but there needs to be a good reason. I'll unit test an area that has proven to be buggy even with integration tests, but this is rare. Finally, some code is just really complex and/or critical. These are good candidates for unit tests.

Yodel0914 3y ago
Agreed, too :) The one place I still use unit tests is around calculation algorithms where you want to push a lot of different data through. You could do it with an integration test, but the test run overhead of say 100 parameterized integration tests is far too high. In these cases the algorithm will still be exercised by an integration test, but the nitty-gritty will be done in a unit test.

JohnBooty 3y ago

First thing I'd say is that there's no "one size fits all" approach. The right answer depends on the language, the project, the requirements, the amount of churn, the timeline, etc. So I 100% believe you when you say you've found success by moving the focus toward implementation tests. =) But in my experiences, I've had major issues with a reliance on integration tests. This is almost exclusively in Rails. - They're slow, because there wasn't an easy way to stub out the slow stuff - If you stub out the slow stuff now you have no coverage for the stubbed stuff, unless you also have unit tests covering the guts of the app - When integration tests fail, it can take a lot of work to investigate the cause, as opposed to granular unit tests

Yodel0914 3y ago
Our default approach is to mock and stub as little as possible, but of course there are things we just have to mock, like external service calls. However, we also have UI automation tests that run on a fully deployed and integrated system, which mostly covers that. Slowness is only a problem when it's a problem, and can sometimes be solved with better tooling. I'd feel more confident in a system with 1000 integration tests that take 2 hours to run over one with 10,000 unit tests that take 10 minutes to run. On your last point, this is positive and negative. Yes, it can be harder to find the issue, but IME integration tests actually find the issues that will cause things to break in production, and it's easier to find out why a test is failing than why an error is being logged.

audunw 3y ago

I’m wondering if the software world has something to learn from digital design here (for once), where there’s a huge emphasis on code coverage collection. You really shouldn’t care how a function is tested. That there exists a test for that particular function isn’t particularly useful measure of how well the function is tested. So you need a methodology that tells you which of your functions have been covered by your tests. If it’s not covered you should make a judgement about how to get that coverage. Getting your private methods covered by the public interface is good because it encourages you to write thorough tests for those public methods that covers all the ways the method might be used.

sfn42 3y ago
I agree a lot with this, and want to add that if you have a lot of complex private logic in a class it might make sense to extract an abstraction from it (if possible) and make it its own class for which you can then test the public (previously private) methods.

appplication 3y ago

Integration tests are great, but the very real downside is that they tend to be much more complex to write than unit tests, and can in some cases require setup and tear down overhead that makes it impractical to lean on them for the majority of testing logic. Unit tests for each unit of functionality, and a small number of integration tests to validate all works well together.

Yodel0914 3y ago
Again, I think it comes down to tooling. Our app is basically a document management system with quite complex business rules. I can write a test that takes a document from inception to completion in about 15 LoC because we've put time into our test framework to provide sensible abstractions over our API. Setup and teardown can be an issue, but we've found a good balance. We spin up a clean DB for each test in less than a second. Running out full suite does take a couple of hours, but that's what CI servers are for.