4 ms·

A reason not mentioned is that testing public methods only is more black box testing. It's important to do this not just to allow significant changes to the imp

by veltas 5y ago

A reason not mentioned is that testing public methods only is more black box testing. It's important to do this not just to allow significant changes to the implementation without affecting the test, but also because if you tie testing too much to your implementation you may miss holes in your interpretation of the requirements/interface that are better caught in a black box test. If you want coverage then private method testing is giving you fake or inferior coverage. Those hard to reach paths are only truly tested via the public interface (i.e. called by the function that will call them in practice, not your contrived test function).

bedobi 5y ago
This is so important but even most professional software developers and testers don't seem to understand it. Tests are not supposed to break on every little change to the code, or even sweeping changes in implementation. If they do, they're not very good test. https://testing.googleblog.com/2013/08/testing-on-toilet-test-behavior-not.html https://testing.googleblog.com/2013/08/testing-on-toilet-tes... https://medium.com/expedia-group-tech/test-your-tests-f4e361818720 https://medium.com/expedia-group-tech/test-your-tests-f4e361...
- Sankozi 5y ago
  I disagree. You are basically saying that only end to end tests are very good tests. There are different reasons to have tests. Unit tests often break even during small refactorings but are great to quickly cover of lot of edge cases of some complex piece of code. They also quickly point to what exactly is wrong. End to end tests on the other hand should be almost immune to refactoring problems, but they are very slow and often it is tricky to find where the bug is or even if there is a bug at all in the application - in those tests bugs are very often in test code.
  - carlmr 5y ago
    As always, all the extremes are wrong.
  - marcosdumay 5y ago
    There are plenty of non-end-to-end interfaces that are well defined so tests don't break on changes and still encapsulate complex and important code. There is somebody on this thread talking about one that calculates the costs on a contract. Tests for those are unit tests, and fit the GP's description. And those are the best units to test, often you don't want to unit test anything else. Of course, YMMV. For example, on some languages you must test if the data has the correct structure and the functions have the correct interface, what is best done by unit tests, and those tests will break often. Some problems are so complex that you must test intermediate steps, otherwise you will never get it right; those tests also break often, but when you need them, you need them.
  - hinkley 5y ago
    If refactoring is causing you to struggle to fix the unit tests, then what you have really aren’t unit tests. Unit tests should be cheap to write and cheap to reimplement. Otherwise GP’s right and the tests end up shackling you to an implementation, instead of defining the requirements programmatically.
    - travisjungroth 5y ago
      Just stop trying to fix unit tests. All of this noise about test bloat and the pain of refactorings goes away if you make one mindset change: feel free to delete tests. If you make a change and a unit test breaks, do your best to make sure it’s not an actual defect. If it isn’t and fixing the test looks like more work than you want, then delete the test. Maybe make a new one to cover a similar case if that makes sense. “Less tests” is a really easy state to get to. Just highlight and delete. If I think having more tests is good and you think that it’s bad because you don’t want the tests there if they break, that’s not a problem! We can get back to the way you want things in less than a minute. Someone writing a test is not a sworn oath to preserve it forever. It’s not even like a public method. No one is using it.
      - hinkley 5y ago
        There's an inflection point with unit tests where a good description, a good matcher, and a small number of preconditions (mostly handled by nested test suites) make it easier to rewrite the test than to fix it. When I say 'cheap' tests, these are what I mean. Tests you can treat like livestock instead of pets. The sentiment of the test may still apply, but the implementation details mean you're testing, for instance, for empty set using a different strategy. Block delete and start over. Or the feature you're adding is meant to allow us to deal with empty set: let our customers put things in their shopping cart even though we don't have any shipping addresses for them. In that case we 'broke' that requirement, and just delete the test and move on. But getting cheap tests and high coverage is a bit of an art, often learned via the school of hard knocks. If you let too many requirements leak up into integration tests, you may find you have many, many tests to fix when a "never" or "always" changes to "occasionally", "mostly". Some people get brave and delete these tests, and discover Chesterton's Fence later when someone yells at them for deleting regression tests, because the test described 2 scenarios it was asserting, but it surreptitiously asserted 2 more, one of which the New Guy just broke but got a green build. With a unit test it's tenable to make a test that looks like it checks one rule while actually checking only one rule.
pfdietz 5y ago
Testing public methods is not an argument for not testing private methods. In testing, unlike in construction of software, it can be good to combine many different approaches, because each has diminishing returns and they compensate for each others weak points.
nicoburns 5y ago
What if you have a "module" within your private code? An example I have is in a pay computation engine. I have a public function that computes how much someone ought to get paid for a given shift based on who was working the shift, what they were doing and a bunch of other criteria, what day they were working on, what time of day, etc. Within that I have a private function that works a with a narrower scope that just does the pro-rata calculation for time-of-day and day-of-week. That function itself has a fairly high-level interface with a highly complex implementation. IMO it makes sense to test that function, even though that function is private and is only used by the one public function. My point being whether it's private or public isn't really the point. The key thing is whether the functionality is self-contained and decoupled from the rest of the code with a stable interface.
- snidane 5y ago
  The 'module' is a library of pay computation functions that your app uses privately. The library itself is public and could have other users, eg. you when invoking manually from cli when testing things out. Test your library code with unit tests - this is easier because it can be done in isolation feom external systems. Itegration testing of apps is difficult. Often it is sufficient to test by use - have a dev environment which is usable or test in prod if it's not a critically important app. This is often sufficient, but I'm sure testing gurus have a recipe to apply unit tests to the app itself as well. I personally prefer not to have 10:1 test to code ratio, so I keep things practical.
mobjack 5y ago
I find testing everything through public methods can lead to excessive mocking, which ties you even more to the implementation.
- rtheunissen 5y ago
  How does mocking tie you to the implementation?
raverbashing 5y ago
> coverage then private method testing is giving you fake or inferior coverage Absolutely not Your private methods contain logic no? Every coverage is worth In fact I don't see any "controversy" here. Not testing private methods seems like a lousy excuse exactly because they change less with implementation/api changes and are more "logic focused" Sure, do test your API methods, but that might not always be easy or practical
- nisegami 5y ago
  I think you misunderstand. GP is recommending that coverage of private methods be achieved by testing their public consumers. If you can't test part of a private method through a public consumer, then that's a sign that those unused code branches might not be necessary.
magicalhippo 5y ago
Based on my experiences, I don't think anything should be truly private. I found it's almost always better to put the "private" code in a separate "implementation detail" namespace that makes it clear it's intended to be private. This hides the private implementation details, but keeps them accessible for when it's needed for internal tests or workarounds. It's easy to find code which relies on the private namespace, and one can't accidentally use it.
- hinkley 5y ago
  In Java I would sometimes make them protected methods and use a subclass at the top of the suite to expose them. These tests that cross from black to white box testing could be construed as functional tests, in which case my solution and yours aren’t so different. Also important to remember that it’s a mistake to treat library and application code as the same thing. They are not the same, and from my observations a lot of arguments about The Right Way come down to trying to treat them as if they are.
  - magicalhippo 5y ago
    Indeed protected methods can be useful for this. Depending on the task at hand, I find it can be useful to separate code into it's own namespace (one way or another) to make it more clear what the user-facing code does. Regardless, as you noted, my point is to not prevent access but rather make it obvious if you're using implementation details.
    - hinkley 5y ago
      Most test frameworks are very un-opinionated, and perhaps that is a mistake. Framework doesn't care if half your unit tests are functional tests, if you slide E2E test code into the before*() call in an integration test. If your white box and black box tests are snuggled up to each other in the same test suite. If your tests only pass if they run sequentially, in this particular order. It's no wonder we can't get the terminology straight. You can be years into writing tests before anyone or anything calls you out.
  - 5y ago

by veltas 5y ago

bedobi 5y ago

This is so important but even most professional software developers and testers don't seem to understand it. Tests are not supposed to break on every little change to the code, or even sweeping changes in implementation. If they do, they're not very good test. https://testing.googleblog.com/2013/08/testing-on-toilet-test-behavior-not.html https://testing.googleblog.com/2013/08/testing-on-toilet-tes... https://medium.com/expedia-group-tech/test-your-tests-f4e361818720 https://medium.com/expedia-group-tech/test-your-tests-f4e361...

Sankozi 5y ago
I disagree. You are basically saying that only end to end tests are very good tests. There are different reasons to have tests. Unit tests often break even during small refactorings but are great to quickly cover of lot of edge cases of some complex piece of code. They also quickly point to what exactly is wrong. End to end tests on the other hand should be almost immune to refactoring problems, but they are very slow and often it is tricky to find where the bug is or even if there is a bug at all in the application - in those tests bugs are very often in test code.
- carlmr 5y ago
  As always, all the extremes are wrong.
- marcosdumay 5y ago
  There are plenty of non-end-to-end interfaces that are well defined so tests don't break on changes and still encapsulate complex and important code. There is somebody on this thread talking about one that calculates the costs on a contract. Tests for those are unit tests, and fit the GP's description. And those are the best units to test, often you don't want to unit test anything else. Of course, YMMV. For example, on some languages you must test if the data has the correct structure and the functions have the correct interface, what is best done by unit tests, and those tests will break often. Some problems are so complex that you must test intermediate steps, otherwise you will never get it right; those tests also break often, but when you need them, you need them.
- hinkley 5y ago
  If refactoring is causing you to struggle to fix the unit tests, then what you have really aren’t unit tests. Unit tests should be cheap to write and cheap to reimplement. Otherwise GP’s right and the tests end up shackling you to an implementation, instead of defining the requirements programmatically.
  - travisjungroth 5y ago
    Just stop trying to fix unit tests. All of this noise about test bloat and the pain of refactorings goes away if you make one mindset change: feel free to delete tests. If you make a change and a unit test breaks, do your best to make sure it’s not an actual defect. If it isn’t and fixing the test looks like more work than you want, then delete the test. Maybe make a new one to cover a similar case if that makes sense. “Less tests” is a really easy state to get to. Just highlight and delete. If I think having more tests is good and you think that it’s bad because you don’t want the tests there if they break, that’s not a problem! We can get back to the way you want things in less than a minute. Someone writing a test is not a sworn oath to preserve it forever. It’s not even like a public method. No one is using it.
    - hinkley 5y ago
      There's an inflection point with unit tests where a good description, a good matcher, and a small number of preconditions (mostly handled by nested test suites) make it easier to rewrite the test than to fix it. When I say 'cheap' tests, these are what I mean. Tests you can treat like livestock instead of pets. The sentiment of the test may still apply, but the implementation details mean you're testing, for instance, for empty set using a different strategy. Block delete and start over. Or the feature you're adding is meant to allow us to deal with empty set: let our customers put things in their shopping cart even though we don't have any shipping addresses for them. In that case we 'broke' that requirement, and just delete the test and move on. But getting cheap tests and high coverage is a bit of an art, often learned via the school of hard knocks. If you let too many requirements leak up into integration tests, you may find you have many, many tests to fix when a "never" or "always" changes to "occasionally", "mostly". Some people get brave and delete these tests, and discover Chesterton's Fence later when someone yells at them for deleting regression tests, because the test described 2 scenarios it was asserting, but it surreptitiously asserted 2 more, one of which the New Guy just broke but got a green build. With a unit test it's tenable to make a test that looks like it checks one rule while actually checking only one rule.

pfdietz 5y ago

Testing public methods is not an argument for not testing private methods. In testing, unlike in construction of software, it can be good to combine many different approaches, because each has diminishing returns and they compensate for each others weak points.

nicoburns 5y ago

What if you have a "module" within your private code? An example I have is in a pay computation engine. I have a public function that computes how much someone ought to get paid for a given shift based on who was working the shift, what they were doing and a bunch of other criteria, what day they were working on, what time of day, etc. Within that I have a private function that works a with a narrower scope that just does the pro-rata calculation for time-of-day and day-of-week. That function itself has a fairly high-level interface with a highly complex implementation. IMO it makes sense to test that function, even though that function is private and is only used by the one public function. My point being whether it's private or public isn't really the point. The key thing is whether the functionality is self-contained and decoupled from the rest of the code with a stable interface.

snidane 5y ago
The 'module' is a library of pay computation functions that your app uses privately. The library itself is public and could have other users, eg. you when invoking manually from cli when testing things out. Test your library code with unit tests - this is easier because it can be done in isolation feom external systems. Itegration testing of apps is difficult. Often it is sufficient to test by use - have a dev environment which is usable or test in prod if it's not a critically important app. This is often sufficient, but I'm sure testing gurus have a recipe to apply unit tests to the app itself as well. I personally prefer not to have 10:1 test to code ratio, so I keep things practical.

mobjack 5y ago

I find testing everything through public methods can lead to excessive mocking, which ties you even more to the implementation.

rtheunissen 5y ago
How does mocking tie you to the implementation?

raverbashing 5y ago

> coverage then private method testing is giving you fake or inferior coverage Absolutely not Your private methods contain logic no? Every coverage is worth In fact I don't see any "controversy" here. Not testing private methods seems like a lousy excuse exactly because they change less with implementation/api changes and are more "logic focused" Sure, do test your API methods, but that might not always be easy or practical

nisegami 5y ago
I think you misunderstand. GP is recommending that coverage of private methods be achieved by testing their public consumers. If you can't test part of a private method through a public consumer, then that's a sign that those unused code branches might not be necessary.

magicalhippo 5y ago

Based on my experiences, I don't think anything should be truly private. I found it's almost always better to put the "private" code in a separate "implementation detail" namespace that makes it clear it's intended to be private. This hides the private implementation details, but keeps them accessible for when it's needed for internal tests or workarounds. It's easy to find code which relies on the private namespace, and one can't accidentally use it.

hinkley 5y ago
In Java I would sometimes make them protected methods and use a subclass at the top of the suite to expose them. These tests that cross from black to white box testing could be construed as functional tests, in which case my solution and yours aren’t so different. Also important to remember that it’s a mistake to treat library and application code as the same thing. They are not the same, and from my observations a lot of arguments about The Right Way come down to trying to treat them as if they are.
- magicalhippo 5y ago
  Indeed protected methods can be useful for this. Depending on the task at hand, I find it can be useful to separate code into it's own namespace (one way or another) to make it more clear what the user-facing code does. Regardless, as you noted, my point is to not prevent access but rather make it obvious if you're using implementation details.
  - hinkley 5y ago
    Most test frameworks are very un-opinionated, and perhaps that is a mistake. Framework doesn't care if half your unit tests are functional tests, if you slide E2E test code into the before*() call in an integration test. If your white box and black box tests are snuggled up to each other in the same test suite. If your tests only pass if they run sequentially, in this particular order. It's no wonder we can't get the terminology straight. You can be years into writing tests before anyone or anything calls you out.
- 5y ago