4 ms·

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That way tests actually assist with refactoring rather than be something that just exactly follows the code and breaks whenever a minor internal detail changes. However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception.

ardit33 9y ago
I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.
- camus2 9y ago
  > even private ones. You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members. Now the use of the private member might lead to a different code path : it has to be tested. Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked. Functional Testing : test multiple unit at the same time to check the behavior of an entire functionality, that's why it is called functional testing. Stub the IO. End-to-end testing: Test the entire system with the IO.
  - bpicolo 9y ago
    > it makes absolutely no sense to test private members Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.
    - mdpopescu 9y ago
      > I've never seen a real case for excluding private methods as a testable unit. Here's the reason: We're not writing tests; we're writing SPECIFICATIONS. Private methods are implementation details; they're not part of the specification.
      - bpicolo 9y ago
        That's potentially one philosophy on it, sure. Yet that still doesn't preclude private methods from being an independently testable unit. Implementation details are the meat of the whole thing, and so they seem inherently testworthy. The glue code itself, less so. Thanks for that thought though, definitely something I'll think on :)
        clay_to_n 9y ago
        You should be able to get the code coverage you need (even if 100%) by testing the public API of any class. If this isn't true, it means you have some private methods you can delete. That's how the unit you're testing is used - through it's public API. The public API is the specification of how it works. By only testing the public API, you allow yourself maximum ability to refactor in the future, while still maximizing code coverage. It means that simple refactoring (inlining methods, for example) won't break tests. More importantly, a failing test means something is wrong that is potentially relied upon elsewhere. If you test private methods, you will get test failures without the public API of the unit having changed at all. Your point about "each function is a unit" is fine, you can justify testing private methods with that - but it's inefficient. If it's not necessary to be in the spec (public API), why have you made it so? You're over-complicating the design by locking yourself into implementation details in places where you don't need to. In my experience I've found that testing private methods directly is a code smell. It shouldn't be necessary.
- phlakaton 9y ago
  Some counterpoints: - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. - If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful. - Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up. I think any test strategy – unit, integration, e2e, acceptance, UI – pursued to exclusion is a bad idea. Different projects and different teams call for different balances between them.
  - algesten 9y ago
    > - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. I think we talk about the same thing, sometimes I will test something internal. I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.
    - phlakaton 9y ago
      I'm kind of flexible on my terminology, perhaps more flexible than others, since my approach to testing has largely been self-taught from the experience of several shops. To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue. Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient. I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.
- tybit 9y ago
  I think that depends on the code base, I agree with your points for more procedural code bases, but for functional or OO code you should be able to test most of your logic without mocks or wiring too much up.
  - zer0tonin 9y ago
    You usually still have to mock your data layer.
- rimliu 9y ago
  Sandy Metz's talk abou what should be tested and that should not: https://www.youtube.com/watch?v=URSWYvyc42M https://www.youtube.com/watch?v=URSWYvyc42M
- UK-AL 9y ago
  Apprantly people misunderstand a unit. Apprantly Kent beck meant unit to mean a single piece of behaviour, not classes or methods. https://www.thoughtworks.com/insights/blog/mockists-are-dead-long-live-classicists https://www.thoughtworks.com/insights/blog/mockists-are-dead...
  - flukus 9y ago
    > Apprantly Kent beck meant unit to mean a single piece of behaviour Which is just as ambiguous. Personally I stick with the single assert principal (which may be more than one literal assert) so that whatever I'm asserting is the behavior that this test is verifying.
- notacoward 9y ago
  I think the problem lies with the definition of "unit" which you quote above. As with designs stacking on top of one another at various levels, one level's functional test is often the next level's unit test. Your app's unit test might mock the very same things that are covered by the functional test for an underlying library. That library's unit tests might mock the very same things that are covered by the functional tests for one of its own dependencies, including the OS. And so on, even down to the level of functional units within a chip. (Chip verifiers have a lot to teach us software types about this kind of thing BTW, and I know because I've worked with a few.) The distinction I always try to make is whether you're testing a contract or an implementation. If you're testing a contract, ROI is likely to remain high even as you move to finder granularities. If you're testing an implementation, so that the tests fail even when the implementation is 100% correct, then ROI falls off a cliff. That's exactly what happens when the "unit" gets too small. If certain private methods can't be called a certain way because of constraints imposed by the class's public methods, then testing those calls is testing things that can't happen. That takes time away from testing (and implementing fixes for) things that can happen. If the class's contract changes so that those internal methods can be called in new ways, then yes, a 100% code-coverage unit test might catch the error. So, in all likelihood, would the new functional test accompanying the public-method change. The delta between the two, times the likelihood of such a scenario occurring in the first place, is too small to justify the cost of writing those tests and the likelihood that they'll generate false negatives.
- frubar 9y ago
  >Unit testing: Test all methods and paths of a class, even private ones. I disagree with this. Unit testing means testing the "unit" (i.e. the class/object) but still at a boundary level: the public API of that class. You should mock out any dependencies that object would have to ensure you are only testing that class but the class API has to be respected. A private method is a hidden implementation detail, testing those would be overfitting your tests to the current implementation meaning if you change one character anywhere in your source you will almost certainly have to change one or more tests. Plus, if your tests are so tightly coupled to the implementation, it's likely to suffer from any bugs the implementation does causing them to be hidden (never forget that tests are also code and therefor have bugs at a similar rate to any other code). Writing test code requires the same level (if not more) of engineering discipline that writing the code does. >ROI of functional testing is high, as it is usually done with real data. I also disagree with all of this. Functional testing is a kind of sanity check that the parts actually work together once assembled. If you have proper unit test coverage (and properly designed/engineered tests!) the functional testing is basically checking configuration. The problems with function testing are (1) testing is about checking code paths but which paths functional tests take can be hard to predict and differ between subsequent runs, making it hard to make any statement about what passing actually means. (2) is exactly what you mentioned as a positive: people tend to want to use what they call "real" data, i.e. data they have actually seen before. Which means it's probably only good for catching bugs they know about, not ones they've never seen before. >In my opinion unit testing is a huge waste of time. I would actually agree to this because I think answering the question of "how do we detect bad code" with "write more code" is problematic. I'd rather go the Haskell/Idris route and be able to prove that I have no bugs. >Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output. This kind of response sounds like a self fulling prophecy. The exact point of mocking is to test code paths, i.e. if this dependency returns this result how will the class under inspection behave in response. Ideally you would use mocks to test everything that every dependency could respond. Unless the types involved have very few inhabitants, this isn't generally possible (even programatically) but the closer you get to this the more you can trust your tests.
TeMPOraL 9y ago
> However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception. An example of that exception: just yesterday I wrote tests for a serialization/deserialization utility, that translates between an object tree and a wire format used by our client. There was some tricky code around the "deserialize message into one of many possible objects" part, so I wrote a bunch of unit tests for the whole thing. I find myself writing unit tests mostly for this kind of code - complex logic that transform data and/or execute "advanced" algorithms on it. I also write tests as close as possible to the boundaries of the complex logic - this way, when I'm sure the logic itself is sound, all integration bugs tend to be trivial to notice (it means someone fucked up the inputs or didn't handle outputs properly). The more I think of it, the more I notice that I tend to structure my programs in a functional (as in functional programming) way - lots of "services" that take things as inputs and return them as outputs, without using any external state. So e.g. that serialization/deserialization service mentioned only takes a String as input, and returns a reference to a base-class object as an output (or the other way around for serialization). Making code conform to functional style makes it not only easier to test, it helps avoid some tests entirely.
sqeaky 9y ago
> breaks whenever a minor internal detail changes. This is usually where I draw the line for what to test. I don't care how a unit does what it does, I only care about what it does. In my opinion a unit should have minimum side effects, none if possible, and the results should depend on the inputs. Write the test to the contract of what the caller should provide and caller will get back (and then any side that had to happening, like checking logs.)

ardit33 9y ago

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.

camus2 9y ago
> even private ones. You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members. Now the use of the private member might lead to a different code path : it has to be tested. Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked. Functional Testing : test multiple unit at the same time to check the behavior of an entire functionality, that's why it is called functional testing. Stub the IO. End-to-end testing: Test the entire system with the IO.
- bpicolo 9y ago
  > it makes absolutely no sense to test private members Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.
  - mdpopescu 9y ago
    > I've never seen a real case for excluding private methods as a testable unit. Here's the reason: We're not writing tests; we're writing SPECIFICATIONS. Private methods are implementation details; they're not part of the specification.
    - bpicolo 9y ago
      That's potentially one philosophy on it, sure. Yet that still doesn't preclude private methods from being an independently testable unit. Implementation details are the meat of the whole thing, and so they seem inherently testworthy. The glue code itself, less so. Thanks for that thought though, definitely something I'll think on :)
      - clay_to_n 9y ago
        You should be able to get the code coverage you need (even if 100%) by testing the public API of any class. If this isn't true, it means you have some private methods you can delete. That's how the unit you're testing is used - through it's public API. The public API is the specification of how it works. By only testing the public API, you allow yourself maximum ability to refactor in the future, while still maximizing code coverage. It means that simple refactoring (inlining methods, for example) won't break tests. More importantly, a failing test means something is wrong that is potentially relied upon elsewhere. If you test private methods, you will get test failures without the public API of the unit having changed at all. Your point about "each function is a unit" is fine, you can justify testing private methods with that - but it's inefficient. If it's not necessary to be in the spec (public API), why have you made it so? You're over-complicating the design by locking yourself into implementation details in places where you don't need to. In my experience I've found that testing private methods directly is a code smell. It shouldn't be necessary.
phlakaton 9y ago
Some counterpoints: - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. - If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful. - Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up. I think any test strategy – unit, integration, e2e, acceptance, UI – pursued to exclusion is a bad idea. Different projects and different teams call for different balances between them.
- algesten 9y ago
  > - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. I think we talk about the same thing, sometimes I will test something internal. I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.
  - phlakaton 9y ago
    I'm kind of flexible on my terminology, perhaps more flexible than others, since my approach to testing has largely been self-taught from the experience of several shops. To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue. Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient. I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.
tybit 9y ago
I think that depends on the code base, I agree with your points for more procedural code bases, but for functional or OO code you should be able to test most of your logic without mocks or wiring too much up.
- zer0tonin 9y ago
  You usually still have to mock your data layer.
rimliu 9y ago
Sandy Metz's talk abou what should be tested and that should not: https://www.youtube.com/watch?v=URSWYvyc42M https://www.youtube.com/watch?v=URSWYvyc42M
UK-AL 9y ago
Apprantly people misunderstand a unit. Apprantly Kent beck meant unit to mean a single piece of behaviour, not classes or methods. https://www.thoughtworks.com/insights/blog/mockists-are-dead-long-live-classicists https://www.thoughtworks.com/insights/blog/mockists-are-dead...
- flukus 9y ago
  > Apprantly Kent beck meant unit to mean a single piece of behaviour Which is just as ambiguous. Personally I stick with the single assert principal (which may be more than one literal assert) so that whatever I'm asserting is the behavior that this test is verifying.
notacoward 9y ago
I think the problem lies with the definition of "unit" which you quote above. As with designs stacking on top of one another at various levels, one level's functional test is often the next level's unit test. Your app's unit test might mock the very same things that are covered by the functional test for an underlying library. That library's unit tests might mock the very same things that are covered by the functional tests for one of its own dependencies, including the OS. And so on, even down to the level of functional units within a chip. (Chip verifiers have a lot to teach us software types about this kind of thing BTW, and I know because I've worked with a few.) The distinction I always try to make is whether you're testing a contract or an implementation. If you're testing a contract, ROI is likely to remain high even as you move to finder granularities. If you're testing an implementation, so that the tests fail even when the implementation is 100% correct, then ROI falls off a cliff. That's exactly what happens when the "unit" gets too small. If certain private methods can't be called a certain way because of constraints imposed by the class's public methods, then testing those calls is testing things that can't happen. That takes time away from testing (and implementing fixes for) things that can happen. If the class's contract changes so that those internal methods can be called in new ways, then yes, a 100% code-coverage unit test might catch the error. So, in all likelihood, would the new functional test accompanying the public-method change. The delta between the two, times the likelihood of such a scenario occurring in the first place, is too small to justify the cost of writing those tests and the likelihood that they'll generate false negatives.
frubar 9y ago
>Unit testing: Test all methods and paths of a class, even private ones. I disagree with this. Unit testing means testing the "unit" (i.e. the class/object) but still at a boundary level: the public API of that class. You should mock out any dependencies that object would have to ensure you are only testing that class but the class API has to be respected. A private method is a hidden implementation detail, testing those would be overfitting your tests to the current implementation meaning if you change one character anywhere in your source you will almost certainly have to change one or more tests. Plus, if your tests are so tightly coupled to the implementation, it's likely to suffer from any bugs the implementation does causing them to be hidden (never forget that tests are also code and therefor have bugs at a similar rate to any other code). Writing test code requires the same level (if not more) of engineering discipline that writing the code does. >ROI of functional testing is high, as it is usually done with real data. I also disagree with all of this. Functional testing is a kind of sanity check that the parts actually work together once assembled. If you have proper unit test coverage (and properly designed/engineered tests!) the functional testing is basically checking configuration. The problems with function testing are (1) testing is about checking code paths but which paths functional tests take can be hard to predict and differ between subsequent runs, making it hard to make any statement about what passing actually means. (2) is exactly what you mentioned as a positive: people tend to want to use what they call "real" data, i.e. data they have actually seen before. Which means it's probably only good for catching bugs they know about, not ones they've never seen before. >In my opinion unit testing is a huge waste of time. I would actually agree to this because I think answering the question of "how do we detect bad code" with "write more code" is problematic. I'd rather go the Haskell/Idris route and be able to prove that I have no bugs. >Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output. This kind of response sounds like a self fulling prophecy. The exact point of mocking is to test code paths, i.e. if this dependency returns this result how will the class under inspection behave in response. Ideally you would use mocks to test everything that every dependency could respond. Unless the types involved have very few inhabitants, this isn't generally possible (even programatically) but the closer you get to this the more you can trust your tests.

TeMPOraL 9y ago

> However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception. An example of that exception: just yesterday I wrote tests for a serialization/deserialization utility, that translates between an object tree and a wire format used by our client. There was some tricky code around the "deserialize message into one of many possible objects" part, so I wrote a bunch of unit tests for the whole thing. I find myself writing unit tests mostly for this kind of code - complex logic that transform data and/or execute "advanced" algorithms on it. I also write tests as close as possible to the boundaries of the complex logic - this way, when I'm sure the logic itself is sound, all integration bugs tend to be trivial to notice (it means someone fucked up the inputs or didn't handle outputs properly). The more I think of it, the more I notice that I tend to structure my programs in a functional (as in functional programming) way - lots of "services" that take things as inputs and return them as outputs, without using any external state. So e.g. that serialization/deserialization service mentioned only takes a String as input, and returns a reference to a base-class object as an output (or the other way around for serialization). Making code conform to functional style makes it not only easier to test, it helps avoid some tests entirely.

sqeaky 9y ago

> breaks whenever a minor internal detail changes. This is usually where I draw the line for what to test. I don't care how a unit does what it does, I only care about what it does. In my opinion a unit should have minimum side effects, none if possible, and the results should depend on the inputs. Write the test to the contract of what the caller should provide and caller will get back (and then any side that had to happening, like checking logs.)