5 ms·

I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in pyt

by Joeboy 4y ago

I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).

RHSeeger 4y ago
> Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
- bluGill 4y ago
  A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
  - RHSeeger 4y ago
    > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
    - dmitriid 4y ago
      > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
      - RHSeeger 4y ago
        >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
        dmitriid 4y ago
        And the only actual useful tests are functional (depending on how you write them) and above. If those fail, it means that neither your design nor your code works. The absolute vast majority of unit tests are meaningless because you just repeat them again in the higher level tests.
        RHSeeger 4y ago
        That seems like a silly opinion to me. I use unit tests to make sure that individual units work like I expect them to. And I use them to test edge cases that can be tested separately from their caller. If I had to test all the use cases for each function, all combined together, there number of tests would grow by the multiplication of the partitions of each one, N x M x O x P, ... rather than the sum, plus a much smaller set of tests for how they work together (N + M + O + P + N_M + M_O + O_P, etc). It's much simpler to thoroughly test each unit. Then test how they work together.
  - __alexs 4y ago
    I recently went to the effort of trying to work out where the term unit test came from in some desperate effort to find what a unit was meant to be. After much googling and buying or ancient text books I hit a dead end. At this point I think "unit" is just noise that confuses people into making distinctions that don't exist.
    - switchbak 4y ago
      As I recall the TDD mailing list has some background on the use of the word "unit", it goes WAY back, I believe it goes back to the mainframe/ punch card era. Regardless, I think it roughly translates to C's notion of the unit of compilation. Which is obviously not what people really mean these days, but the phrase stuck. The early Xp'ers even found it an issue back then. For a while people tried to push the term "micro tests", but that didn't really take off. I agree with Gerard Mezaros and Martin Fowler and typically follow their (very mainstream) definitions on this stuff. Integration and functional testing have their own ambiguities too, it's definitely a frustrating situation to not have solidly defined foundational terms.
    - Izkata 4y ago
      IIRC the "unit" in "unit test" was meant to mean "semantic unit" ("the access module", for example, should be distinct with a well-defined interface that all the tests go through), but very quickly turned into "syntactic units" ("a single function", for example, where the "well-defined interface" ends up just being function arguments/return value) because most people didn't understand what the original proponents meant.
      - branko_d 4y ago
        I have a Web API which calls a DB API which calls a stored procedure which executes several SQL statements. The Web API has a well-defined and documented interface. Is it a “unit”?
        Izkata 4y ago
        In the semantic sense? No, or at least probably not if that's where you jump to. You're still thinking in terms of syntax. Think in terms of business rules, not the code structure: What's one thing your API does?
  - JackFr 4y ago
    > It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one. That’s not true. A correct change might expose an existing bug which hadn’t been tested or expose flaky behavior which existed but hadn’t been exercised. In both cases the solution is not to revert the correct change, but to fix the buggy behavior.
  - sdenton4 4y ago
    Michael Pollan was right: Write tests, not too many, mostly integration.
    - teddyh 4y ago
      Contrary viewpoint: Integrated Tests Are A Scam (J.B. Rainsberger): https://www.youtube.com/watch?v=fhFa4tkFUFw https://www.youtube.com/watch?v=fhFa4tkFUFw
      - sdenton4 4y ago
        Watched a bit of this... It's typical test-driven zealotry; the main criticism of integration tests seems to be that they don't force your hand in system design in the way that unit tests do? Which seems very silly, but then, I'm not a person who goes to conferences about testing philosophy.
        switchbak 4y ago
        Did you miss his follow-up? "Integration tests are a scam is a scam". For real. I like J.B., but I think he muddies the water too much and overall understanding suffers.

by Joeboy 4y ago

RHSeeger 4y ago

> Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.

bluGill 4y ago
A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
- RHSeeger 4y ago
  > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
  - dmitriid 4y ago
    > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
    - RHSeeger 4y ago
      >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
      - dmitriid 4y ago
        And the only actual useful tests are functional (depending on how you write them) and above. If those fail, it means that neither your design nor your code works. The absolute vast majority of unit tests are meaningless because you just repeat them again in the higher level tests.
        RHSeeger 4y ago
        That seems like a silly opinion to me. I use unit tests to make sure that individual units work like I expect them to. And I use them to test edge cases that can be tested separately from their caller. If I had to test all the use cases for each function, all combined together, there number of tests would grow by the multiplication of the partitions of each one, N x M x O x P, ... rather than the sum, plus a much smaller set of tests for how they work together (N + M + O + P + N_M + M_O + O_P, etc). It's much simpler to thoroughly test each unit. Then test how they work together.
- __alexs 4y ago
  I recently went to the effort of trying to work out where the term unit test came from in some desperate effort to find what a unit was meant to be. After much googling and buying or ancient text books I hit a dead end. At this point I think "unit" is just noise that confuses people into making distinctions that don't exist.
  - switchbak 4y ago
    As I recall the TDD mailing list has some background on the use of the word "unit", it goes WAY back, I believe it goes back to the mainframe/ punch card era. Regardless, I think it roughly translates to C's notion of the unit of compilation. Which is obviously not what people really mean these days, but the phrase stuck. The early Xp'ers even found it an issue back then. For a while people tried to push the term "micro tests", but that didn't really take off. I agree with Gerard Mezaros and Martin Fowler and typically follow their (very mainstream) definitions on this stuff. Integration and functional testing have their own ambiguities too, it's definitely a frustrating situation to not have solidly defined foundational terms.
  - Izkata 4y ago
    IIRC the "unit" in "unit test" was meant to mean "semantic unit" ("the access module", for example, should be distinct with a well-defined interface that all the tests go through), but very quickly turned into "syntactic units" ("a single function", for example, where the "well-defined interface" ends up just being function arguments/return value) because most people didn't understand what the original proponents meant.
    - branko_d 4y ago
      I have a Web API which calls a DB API which calls a stored procedure which executes several SQL statements. The Web API has a well-defined and documented interface. Is it a “unit”?
      - Izkata 4y ago
        In the semantic sense? No, or at least probably not if that's where you jump to. You're still thinking in terms of syntax. Think in terms of business rules, not the code structure: What's one thing your API does?
- JackFr 4y ago
  > It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one. That’s not true. A correct change might expose an existing bug which hadn’t been tested or expose flaky behavior which existed but hadn’t been exercised. In both cases the solution is not to revert the correct change, but to fix the buggy behavior.
- sdenton4 4y ago
  Michael Pollan was right: Write tests, not too many, mostly integration.
  - teddyh 4y ago
    Contrary viewpoint: Integrated Tests Are A Scam (J.B. Rainsberger): https://www.youtube.com/watch?v=fhFa4tkFUFw https://www.youtube.com/watch?v=fhFa4tkFUFw
    - sdenton4 4y ago
      Watched a bit of this... It's typical test-driven zealotry; the main criticism of integration tests seems to be that they don't force your hand in system design in the way that unit tests do? Which seems very silly, but then, I'm not a person who goes to conferences about testing philosophy.
      - switchbak 4y ago
        Did you miss his follow-up? "Integration tests are a scam is a scam". For real. I like J.B., but I think he muddies the water too much and overall understanding suffers.