6 ms·

Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really

Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.

Joeboy 4y ago
I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
- RHSeeger 4y ago
  > Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
  - bluGill 4y ago
    A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
    - RHSeeger 4y ago
      > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
      - dmitriid 4y ago
        > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
        RHSeeger 4y ago
        >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
        dmitriid 4y ago
        And the only actual useful tests are functional (depending on how you write them) and above. If those fail, it means that neither your design nor your code works. The absolute vast majority of unit tests are meaningless because you just repeat them again in the higher level tests.
jcelerier 4y ago
Why ? In c++ if I have a test case with REQUIRES(v1::foo(0) == v2::foo(0)); REQUIRES(v1::foo(1) == v2::foo(1)); And the second assert fails the error message will tell me exactly that, the line, and the value of both function calls if they are printable. What more do you want to know "what actually happened"?
jjav 4y ago
> when the test fails it will be harder to know what actually happened This should not ever be possible in any semi-sane test environment. One could in theory write a single test function with thousands of asserts for all kinds of conditions and it still should be 100% obvious which one failed when something fails. Not that I'd suggest going to that extreme either, but it illustrates that it'll work fine.
ambrozk 4y ago
> when the test fails it will be harder to know what actually happened. Yeah, and if you write one assertion at a time, it will be harder to write the tests. Decreasing #assertions/test decreases the speed of test debugging while increasing the time spent writing non-production code. It's a tradeoff. Declaring that the optimal number of assertions per test is 1 completely ignores the reality of this tradeoff.
- magic_hamster 4y ago
  That's true and it boils down to what's acceptable in your team (or just you). I worked in some places where coverage was the only metric and in places where every single function had to have all cases covered, and testing took longer than writing the code. As for me, I tend to write reasonable tests and cover several cases that guard the intended behavior of each function (if someone decides the function should behave differently in the future, a test should fail). One emerging pattern is that sometimes during testing I realize I need to refactor something, which might have been lost on me if I skimmed on tests. It's both a sanity check and a guardrail for future readers.

Joeboy 4y ago

I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).

RHSeeger 4y ago
> Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
- bluGill 4y ago
  A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
  - RHSeeger 4y ago
    > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
    - dmitriid 4y ago
      > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
      - RHSeeger 4y ago
        >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
        dmitriid 4y ago
        And the only actual useful tests are functional (depending on how you write them) and above. If those fail, it means that neither your design nor your code works. The absolute vast majority of unit tests are meaningless because you just repeat them again in the higher level tests.

jcelerier 4y ago

Why ? In c++ if I have a test case with REQUIRES(v1::foo(0) == v2::foo(0)); REQUIRES(v1::foo(1) == v2::foo(1)); And the second assert fails the error message will tell me exactly that, the line, and the value of both function calls if they are printable. What more do you want to know "what actually happened"?

jjav 4y ago

> when the test fails it will be harder to know what actually happened This should not ever be possible in any semi-sane test environment. One could in theory write a single test function with thousands of asserts for all kinds of conditions and it still should be 100% obvious which one failed when something fails. Not that I'd suggest going to that extreme either, but it illustrates that it'll work fine.

ambrozk 4y ago

> when the test fails it will be harder to know what actually happened. Yeah, and if you write one assertion at a time, it will be harder to write the tests. Decreasing #assertions/test decreases the speed of test debugging while increasing the time spent writing non-production code. It's a tradeoff. Declaring that the optimal number of assertions per test is 1 completely ignores the reality of this tradeoff.

magic_hamster 4y ago
That's true and it boils down to what's acceptable in your team (or just you). I worked in some places where coverage was the only metric and in places where every single function had to have all cases covered, and testing took longer than writing the code. As for me, I tend to write reasonable tests and cover several cases that guard the intended behavior of each function (if someone decides the function should behave differently in the future, a test should fail). One emerging pattern is that sometimes during testing I realize I need to refactor something, which might have been lost on me if I skimmed on tests. It's both a sanity check and a guardrail for future readers.