5 ms·

So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?

magic_hamster 4y ago
Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.
- Joeboy 4y ago
  I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
  - RHSeeger 4y ago
    > Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
    - bluGill 4y ago
      A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
      - RHSeeger 4y ago
        > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
        dmitriid 4y ago
        > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
        RHSeeger 4y ago
        >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
- jcelerier 4y ago
  Why ? In c++ if I have a test case with REQUIRES(v1::foo(0) == v2::foo(0)); REQUIRES(v1::foo(1) == v2::foo(1)); And the second assert fails the error message will tell me exactly that, the line, and the value of both function calls if they are printable. What more do you want to know "what actually happened"?
- jjav 4y ago
  > when the test fails it will be harder to know what actually happened This should not ever be possible in any semi-sane test environment. One could in theory write a single test function with thousands of asserts for all kinds of conditions and it still should be 100% obvious which one failed when something fails. Not that I'd suggest going to that extreme either, but it illustrates that it'll work fine.
- ambrozk 4y ago
  > when the test fails it will be harder to know what actually happened. Yeah, and if you write one assertion at a time, it will be harder to write the tests. Decreasing #assertions/test decreases the speed of test debugging while increasing the time spent writing non-production code. It's a tradeoff. Declaring that the optimal number of assertions per test is 1 completely ignores the reality of this tradeoff.
  - magic_hamster 4y ago
    That's true and it boils down to what's acceptable in your team (or just you). I worked in some places where coverage was the only metric and in places where every single function had to have all cases covered, and testing took longer than writing the code. As for me, I tend to write reasonable tests and cover several cases that guard the intended behavior of each function (if someone decides the function should behave differently in the future, a test should fail). One emerging pattern is that sometimes during testing I realize I need to refactor something, which might have been lost on me if I skimmed on tests. It's both a sanity check and a guardrail for future readers.
deleted 4y ago
[deleted]
mjul 4y ago
Just use you best judgement and you will be fine. Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation. Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution. (This formulaic approach is also why pattern books tend to be a dry read).
robertlagrant 4y ago
Devious commenter was describing a (normal) scenario where a unit test is not precise. No need to follow up with an aggressive "so what you're saying is".
- robertlagrant 4y ago
  Previous* hah.
simplotek 4y ago
> So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test? You're falling prey to slippery slope fallacy, which at best is specious reasoning. The rationale is easy to understand. Running 100 assertions in a single test renders tests unusable. Running 10 assertions suffers from the same problem. Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. Arguing whether two or three or five assertions should be banned misses the whole point and completely ignores the root cause that led to this guideline.
- BlargMcLarg 4y ago
  >Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. As if this actually happens in practice, regardless of multiple or single asserts. Anything that isn't non-trivial will at most tell you what doesn't work, but it won't tell you why it doesn't work. Maybe allowing an educated guess when multiple tests fail to function. You want test sets to be user friendly? Start at taking down all this dogmatism and listening to the people as to why they dislike writing tests. We're pushing 'guidelines' (really more like rules) while individuals think to themselves 'F this, Jake's going to complain about something trivial again, and we know these tests do jack-all because our code is a mess and doing anything beyond this simple algorithm is a hell in a handbasket". These discussions are beyond useless when all people do is talk while doing zero to actually tackle the issues of the majority not willing to write tests. "Laziness" is a cop-out.

magic_hamster 4y ago

Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.

Joeboy 4y ago
I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
- RHSeeger 4y ago
  > Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
  - bluGill 4y ago
    A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
    - RHSeeger 4y ago
      > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
      - dmitriid 4y ago
        > this is the smallest thing I _can_ test usefully Then you're testing useless things. Usefulness is when different parts of a program work together as a coherent whole. Testing DB access layer and service layer separately (as units are often defined) has no meaning (but is often enforced). Queue in memes about "unit tests with 100% code coverage, no integration tests" https://mobile.twitter.com/thepracticaldev/status/687672086152753152 https://mobile.twitter.com/thepracticaldev/status/6876720861...
        RHSeeger 4y ago
        >> this is the smallest thing I _can_ test usefully > Then you're testing useless things. We'll have to agree to disagree then. > Testing DB access layer and service layer separately (as units are often defined) Not at all. For me, a unit is a small part of a layer; one method. Testing the various parts in one system/layer is another type of test. Testing that different systems work together is yet another. I tend to think in terms of the following - Unit test = my code works - Functional test = my design works - Integration test = my code is using your 3rd party stuff correctly (databases, etc) - Factory Acceptance Test = my system works - Site Acceptance Test = your code sucks, this totally isn't what I asked for!?! The "my code works" part is the smallest piece possible. Think "the sorting function" of a library that can return it's results sorted in a specific order.
jcelerier 4y ago
Why ? In c++ if I have a test case with REQUIRES(v1::foo(0) == v2::foo(0)); REQUIRES(v1::foo(1) == v2::foo(1)); And the second assert fails the error message will tell me exactly that, the line, and the value of both function calls if they are printable. What more do you want to know "what actually happened"?
jjav 4y ago
> when the test fails it will be harder to know what actually happened This should not ever be possible in any semi-sane test environment. One could in theory write a single test function with thousands of asserts for all kinds of conditions and it still should be 100% obvious which one failed when something fails. Not that I'd suggest going to that extreme either, but it illustrates that it'll work fine.
ambrozk 4y ago
> when the test fails it will be harder to know what actually happened. Yeah, and if you write one assertion at a time, it will be harder to write the tests. Decreasing #assertions/test decreases the speed of test debugging while increasing the time spent writing non-production code. It's a tradeoff. Declaring that the optimal number of assertions per test is 1 completely ignores the reality of this tradeoff.
- magic_hamster 4y ago
  That's true and it boils down to what's acceptable in your team (or just you). I worked in some places where coverage was the only metric and in places where every single function had to have all cases covered, and testing took longer than writing the code. As for me, I tend to write reasonable tests and cover several cases that guard the intended behavior of each function (if someone decides the function should behave differently in the future, a test should fail). One emerging pattern is that sometimes during testing I realize I need to refactor something, which might have been lost on me if I skimmed on tests. It's both a sanity check and a guardrail for future readers.

deleted 4y ago

[deleted]

mjul 4y ago

Just use you best judgement and you will be fine. Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation. Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution. (This formulaic approach is also why pattern books tend to be a dry read).

robertlagrant 4y ago

Devious commenter was describing a (normal) scenario where a unit test is not precise. No need to follow up with an aggressive "so what you're saying is".

robertlagrant 4y ago
Previous* hah.

simplotek 4y ago

> So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test? You're falling prey to slippery slope fallacy, which at best is specious reasoning. The rationale is easy to understand. Running 100 assertions in a single test renders tests unusable. Running 10 assertions suffers from the same problem. Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. Arguing whether two or three or five assertions should be banned misses the whole point and completely ignores the root cause that led to this guideline.

BlargMcLarg 4y ago
>Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. As if this actually happens in practice, regardless of multiple or single asserts. Anything that isn't non-trivial will at most tell you what doesn't work, but it won't tell you why it doesn't work. Maybe allowing an educated guess when multiple tests fail to function. You want test sets to be user friendly? Start at taking down all this dogmatism and listening to the people as to why they dislike writing tests. We're pushing 'guidelines' (really more like rules) while individuals think to themselves 'F this, Jake's going to complain about something trivial again, and we know these tests do jack-all because our code is a mess and doing anything beyond this simple algorithm is a hell in a handbasket". These discussions are beyond useless when all people do is talk while doing zero to actually tackle the issues of the majority not willing to write tests. "Laziness" is a cop-out.