8 ms·

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine this barebones use of assertions in tests which are entirely too long, not isolating the test cases properly, or even completely irrelevant to what's (supposedly) being tested! If that's not enough, imagine this nightmare failing not after it has been written, but, let's say 18 months later, while being part of a massive test suite running for a while. All you have is a the name of the test that failed, you look into it to find a 630 lines long test "case" with 22 nondescript assertions along the way. You might know which line failed the test, but not always. And of course debugging the test function line by line doesn't work because the test depends on intricate timing for some reason. The person who wrote this might not be around and now this is your dragon to slay. I think I should stop here before triggering myself any further. Therapy is expensive.

ramraj07 4y ago
So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?
- magic_hamster 4y ago
  Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.
  - Joeboy 4y ago
    I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
    - RHSeeger 4y ago
      > Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
      - bluGill 4y ago
        A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
        RHSeeger 4y ago
        > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
- deleted 4y ago
  [deleted]
- mjul 4y ago
  Just use you best judgement and you will be fine. Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation. Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution. (This formulaic approach is also why pattern books tend to be a dry read).
- robertlagrant 4y ago
  Devious commenter was describing a (normal) scenario where a unit test is not precise. No need to follow up with an aggressive "so what you're saying is".
  - robertlagrant 4y ago
    Previous* hah.
- simplotek 4y ago
  > So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test? You're falling prey to slippery slope fallacy, which at best is specious reasoning. The rationale is easy to understand. Running 100 assertions in a single test renders tests unusable. Running 10 assertions suffers from the same problem. Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. Arguing whether two or three or five assertions should be banned misses the whole point and completely ignores the root cause that led to this guideline.
  - BlargMcLarg 4y ago
    >Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. As if this actually happens in practice, regardless of multiple or single asserts. Anything that isn't non-trivial will at most tell you what doesn't work, but it won't tell you why it doesn't work. Maybe allowing an educated guess when multiple tests fail to function. You want test sets to be user friendly? Start at taking down all this dogmatism and listening to the people as to why they dislike writing tests. We're pushing 'guidelines' (really more like rules) while individuals think to themselves 'F this, Jake's going to complain about something trivial again, and we know these tests do jack-all because our code is a mess and doing anything beyond this simple algorithm is a hell in a handbasket". These discussions are beyond useless when all people do is talk while doing zero to actually tackle the issues of the majority not willing to write tests. "Laziness" is a cop-out.
Kuinox 4y ago
Tech debt can often be fixed by more tech debt instead of fixing the root problem.
FartyMcFarter 4y ago
> You might know which line failed the test, but not always. If that's the case, the test framework itself is severely flawed and needs fixing even more than the tests do. There's no excuse to have an assert function that doesn't print out the location of the failure.
- magic_hamster 4y ago
  Even if the framework is fine, you can see something like an elaborate if-else tree, or even a try-catch block, and after it's all done, there's a condition check with `fail()`. So the point of failure could be manually detached from the actual point of failure. Granted, this is not the way to do things. But it happens anyway.
  - mynameisvlad 4y ago
    I mean in your example it’s someone choosing not to use asserts. Which is a problem, don’t get me wrong, but it’s not the problem being talked about here. The comment thread is about “Assertion Roulette” — having so many assertions you don’t know which went off. Which really seems like a test framework issue more than a test issue.
ethbr0 4y ago
> ... where this is not only possible, but the common case. Couldn't not read that in Peter Sellers' voice https://m.youtube.com/watch?v=2yfXgu37iyI&t=2m36s https://m.youtube.com/watch?v=2yfXgu37iyI&t=2m36s > ... to find a 630 lines long test "case" with 22 nondescript assertions along the way. This is where tech team managers are abrogating their responsibility and job. It's the job of the organization to set policy standards to outlaw things like this. It's the job of the developer to cut as many corners of those policies as possible to ship code ASAP. And it's the job of a tech team manager to set up a detailed but efficient process (code review sign offs!) that paper over the gap between the two in a sane way. ... none of which helps immediately with a legacy codebase that's @$&@'d, though.
- RHSeeger 4y ago
  > It's the job of the developer to cut as many corners of those policies as possible to ship code ASAP. I can't tell if this is supposed to be humor, or if you actually believe it. It's certainly not my job as a developer to ship worse code so that I can release it ASAP. Rather, it's my job to push back against ASAP where it conflicts with writing better code.
  - ethbr0 4y ago
    You are not most developers. And furthermore, you are not the developer most non-tech companies want. Those sorts of companies want to lock the door to the development section, occasionally slide policy from memos under the door, and get software projects delivered on time, without wasting any more thought on how the sausage gets made.
marcosdumay 4y ago
> Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Well, avoiding ecosystems where people act dumb is a sure way to improve one's life. For a start, you won't need to do stupid things in reaction of your tools. Yes, it's not always possible. But the practices you create for surviving it are part of the dumb ecosystem survival kit, not part of any best practices BOK.
mrits 4y ago
22 assertions in a test is a lot better than 22 separate tests that fail for the same reason.
- bluGill 4y ago
  They are the same. I don't care, as I'll fix them one at a time, and if the fix happens to fix more than one great.
  - mrits 4y ago
    Fixing one at a time like that is a good way to get into an endless cycle. Zoom out a bit and make a plan before you start coding.
    - bluGill 4y ago
      That should be understood wothout saying. Even with a plan I'm going to fix one at a time.
- Jabbles 4y ago
  With 22 separate tests you have the possibility of knowing that only a subset of them fail. Knowing which fail and which pass may help you debug. In Go, in general, tests fail and continue, rather than causing the test to stop early, so you can tell which of those 22 checks failed. Other languages may have the option to do something similar. https://pkg.go.dev/testing#T.Fail https://pkg.go.dev/testing#T.Fail

ramraj07 4y ago

So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?

magic_hamster 4y ago
Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.
- Joeboy 4y ago
  I guess you might be talking about a different language / environment than I'm used to, but even in the 100 assertion test case you get useful tracebacks in python. Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test, but still I don't see how it's a bad test. It's easy and stops buggy PRs going into production. The test failures I see that are actually hard to debug are ones where the failures are difficult to reproduce due to random input, tests running in parallel and sharing the same filesystem etc. I don't think I've ever not known what assert was failing (although I guess in theory you could make that happen by catching AssertionError).
  - RHSeeger 4y ago
    > Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.
    - bluGill 4y ago
      A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the bug is in the last thing you changed. Sure the test covers hundreds of lines, but you only changed one.
      - RHSeeger 4y ago
        > The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.
deleted 4y ago
[deleted]
mjul 4y ago
Just use you best judgement and you will be fine. Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation. Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution. (This formulaic approach is also why pattern books tend to be a dry read).
robertlagrant 4y ago
Devious commenter was describing a (normal) scenario where a unit test is not precise. No need to follow up with an aggressive "so what you're saying is".
- robertlagrant 4y ago
  Previous* hah.
simplotek 4y ago
> So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test? You're falling prey to slippery slope fallacy, which at best is specious reasoning. The rationale is easy to understand. Running 100 assertions in a single test renders tests unusable. Running 10 assertions suffers from the same problem. Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. Arguing whether two or three or five assertions should be banned misses the whole point and completely ignores the root cause that led to this guideline.
- BlargMcLarg 4y ago
  >Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs. As if this actually happens in practice, regardless of multiple or single asserts. Anything that isn't non-trivial will at most tell you what doesn't work, but it won't tell you why it doesn't work. Maybe allowing an educated guess when multiple tests fail to function. You want test sets to be user friendly? Start at taking down all this dogmatism and listening to the people as to why they dislike writing tests. We're pushing 'guidelines' (really more like rules) while individuals think to themselves 'F this, Jake's going to complain about something trivial again, and we know these tests do jack-all because our code is a mess and doing anything beyond this simple algorithm is a hell in a handbasket". These discussions are beyond useless when all people do is talk while doing zero to actually tackle the issues of the majority not willing to write tests. "Laziness" is a cop-out.

Kuinox 4y ago

Tech debt can often be fixed by more tech debt instead of fixing the root problem.

FartyMcFarter 4y ago

> You might know which line failed the test, but not always. If that's the case, the test framework itself is severely flawed and needs fixing even more than the tests do. There's no excuse to have an assert function that doesn't print out the location of the failure.

magic_hamster 4y ago
Even if the framework is fine, you can see something like an elaborate if-else tree, or even a try-catch block, and after it's all done, there's a condition check with `fail()`. So the point of failure could be manually detached from the actual point of failure. Granted, this is not the way to do things. But it happens anyway.
- mynameisvlad 4y ago
  I mean in your example it’s someone choosing not to use asserts. Which is a problem, don’t get me wrong, but it’s not the problem being talked about here. The comment thread is about “Assertion Roulette” — having so many assertions you don’t know which went off. Which really seems like a test framework issue more than a test issue.

ethbr0 4y ago

> ... where this is not only possible, but the common case. Couldn't not read that in Peter Sellers' voice https://m.youtube.com/watch?v=2yfXgu37iyI&t=2m36s https://m.youtube.com/watch?v=2yfXgu37iyI&t=2m36s > ... to find a 630 lines long test "case" with 22 nondescript assertions along the way. This is where tech team managers are abrogating their responsibility and job. It's the job of the organization to set policy standards to outlaw things like this. It's the job of the developer to cut as many corners of those policies as possible to ship code ASAP. And it's the job of a tech team manager to set up a detailed but efficient process (code review sign offs!) that paper over the gap between the two in a sane way. ... none of which helps immediately with a legacy codebase that's @$&@'d, though.

RHSeeger 4y ago
> It's the job of the developer to cut as many corners of those policies as possible to ship code ASAP. I can't tell if this is supposed to be humor, or if you actually believe it. It's certainly not my job as a developer to ship worse code so that I can release it ASAP. Rather, it's my job to push back against ASAP where it conflicts with writing better code.
- ethbr0 4y ago
  You are not most developers. And furthermore, you are not the developer most non-tech companies want. Those sorts of companies want to lock the door to the development section, occasionally slide policy from memos under the door, and get software projects delivered on time, without wasting any more thought on how the sausage gets made.

marcosdumay 4y ago

> Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Well, avoiding ecosystems where people act dumb is a sure way to improve one's life. For a start, you won't need to do stupid things in reaction of your tools. Yes, it's not always possible. But the practices you create for surviving it are part of the dumb ecosystem survival kit, not part of any best practices BOK.

mrits 4y ago

22 assertions in a test is a lot better than 22 separate tests that fail for the same reason.

bluGill 4y ago
They are the same. I don't care, as I'll fix them one at a time, and if the fix happens to fix more than one great.
- mrits 4y ago
  Fixing one at a time like that is a good way to get into an endless cycle. Zoom out a bit and make a plan before you start coding.
  - bluGill 4y ago
    That should be understood wothout saying. Even with a plan I'm going to fix one at a time.
Jabbles 4y ago
With 22 separate tests you have the possibility of knowing that only a subset of them fail. Knowing which fail and which pass may help you debug. In Go, in general, tests fail and continue, rather than causing the test to stop early, so you can tell which of those 22 checks failed. Other languages may have the option to do something similar. https://pkg.go.dev/testing#T.Fail https://pkg.go.dev/testing#T.Fail