4 ms·

We have a test environment with this taken to the extreme at work, i.e. integration tests where the whole codebase softly fails and continues whatever happens.

We have a test environment with this taken to the extreme at work, i.e. integration tests where the whole codebase softly fails and continues whatever happens. IMO the drawbacks are worse than the upside of seeing more errors. Now one errors ends up giving you a lot of failing soft assertions after the first failure which are not useful at all. There is certainly benefit to being able to go through a failure, but it seems better handled at the framework level rather than writing the tests with this in mind. There's already enough unknown in the tests to add possible earlier failing assertions.

r0s 6y ago
Interesting, it sounds like you would benefit from breaking each test out into one that focuses on a single target state. Another layer would be conditional test execution hierarchy where one early failure would skip a set of tests that depend on that state.
- tracnar 6y ago
  I agree, but the tests are in the order of minutes (sometimes hours), not your typical unit tests. That's basically why it uses soft assertions, it's costly to run. Ideally you'd still want tests with a single assertion/target state, but it's hard to write it that way. We also do have this conditional test hierarchy, but again it's hard to properly define when the system under test behaves unexpectedly...
  - r0s 6y ago
    > it's hard to properly define when the system under test behaves unexpectedly... I feel your pain. I think at that point it's good to look deeper and ask if exceptions are being properly thrown, as I try to call for in the final section on Exceptions. You can't plan for everything.
OnlyOneCannolo 6y ago
Seems like the obvious solution is to use hard assertions unless you have a good reason not to.
- tracnar 6y ago
  Indeed that's what I try to advocate. Of course it's not always easy to know if something should be a hard assertion or not until you hit a failure, which is where I think some framework support to run it in a 'soft-assertion-by-default' way would be handy.

r0s 6y ago

Interesting, it sounds like you would benefit from breaking each test out into one that focuses on a single target state. Another layer would be conditional test execution hierarchy where one early failure would skip a set of tests that depend on that state.

tracnar 6y ago
I agree, but the tests are in the order of minutes (sometimes hours), not your typical unit tests. That's basically why it uses soft assertions, it's costly to run. Ideally you'd still want tests with a single assertion/target state, but it's hard to write it that way. We also do have this conditional test hierarchy, but again it's hard to properly define when the system under test behaves unexpectedly...
- r0s 6y ago
  > it's hard to properly define when the system under test behaves unexpectedly... I feel your pain. I think at that point it's good to look deeper and ask if exceptions are being properly thrown, as I try to call for in the final section on Exceptions. You can't plan for everything.

OnlyOneCannolo 6y ago

Seems like the obvious solution is to use hard assertions unless you have a good reason not to.

tracnar 6y ago
Indeed that's what I try to advocate. Of course it's not always easy to know if something should be a hard assertion or not until you hit a failure, which is where I think some framework support to run it in a 'soft-assertion-by-default' way would be handy.