5 ms·

Maybe I'm misunderstanding this, but assuming you require all tests to pass there should be no more code coverage by using "soft assertions", because no asserti

Maybe I'm misunderstanding this, but assuming you require all tests to pass there should be no more code coverage by using "soft assertions", because no assertions have thrown in your tests. The code re-use benefits don't seem that substantial. I can see this being useful in the development cycle to see more failures that are happening, but there's an overhead to structuring a test that can fail at multiple points – early exits are easy to write. Maybe I'm missing the point, but I'm not sure I agree that this is something we should be striving for. It doesn't seem to solve problems I have, and seems to introduce additional complexity.

trumpeta 6y ago
I’d say when you run a big suite in CI and it takes an hour and your first assertion of many in a test fails, you don’t know if the others also fail or not. You fix it only to wait another hour for the tests to fail again.
- danpalmer 6y ago
  I can imagine a test _suite_ taking hours, but each test should be a small fraction of that surely? And each assertion fails just the test that it's in as tests need to be isolated to be effective?
  - r0s 6y ago
    If you architect each test to have a single assertion, yes, I agree it's essentially the same effect.
- rightbyte 6y ago
  I mean you have to run the tests again anyway. If the test is in a invalid state then you can't trust soft asserts that pass. It might be meaningfull but fast aborts is easier to reason about. Even CUnit has specific test invocations via eg. an interactive ncurses session, so any fancy pancy Jenkins monster surely can retest failing tests seperately?
  - r0s 6y ago
    > If the test is in a invalid state then you can't trust soft asserts that pass. Agreed, which is why I suggest designing each test for a single target state. That target may then legitimately benefit from multiple assertions, and you have the same problem.
    - rightbyte 6y ago
      Ye at-least for projects with many developers involved. My experience is that tests that tests too much makes QA collapse when the original authors of the tests are gone and no-one know what tests are actually testing implementation or spec. I use #ifdef:s to enable "intermediate state" and implementation asserts usually to mark them as development tests.

trumpeta 6y ago

I’d say when you run a big suite in CI and it takes an hour and your first assertion of many in a test fails, you don’t know if the others also fail or not. You fix it only to wait another hour for the tests to fail again.

danpalmer 6y ago
I can imagine a test _suite_ taking hours, but each test should be a small fraction of that surely? And each assertion fails just the test that it's in as tests need to be isolated to be effective?
- r0s 6y ago
  If you architect each test to have a single assertion, yes, I agree it's essentially the same effect.
rightbyte 6y ago
I mean you have to run the tests again anyway. If the test is in a invalid state then you can't trust soft asserts that pass. It might be meaningfull but fast aborts is easier to reason about. Even CUnit has specific test invocations via eg. an interactive ncurses session, so any fancy pancy Jenkins monster surely can retest failing tests seperately?
- r0s 6y ago
  > If the test is in a invalid state then you can't trust soft asserts that pass. Agreed, which is why I suggest designing each test for a single target state. That target may then legitimately benefit from multiple assertions, and you have the same problem.
  - rightbyte 6y ago
    Ye at-least for projects with many developers involved. My experience is that tests that tests too much makes QA collapse when the original authors of the tests are gone and no-one know what tests are actually testing implementation or spec. I use #ifdef:s to enable "intermediate state" and implementation asserts usually to mark them as development tests.