3 ms·

Short answer: Not all failed test coverage is invalid. Granted, unit tests benefit less from soft assertions, but I'll take a shot at an example anyway: Consi

by r0s 6y ago

Short answer: Not all failed test coverage is invalid. Granted, unit tests benefit less from soft assertions, but I'll take a shot at an example anyway: Consider a complex object returned from a unit test result (this would be a single target state). That structure could have a missing field that should be reported, but the rest of the fields are evaluated fine, and so that information would be lost if the first assert halted the test. For functional tests, the value is usually easier to discern, so for another contrived example: A login form has a failed assert on it's UI structure, say a button is the wrong color, but the login is successful, and the following assertion post-login is also successful. Ideally, you would assert post-login tests in separate target state, perhaps mocking the login as a pre-condition to another test. For functional tests that could dramatically increase the testing time and complexity, and so this makes a non-halting assertion valuable.

rgoulter 6y ago
Ah, I misread "test coverage" as "code coverage". Whoops. > Consider a complex object returned from a unit test result (this would be a single target state). That structure could have a missing field that should be reported, but the rest of the fields are evaluated fine, and so that information would be lost if the first assert halted the test. Right, I'd understand this as 'soft' assertions provide an easier way of making a composite assertion. e.g. I think with RSpec, its matches can be composed; its matchers can nicely return whether a field is missing, or a custom matcher can be written. -- Either way, the point is that a test failure results in the signal that's wholly useful. > [...say, for cases where an early part of the test fails in an unimportant way...] Hmm. I think I'd take this suggestion as: test for 'insignificant' things as low as possible; it's expensive to halt a slow functional test over something that doesn't matter. -- Once something that matters fails, that's the information which is useful for the test to signal. And maybe it can be useful to warn about insignificant things that do get picked up during the functional test.
- r0s 6y ago
  I appreciate the conversation! That's interesting about RSpec, I'll read more about it, thanks.

by r0s 6y ago

rgoulter 6y ago

Ah, I misread "test coverage" as "code coverage". Whoops. > Consider a complex object returned from a unit test result (this would be a single target state). That structure could have a missing field that should be reported, but the rest of the fields are evaluated fine, and so that information would be lost if the first assert halted the test. Right, I'd understand this as 'soft' assertions provide an easier way of making a composite assertion. e.g. I think with RSpec, its matches can be composed; its matchers can nicely return whether a field is missing, or a custom matcher can be written. -- Either way, the point is that a test failure results in the signal that's wholly useful. > [...say, for cases where an early part of the test fails in an unimportant way...] Hmm. I think I'd take this suggestion as: test for 'insignificant' things as low as possible; it's expensive to halt a slow functional test over something that doesn't matter. -- Once something that matters fails, that's the information which is useful for the test to signal. And maybe it can be useful to warn about insignificant things that do get picked up during the functional test.

r0s 6y ago
I appreciate the conversation! That's interesting about RSpec, I'll read more about it, thanks.