4 ms·
I’d say when you run a big suite in CI and it takes an hour and your first assertion of many in a test fails, you don’t know if the others also fail or not. You
by trumpeta 6y ago
I’d say when you run a big suite in CI and it takes an hour and your first assertion of many in a test fails, you don’t know if the others also fail or not. You fix it only to wait another hour for the tests to fail again.
- danpalmer 6y agoI can imagine a test _suite_ taking hours, but each test should be a small fraction of that surely? And each assertion fails just the test that it's in as tests need to be isolated to be effective?
- r0s 6y agoIf you architect each test to have a single assertion, yes, I agree it's essentially the same effect.
- rightbyte 6y agoI mean you have to run the tests again anyway. If the test is in a invalid state then you can't trust soft asserts that pass. It might be meaningfull but fast aborts is easier to reason about. Even CUnit has specific test invocations via eg. an interactive ncurses session, so any fancy pancy Jenkins monster surely can retest failing tests seperately?
- r0s 6y ago> If the test is in a invalid state then you can't trust soft asserts that pass. Agreed, which is why I suggest designing each test for a single target state. That target may then legitimately benefit from multiple assertions, and you have the same problem.
- rightbyte 6y agoYe at-least for projects with many developers involved. My experience is that tests that tests too much makes QA collapse when the original authors of the tests are gone and no-one know what tests are actually testing implementation or spec. I use #ifdef:s to enable "intermediate state" and implementation asserts usually to mark them as development tests.