3 ms·
Captain – CLI tool that can detect and quarantine flaky tests
- 8organicbits 4y ago> Captain can be configured to retry flaky tests more aggressively than new failures. > Captain's quarantining allows you to continue running tests that are known to be flaky or failing while preventing them from failing your builds. This is really clever. Most test frameworks operate under the assumption that a failed test is a full blocker. But in reality, flaky tests are often symptoms of flaky test fixture setup which isn't a strong signal of a real issue. Usually I've seen engineers learn which tests are flaky so they can manually override test failure or fully disable the flaky tests. This looks like a very reasonable middle ground.