3 ms·

Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.

vlovich123 3y ago
It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot). https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/ https://engineering.fb.com/2020/12/10/developer-tools/probab...
- erik_seaberg 3y ago
  If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
  - vlovich123 3y ago
    What makes you think that’s not the case?
    - erik_seaberg 3y ago
      Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
      - vlovich123 3y ago
        I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on. > Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.

vlovich123 3y ago

It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot). https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/ https://engineering.fb.com/2020/12/10/developer-tools/probab...

erik_seaberg 3y ago
If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
- vlovich123 3y ago
  What makes you think that’s not the case?
  - erik_seaberg 3y ago
    Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
    - vlovich123 3y ago
      I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on. > Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.