3 ms·

I totally agree with you. We have a monorepo with maybe a hundred thousand tests, and we have a nightly build that runs against all of them, and publishes the r

by gen220 6y ago

I totally agree with you. We have a monorepo with maybe a hundred thousand tests, and we have a nightly build that runs against all of them, and publishes the results in an email to all engineering. 3/5 days we come out clean. 2/5 days we discover and fix a flaky test or broken build target. If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. Edit: looks like our numbers are quite close to Shopify’s. We don’t need anything close to hundreds of docker containers running in parallel. But we’re mostly a python shop, maybe they’re being a ruby shop slows things down here.

dudzik 6y ago
Ruby is not really slowing us down, the bigger issue is that we have a lot of integration tests and few isolated unit tests. So we are massively parallelizing our builds so we can get p50 build times of 23 minutes when running the whole test suite. The time mentioned in the article is a little bit closer to the p95 time. > If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. That is a good approach and we also used to disable tests that would fail on full master builds. When it comes to fixing intermittently failing tests we are now storing every test run so we can analyze our test suite. This way we are able to find the slowest and flakiest tests in the codebase, disable them, and notify the owners of the code to investigate.
- gen220 6y ago
  That’s interesting and definitely helps make the numbers make more sense! :) In your experience, do you find high value in having your tests be 80/20 (rough numbers) integration/unit vs the other way around? I guess it depends on the programming style, but here it’s the other way around. Some minority of engineers here clamor for end to end tests every once in a while, but we’ve rarely found the difficulty of establishing the test harness to be worth the marginal gains in reliability. We’re also wary of the temptation to write e2e tests to test unit-test-level correctness, because of the implications on test performance, like you describe. On our side, the closest thing we have to ubiquitous e2e testing is that we spin up and spin down one ephemeral database instance per test module (could be used by many test functions).

by gen220 6y ago

dudzik 6y ago

Ruby is not really slowing us down, the bigger issue is that we have a lot of integration tests and few isolated unit tests. So we are massively parallelizing our builds so we can get p50 build times of 23 minutes when running the whole test suite. The time mentioned in the article is a little bit closer to the p95 time. > If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet. That is a good approach and we also used to disable tests that would fail on full master builds. When it comes to fixing intermittently failing tests we are now storing every test run so we can analyze our test suite. This way we are able to find the slowest and flakiest tests in the codebase, disable them, and notify the owners of the code to investigate.

gen220 6y ago
That’s interesting and definitely helps make the numbers make more sense! :) In your experience, do you find high value in having your tests be 80/20 (rough numbers) integration/unit vs the other way around? I guess it depends on the programming style, but here it’s the other way around. Some minority of engineers here clamor for end to end tests every once in a while, but we’ve rarely found the difficulty of establishing the test harness to be worth the marginal gains in reliability. We’re also wary of the temptation to write e2e tests to test unit-test-level correctness, because of the implications on test performance, like you describe. On our side, the closest thing we have to ubiquitous e2e testing is that we spin up and spin down one ephemeral database instance per test module (could be used by many test functions).