3 ms·

The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 t

The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan... Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-test-deploy cycle at least once a day, hence the importance placed on rock-solid testing systems that should only ever fail when the tested code itself fails.

TimothyFitz 18y ago
We empirically have on average 70 builds a day. The number is higher than your calculation because we don't all work 9-5, we're commiting frequently from around 8am to 9pm. We also run builds repeatedly overnight to flush out any intermittently failing tests we may have recently introduced. We'll run the builds as fast as they can go from 2am-4am.
- sbraford 18y ago
  So how often does a commit get checked in that causes a test (or tests) to fail? It just seemed to me like you were bragging that tests get run over and over again. They only need to get run if any new code is committed, of course. And what kind of commit is being checked in every 9 minutes? How big is the dev team? Seems like an awful lot of commits. Is each one a full-fledged feature / bug fix for the site, or are many 1-line changes to the code?