4 ms·
I think you'd be surprised how long some tests queue waiting for a runner - but I do agree with you that it's not the longest delay running tests, and it is rel
by reillyse 4y ago
I think you'd be surprised how long some tests queue waiting for a runner - but I do agree with you that it's not the longest delay running tests, and it is relatively easy to fix with budget.
I would say our solution mostly breaks down to massive parallelization and in order to support this pre-built environments. The problem with massive parallelization with traditional hosted CI is that the first 5-6 minutes (maybe as many as 10 minutes) of your build is just creating your environment. So it is extremely inefficient to spend such a long time building for say 2 minutes of tests. Your total test time is 6 minutes setup + 2 minutes run time. Scaling that up by adding 100 machines is only going to bring the 2 minutes down, you are stuck with the 6 minutes build time.
By maintaining the environment between builds we can add more workers and get that test run time down to the 10 seconds range (with zero build time on most builds). Obviously we add complexity around knowing when to rebuild - but that is the price we pay for getting super fast tests.
- quesera 4y agoFWIW, my big Rails test suite spends about 2 minutes on setup, then about 40 minutes running with a single runner. It parallelizes well, but of course the 2 mins is the lower bound.
- reillyse 4y agoYea, that sounds somewhat normal. I see builds getting bigger when you add in a bunch of yarn packages, building assets and webpacker etc. As projects get larger these tend to bloat up. You might be interested in looking at this demo of a Rails Rspec/Capybara test suite (https://brisktest.com/demos#rails https://brisktest.com/demos#rails) it's over 60 minutes locally on my macbook, we get it down to 90 seconds on brisk. It's got a lot of Selenium Chrome tests which is why it's so long (90 seconds is long for us), and I think fixing their flakiness would shave a lot off those seconds (lots of rspec retry going on) but it's a good example of a real-world test suite. Often fixing flaky tests is not the priority and getting your tests run in 90 seconds is huge.
- Aeolun 4y ago> I think you'd be surprised how long some tests queue waiting for a runner Why would you ever queue? Our runners take about 40 seconds to come up from nothing. If 50 jobs come in we start 50 runners and then kill them again afterwards. It’s not like 50 x 1 is more expensive than 5 x 10 x 1.
- reillyse 4y agoListen I'm not defending queuing, I think that that is not the correct way to do things. But people definitely do. Current services queue.