4 ms·
We managed to speed up our CI to save 168 days of execution time per month
- odie5533 2y agoThey call pytest --collect-only and parse the output before distributing it to the Github Action python matrix. I'm a little surprised pytest doesn't offer the ability to cache collection. Though a slow collection time may be an issue that should be addressed since developers must suffer it locally!
- crohr 2y agoAny time I see engineering time spent on splitting/sharding test suites, I can't help but wonder if access to a single beefier runner (e.g. 64+ cpus) would have alleviated all that work. Also always find the duplicated setup time a bit wasteful on resources.
- edude03 2y agoBasically yeah. At my last company we switch from n GitHub runners to 2 128 core epyc boxes and massively (20x?) decreased our build times. A surprising thing is how much time is spent on IO to upload and restore the cache between steps that’s saved if the entire job runs on a single machine
- datadrivenangel 2y agoLarge box in the closet has always been a competitive strategy. You have to keep your closet clean and organized or else everything is bad, but if you can, it works well.
- stuaxo 2y agoCould do with more detail, is this using pytest xdist ?
- drunkenfly 2y agoThey mention their Github pull request with all changes.
- deleted 2y ago[deleted]
- xnx 2y agoEvery time I see this type of story I can only think how people get rewarded for heroically/dramatically rescuing bad systems. While those who choose boring/proven technology at an earlier point, don't get recognition (or even hired in many cases).
- Arcuru 2y agoI must be misunderstanding. Every test run was spending 3 _hours_ just discovering which tests to run 9 times?