4 ms·

Why does it happen? My CI launches separate kubernetes pod for every incoming task. If there are not enough servers, autoscaler will spin up new and remove it a

Why does it happen? My CI launches separate kubernetes pod for every incoming task. If there are not enough servers, autoscaler will spin up new and remove it after some time. So there are not much expenses, we're paying for what we using. I feel that fast CI iterations are very important for developers.

XorNot 2y ago
This has been the bane of my entire career. I have never, not once, encountered a CI system that could run the test suite faster then my development machine. Which is absurd. Cloud resources on servers with more CPUs then I have, and yet the difference in timing is measured in tens of minutes.
- dilyevsky 2y ago
  In most cases i have experienced this phenomenon is caused by the fact that cloud block devices are garbage compared to your machine
  - tonyarkles 2y ago
    And often CPU as well if your machine has a reasonable number of cores.
  - kevin_nisbet 2y ago
    That combined with the scheduling layers... on the CI service the company I'm working for uses it takes like 2 minutes to just start the job.
    - Aeolun 2y ago
      Even creating whole new AWS instances each time it shouldn’t take more than 1m. You can apparently bring it down to 5s by suspending instead of creating instances.
      - tomohawk 2y ago
        It depends on how much policy your organization has. If your organization requires AV scanners and other security crapware to be on all instances, how long does that take to install? And perhaps they require that you only use their AMIs, which then must be loaded up with extra software to be usable. And perhaps they require all auth use their special auth service, which takes 5m to sync with. And on and on. I've worked in places where it takes 20m to spin up a usable instance, so we ended up just keeping them on most of the time. In an ideal world, the policy groups imposing these inefficiencies would get billed for the cost of their policies, but I've only ever seen situations where everyone else pays the cost of their incompetence.
        Aeolun 2y ago
        Ah, yeah, we kinda have to work around the fact that the antivirus and scanner stuff keeps installing far after the time the instance is done processing the job. On the plus side, I actually get actionable notifications from the stuff, so it’s not purely bad for me.
  - jomohke 2y ago
    And yet the data is ephemeral, so in many cases could be done faster without the "real" block storage guarantees. Github Actions is pay-per-minute-used, unfortunately, so they may have a negative incentive to speed things up. Unless people become frustrated enough to switch to a non-bundled provider.
    - dilyevsky 2y ago
      Not all the data is ephemeral - stuff like node_modules needs to he cached and if you’re suggesting to use tmpfs then it’s like 50x more cost than fully tricked out cloud ssd which is already ridiculously expensive
- adolph 2y ago
  You should just add an admission webhook that kubeadm’s your dev box onto the CI cluster with a taint that only your pods tolerate whenever you need to run the test suite. Call it edge cloud and people will love it.
- dboreham 2y ago
  This is just a symptom of bad management.
- dragonwriter 2y ago
  > Cloud resources on servers with more CPUs then I have The server the resources run on may have more CPUs than your dev machine, but are you allocating more CPU to the specific test job than you desktop has?
- rr808 2y ago
  DHH has talked about this the last few months. He now recommends running locally on your own computer instead of server. CPUs are so fast now. https://world.hey.com/dhh/we-re-moving-continuous-integration-back-to-developer-machines-3ac6c611 https://world.hey.com/dhh/we-re-moving-continuous-integratio...
  - samus 2y ago
    Unless internal IT and security teams force running security spyware on personal devices.
- Too 2y ago
  From what I've seen this is usually caused by CI pipelines being overly cautious, starting from a pristine state and trying to cover every possible edge case. Spin up new node, congrats, now you have to clone the whole repo from scratch. Oh, "npm install" was not cached, let's download half of the internet again, pull ":latest" of some multi-GB docker images while you are at it. For good measure, "make clean", because...you never know and nobody bothered to architect our build system to use a more sane tool than Makefiles, btw remember to only use make -j3, because rumor has it that the build fails intermittently when using more cores! Finally, run every test that couldn't possibly be affected by the commit in question, preferably the big e2e-suite. As someone else mentioned above, all of this running on some dog-slow cloud network attached disk. There you have it. Spinning up more compute is not the solution, it's the cause. Keep one machine on and improve on your build tooling. I guarantee it will be magnitudes faster than any ephemeral pipeline.
  - MathMonkeyMan 2y ago
    But then what will you contribute to the hour long weekly meeting where 13 managers present the status of their poorly defined CI metrics and other "action items"?
  - TeMPOraL 2y ago
    Not my experience. The CI systems I dealt with most recently, for example, were very much compute-bound. Way too weak VMs for the runnres, and way too few of them. With CI builds taking each of upward 2h, and capacity to run maybe 10 of them in parallel, this pretty much destroys any ability to work with small commits - you have to squash them before submitting for review anyway, otherwise you destroy anyone else's ability to get their changeset through CI for the rest of the day. This is entirely solvable by getting more resources. At least 2x more runners, each at least 2x as beefy. You could go 10x and I doubt it would be anywhere as expensive as the amount of money company wastes on devs waiting for CI. Alas, good luck convincing people managing the infra of this.
    - Aeolun 2y ago
      Our runners autoscale, and you can choose whatever instance type you want. It doesn’t seem like a very hard problem to solve.
- mvc 2y ago
  There was a tech talk many years ago when one of the github engineers talked about how fast the github CI tests were (this was way before the microsoft aquisition). Literally seconds. I wonder if they're still that fast.
  - Bognar 2y ago
    They absolutely are not. Hundreds of thousands of Ruby tests end up being pretty slow to execute. The CI machines have all dependencies pre-cached and different tests are parallelized across multiple VMs, yet still a full suite takes ~12 minutes.
- samus 2y ago
  My experience is the opposite due to the insane overhead of the spyware^H^H^H antivirus and security services running on our personal devices. Some colleagues successfully applied for MacBooks since these are reasonably fast even while running the security services.

XorNot 2y ago

This has been the bane of my entire career. I have never, not once, encountered a CI system that could run the test suite faster then my development machine. Which is absurd. Cloud resources on servers with more CPUs then I have, and yet the difference in timing is measured in tens of minutes.

dilyevsky 2y ago
In most cases i have experienced this phenomenon is caused by the fact that cloud block devices are garbage compared to your machine
- tonyarkles 2y ago
  And often CPU as well if your machine has a reasonable number of cores.
- kevin_nisbet 2y ago
  That combined with the scheduling layers... on the CI service the company I'm working for uses it takes like 2 minutes to just start the job.
  - Aeolun 2y ago
    Even creating whole new AWS instances each time it shouldn’t take more than 1m. You can apparently bring it down to 5s by suspending instead of creating instances.
    - tomohawk 2y ago
      It depends on how much policy your organization has. If your organization requires AV scanners and other security crapware to be on all instances, how long does that take to install? And perhaps they require that you only use their AMIs, which then must be loaded up with extra software to be usable. And perhaps they require all auth use their special auth service, which takes 5m to sync with. And on and on. I've worked in places where it takes 20m to spin up a usable instance, so we ended up just keeping them on most of the time. In an ideal world, the policy groups imposing these inefficiencies would get billed for the cost of their policies, but I've only ever seen situations where everyone else pays the cost of their incompetence.
      - Aeolun 2y ago
        Ah, yeah, we kinda have to work around the fact that the antivirus and scanner stuff keeps installing far after the time the instance is done processing the job. On the plus side, I actually get actionable notifications from the stuff, so it’s not purely bad for me.
- jomohke 2y ago
  And yet the data is ephemeral, so in many cases could be done faster without the "real" block storage guarantees. Github Actions is pay-per-minute-used, unfortunately, so they may have a negative incentive to speed things up. Unless people become frustrated enough to switch to a non-bundled provider.
  - dilyevsky 2y ago
    Not all the data is ephemeral - stuff like node_modules needs to he cached and if you’re suggesting to use tmpfs then it’s like 50x more cost than fully tricked out cloud ssd which is already ridiculously expensive
adolph 2y ago
You should just add an admission webhook that kubeadm’s your dev box onto the CI cluster with a taint that only your pods tolerate whenever you need to run the test suite. Call it edge cloud and people will love it.
dboreham 2y ago
This is just a symptom of bad management.
dragonwriter 2y ago
> Cloud resources on servers with more CPUs then I have The server the resources run on may have more CPUs than your dev machine, but are you allocating more CPU to the specific test job than you desktop has?
rr808 2y ago
DHH has talked about this the last few months. He now recommends running locally on your own computer instead of server. CPUs are so fast now. https://world.hey.com/dhh/we-re-moving-continuous-integration-back-to-developer-machines-3ac6c611 https://world.hey.com/dhh/we-re-moving-continuous-integratio...
- samus 2y ago
  Unless internal IT and security teams force running security spyware on personal devices.
Too 2y ago
From what I've seen this is usually caused by CI pipelines being overly cautious, starting from a pristine state and trying to cover every possible edge case. Spin up new node, congrats, now you have to clone the whole repo from scratch. Oh, "npm install" was not cached, let's download half of the internet again, pull ":latest" of some multi-GB docker images while you are at it. For good measure, "make clean", because...you never know and nobody bothered to architect our build system to use a more sane tool than Makefiles, btw remember to only use make -j3, because rumor has it that the build fails intermittently when using more cores! Finally, run every test that couldn't possibly be affected by the commit in question, preferably the big e2e-suite. As someone else mentioned above, all of this running on some dog-slow cloud network attached disk. There you have it. Spinning up more compute is not the solution, it's the cause. Keep one machine on and improve on your build tooling. I guarantee it will be magnitudes faster than any ephemeral pipeline.
- MathMonkeyMan 2y ago
  But then what will you contribute to the hour long weekly meeting where 13 managers present the status of their poorly defined CI metrics and other "action items"?
- TeMPOraL 2y ago
  Not my experience. The CI systems I dealt with most recently, for example, were very much compute-bound. Way too weak VMs for the runnres, and way too few of them. With CI builds taking each of upward 2h, and capacity to run maybe 10 of them in parallel, this pretty much destroys any ability to work with small commits - you have to squash them before submitting for review anyway, otherwise you destroy anyone else's ability to get their changeset through CI for the rest of the day. This is entirely solvable by getting more resources. At least 2x more runners, each at least 2x as beefy. You could go 10x and I doubt it would be anywhere as expensive as the amount of money company wastes on devs waiting for CI. Alas, good luck convincing people managing the infra of this.
  - Aeolun 2y ago
    Our runners autoscale, and you can choose whatever instance type you want. It doesn’t seem like a very hard problem to solve.
mvc 2y ago
There was a tech talk many years ago when one of the github engineers talked about how fast the github CI tests were (this was way before the microsoft aquisition). Literally seconds. I wonder if they're still that fast.
- Bognar 2y ago
  They absolutely are not. Hundreds of thousands of Ruby tests end up being pretty slow to execute. The CI machines have all dependencies pre-cached and different tests are parallelized across multiple VMs, yet still a full suite takes ~12 minutes.
samus 2y ago
My experience is the opposite due to the insane overhead of the spyware^H^H^H antivirus and security services running on our personal devices. Some colleagues successfully applied for MacBooks since these are reasonably fast even while running the security services.