4 ms·

One thing that I still don't understand about github actions is how much computing power gets wasted downloading and reinstalling the same dependencies over and

by ufo 3y ago

One thing that I still don't understand about github actions is how much computing power gets wasted downloading and reinstalling the same dependencies over and over. A build that takes a couple of seconds in my local machine can easily take several minutes in github actions.

0cf8612b2e1e 3y ago
GitHub charges per minute. Some incentive to keep actions on the most under specced machines available.
Raed667 3y ago
Do other CI services do it differently? I worked with Jenkins and Circleci and it was exactly the same. For github actions caching was fairly easy to add.
- ufo 3y ago
  Last time I tried to add caching to my actions, I spent several hours and still couldn't get it to work properly in the end. I really wonder how many computing hours have been wasted because the default and most likely to work is to not cache anything.
- aeyes 3y ago
  On CircleCI it's also easy to add using save_cache and restore_cache, the docs have examples for different popular frameworks. There are a couple of advanced features and I have been able to cache most build artifacts. They also cache the Docker images without you having to do anything but it really only works if you use their base images. With custom images the cache rate doesn't look great in my experience but I have no stats to back that up.
codetrotter 3y ago
Yeah. We have a similar thing going on with self-hosted GitLab at my work. (And there, there isn’t even any incentive why it should be slow.) When I make an edit and rebuild on my computer, it takes a few seconds to rebuild because most things are cached. In CI it takes 20 minutes on the MR branches of our repo. 30 minutes on master. Because a whole bunch of crap is downloaded from scratch and rebuilt from scratch. Our infrastructure guys did set up something that is able to cache stuff that some of the repos use. But with the limited access I have I have not been able to figure out how to make use of it for the repo I work on. And they don’t have time to look into it for me either. That’s my life lol
- mardifoufs 3y ago
  Yeah gitlab has pretty good caching support and it is basically a must if you do bigger jobs, especially on stuff like c++ apps. It is pretty brittle sometimes, and in my experience it won't be able to be as fast as actual local dev but it's still good. You can even make it use a remote cache IIRC.
- pjot 3y ago
  You can define a cache key in your .gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/index.html#cache https://docs.gitlab.com/ee/ci/yaml/index.html#cache
- koromak 3y ago
  On free runners I'd regularly hit the 1 hour limit. Took some time to set up self-hosted through EC2 and really optimize caching, its now down to 10-20 mins, depending on the repo. Most of that is because of the self-hosted runners, the free ones are painfully slow. Still, on my local machine these jobs would take 5 mins so its not perfect. And as the build gets more complicated and more stages are added, the problem compounds since the initial download is the slowest part.
- Already__Taken 3y ago
  Considering gitlab is a very k8s native app it's way of running a build and moving caches into it, vs moving the container over to where the cache's are is very annoying. I've had issues open for years to update the cache-key with a CSI mount and let k8s move the pods to where the mount is fast. As it is now it pulls everything from S3.
mardifoufs 3y ago
I prefer GA to basically every other build system especially because of the plugins but yeah, and I think caching can be turned on... but it has tons of inconsistencies related to packages and deps. This video really highlighted just how brittle the whole thing is even behind the scenes: https://youtu.be/9qljpi5jiMQ https://youtu.be/9qljpi5jiMQ (13:58 time stamp for the part that is especially relevant here)
- ufo 3y ago
  After that video, I think I finally get Github Actions.
HideousKojima 3y ago
In Azure DevOps pipelines there's a fun issue where if you're pulling a dependency from nuget.org it gets pulled very quickly, but pulling from your own artifact repositories hosted in the very same Azure DevOps instance take 40-80 seconds. I'm not the one paying the extra bills when my builds take 5 to 6 times longer than they should but it's still frustrating. And Microsoft has effectively zero incentive to fix this because it still technically works and they get to charge my employer for all that extra build agent time. Here's a few examples of people reporting similar issues and getting zero help: https://developercommunity.visualstudio.com/t/devops-build-pipeline-nuget-restore-very-slow-on-s/632309 https://developercommunity.visualstudio.com/t/devops-build-p... https://developercommunity.visualstudio.com/t/nuget-restore-takes-25-of-the-time-of-my-build/1027210 https://developercommunity.visualstudio.com/t/nuget-restore-... https://developercommunity.visualstudio.com/t/slow-performance-when-downloading-from-private-nug/476627 https://developercommunity.visualstudio.com/t/slow-performan...
- easton 3y ago
  Microsoft hosted agents aren’t charged by the minute on ADO, IIRC. You pay for a certain number of parallel agents per month. Unless you’re booting build servers on demand on your infra, you aren’t paying more for more minutes. Just waiting :( (Unlike GitHub, where they do charge per minute.)
- namtab00 3y ago
  do you know if ADO has any caching mechanisms for pipelines doing builds?
  - HideousKojima 3y ago
    The issue in this case isn't caching, even if I'm not actually pulling any packages from our private nuget repo simply having that repo listed in the project's nuget.config file is enough to cause the issue.
WorldMaker 3y ago
Given GitHub also owns npm these days, at least some of the constantly reinstalled dependencies are just getting copied from the equivalent of "next door".
- eastbound 3y ago
  And/or they can fix GitHub’s caching of NPM if it saves NPM money.
sagarm 3y ago
Use a persistent runner and you can take advantage of on disk caches...at the cost of repeatability you get from a clean image on each run.
- nsonha 3y ago
  repeatability must mean different things to us, a cache hit is repeatability to me, a network call can easily fail for so many reasons
  - paranoidrobot 3y ago
    It sure can fail but running on a clean agent each time reduces the "Well, it works on my machine" scenarios you can run into. For example: The last run didn't clean up, or changed something global. New runs don't work because some expected state is missing that had been set up on a previous run. Blowing the agent away each time reduces the chance that something previously run can impact your current run. This is at the cost of pulling from networks each time. You can reduce those issues by adding network caches for modules/dependencies.
  - sagarm 3y ago
    I use persistent runners, but we had to add additional steps (docker prune, mvn -U, docker image ls, etc) to keep the runner healthy and be better able to debug issues.

by ufo 3y ago

0cf8612b2e1e 3y ago

GitHub charges per minute. Some incentive to keep actions on the most under specced machines available.

Raed667 3y ago

Do other CI services do it differently? I worked with Jenkins and Circleci and it was exactly the same. For github actions caching was fairly easy to add.

ufo 3y ago
Last time I tried to add caching to my actions, I spent several hours and still couldn't get it to work properly in the end. I really wonder how many computing hours have been wasted because the default and most likely to work is to not cache anything.
aeyes 3y ago
On CircleCI it's also easy to add using save_cache and restore_cache, the docs have examples for different popular frameworks. There are a couple of advanced features and I have been able to cache most build artifacts. They also cache the Docker images without you having to do anything but it really only works if you use their base images. With custom images the cache rate doesn't look great in my experience but I have no stats to back that up.

codetrotter 3y ago

Yeah. We have a similar thing going on with self-hosted GitLab at my work. (And there, there isn’t even any incentive why it should be slow.) When I make an edit and rebuild on my computer, it takes a few seconds to rebuild because most things are cached. In CI it takes 20 minutes on the MR branches of our repo. 30 minutes on master. Because a whole bunch of crap is downloaded from scratch and rebuilt from scratch. Our infrastructure guys did set up something that is able to cache stuff that some of the repos use. But with the limited access I have I have not been able to figure out how to make use of it for the repo I work on. And they don’t have time to look into it for me either. That’s my life lol

mardifoufs 3y ago
Yeah gitlab has pretty good caching support and it is basically a must if you do bigger jobs, especially on stuff like c++ apps. It is pretty brittle sometimes, and in my experience it won't be able to be as fast as actual local dev but it's still good. You can even make it use a remote cache IIRC.
pjot 3y ago
You can define a cache key in your .gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/index.html#cache https://docs.gitlab.com/ee/ci/yaml/index.html#cache
koromak 3y ago
On free runners I'd regularly hit the 1 hour limit. Took some time to set up self-hosted through EC2 and really optimize caching, its now down to 10-20 mins, depending on the repo. Most of that is because of the self-hosted runners, the free ones are painfully slow. Still, on my local machine these jobs would take 5 mins so its not perfect. And as the build gets more complicated and more stages are added, the problem compounds since the initial download is the slowest part.
Already__Taken 3y ago
Considering gitlab is a very k8s native app it's way of running a build and moving caches into it, vs moving the container over to where the cache's are is very annoying. I've had issues open for years to update the cache-key with a CSI mount and let k8s move the pods to where the mount is fast. As it is now it pulls everything from S3.

mardifoufs 3y ago

I prefer GA to basically every other build system especially because of the plugins but yeah, and I think caching can be turned on... but it has tons of inconsistencies related to packages and deps. This video really highlighted just how brittle the whole thing is even behind the scenes: https://youtu.be/9qljpi5jiMQ https://youtu.be/9qljpi5jiMQ (13:58 time stamp for the part that is especially relevant here)

ufo 3y ago
After that video, I think I finally get Github Actions.

HideousKojima 3y ago

In Azure DevOps pipelines there's a fun issue where if you're pulling a dependency from nuget.org it gets pulled very quickly, but pulling from your own artifact repositories hosted in the very same Azure DevOps instance take 40-80 seconds. I'm not the one paying the extra bills when my builds take 5 to 6 times longer than they should but it's still frustrating. And Microsoft has effectively zero incentive to fix this because it still technically works and they get to charge my employer for all that extra build agent time. Here's a few examples of people reporting similar issues and getting zero help: https://developercommunity.visualstudio.com/t/devops-build-pipeline-nuget-restore-very-slow-on-s/632309 https://developercommunity.visualstudio.com/t/devops-build-p... https://developercommunity.visualstudio.com/t/nuget-restore-takes-25-of-the-time-of-my-build/1027210 https://developercommunity.visualstudio.com/t/nuget-restore-... https://developercommunity.visualstudio.com/t/slow-performance-when-downloading-from-private-nug/476627 https://developercommunity.visualstudio.com/t/slow-performan...

easton 3y ago
Microsoft hosted agents aren’t charged by the minute on ADO, IIRC. You pay for a certain number of parallel agents per month. Unless you’re booting build servers on demand on your infra, you aren’t paying more for more minutes. Just waiting :( (Unlike GitHub, where they do charge per minute.)
namtab00 3y ago
do you know if ADO has any caching mechanisms for pipelines doing builds?
- HideousKojima 3y ago
  The issue in this case isn't caching, even if I'm not actually pulling any packages from our private nuget repo simply having that repo listed in the project's nuget.config file is enough to cause the issue.

WorldMaker 3y ago

Given GitHub also owns npm these days, at least some of the constantly reinstalled dependencies are just getting copied from the equivalent of "next door".

eastbound 3y ago
And/or they can fix GitHub’s caching of NPM if it saves NPM money.

sagarm 3y ago

Use a persistent runner and you can take advantage of on disk caches...at the cost of repeatability you get from a clean image on each run.

nsonha 3y ago
repeatability must mean different things to us, a cache hit is repeatability to me, a network call can easily fail for so many reasons
- paranoidrobot 3y ago
  It sure can fail but running on a clean agent each time reduces the "Well, it works on my machine" scenarios you can run into. For example: The last run didn't clean up, or changed something global. New runs don't work because some expected state is missing that had been set up on a previous run. Blowing the agent away each time reduces the chance that something previously run can impact your current run. This is at the cost of pulling from networks each time. You can reduce those issues by adding network caches for modules/dependencies.
- sagarm 3y ago
  I use persistent runners, but we had to add additional steps (docker prune, mvn -U, docker image ls, etc) to keep the runner healthy and be better able to debug issues.