4 ms·

GitHub Actions is a horrible CI/CD system. You cannot run steps in parallel on the same VM; container-based workloads are a second-class citizen. The first prob

GitHub Actions is a horrible CI/CD system. You cannot run steps in parallel on the same VM; container-based workloads are a second-class citizen. The first problem means that setting up local credentials and other environment dependencies cannot be parallelized (I'm looking at you, google-github-actions/setup-gcloud, with your 1m+ runtime... grrr), the second makes it quite difficult to put a Dockerfile in a repository to represent setup of the CI environment, and have both (a) the CI rebuild the container images when the image would change, pausing workflows depending on that image until the image is rebuilt, (b) not attempting to rebuild the image when its contents did not change, and immediately running workflows inside that image, including all dependencies already installed. No, in GitHub Actions, you will attempt to repopulate from cache on every single run. Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one. I deeply miss Concourse. Too bad Pivotal dismantled the team and it barely gets even maintenance work. Parallel tasks. Custom pipeline triggers. SSH into the CI environment to debug. Run one-off tasks in CI without needing to set up a pipeline. Transfer directory contents from one job to another without needing to create artifacts (which cost money to store if you're not careful about how long they should stick around for). GitHub Actions is a bastardized toy CI/CD system that only got popular because GitHub make it as simple as uploading a file to .github/workflows in any repository - no additional signup, no in-house ops required, everything you could want is just there. So let's be very clear about what GitHub Actions is good and what it's bad at - it's good at getting people to sign up, but it's not nearly powerful enough to be the "best" system once you start to outgrow the early features.

lawnchair 3y ago
solatic -- I have an existing solution that accounts for a lot of these issues you bring up. Would it be possible to pick your brain? Can you share your email or shoot me an email? lawnchair@lawnchair.net.
bryandollery 3y ago
Have you seen Tekton? (https://tekton.dev/ https://tekton.dev/)
couchand 3y ago
> Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one. Looks like I can move on that "build caching mysteriously broken" issue now. Thanks for the heads up!
txtsd 3y ago
Try the sourcehut build server
- solatic 3y ago
  It's a public alpha and clearly targeted for small hobbyists / FOSS work as a result. I'm looking for something where the creator has more confidence in its reliability...
speed_spread 3y ago
Concourse rocks. I didn't know the team had been dismantled, this sucks. Zito's communication style was the best.
- Huggernaut 3y ago
  Vito is at dagger.io now so hopefully we can expect some good stuff in the CI space there.
  - solatic 3y ago
    Sadly, Dagger doesn't get it either. It's so focused on portability between the underlying infrastructure providers, on not being the underlying infrastructure provider, and therefore it doesn't solve the real problem, which is the underlying infrastructure provider. (a) Consider Dagger's integration with GitHub Actions: https://docs.dagger.io/cookbook#github-actions https://docs.dagger.io/cookbook#github-actions where you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline. So Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point - it's unavoidable. Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it? (b) Consider a usecase where I want to parallelize the computation onto a number of machines dynamically chosen at run-time. Maybe I want to allow a test suite to run on an increasing number of machines without needing to periodically manually increase the number of machines in a configuration file, or maybe I'm using Terraform workspaces where I want to run terraform apply for each workspace on a different VM to let the number of workspaces scale horizontally. This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish. This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines, and running the underlying Concourse workers as an autoscaling Kubernetes statefulset/deployment, combined with other Kubernetes implements like cluster autoscaler.
    - shykes 3y ago
      Hi! Dagger co-founder here. I thought I’d share a few clarifying points - and acknowledge that we should explain certain aspects of Dagger’s design better, to avoid having to clarify in the first place! > you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline You do need to write CI configuration to run a dagger pipeline, but it’s a very small and standardized snippet (“install dagger, the run this command”) and it typically replaces a much larger custom mess of yaml and shell scripts. The main benefit though is that your pipeline logic is decoupled from CI altogether. You can run the same Dagger pipeline post-push (in CI) but also pre-push. Similar to running a makefile or shell script, except it’s real code. > * Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point* Dagger most definitely does do that :) We use Dagger and Github Actions ourselves, and have completely stopped using GHA’s caching system. Why bother, when Dagger caches everything automatically? > Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it? I think that’s your Stockholm syndrome talking. The terrible experience that is CI - the “push and pray” development loop; the drift between post-push yaml and pre-push scripts; the primitive caching system; the lack of good composition system; the total impossibility of testing your pipelines - that pain is avoidable, and you shouldn’t have to embrace it. You deserve better! > This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish. This is possible with Dagger. It certainly shouldn’t be a core feature of the engine, but the beauty of a programmable system is that you can build infinite capabilities on top of it. You do need a truly programmable system though, which Github Actions is not. > This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines Dagger pipelines can dynamically run new pipelines, at arbitrary depth. In other words nodes in the DAG can add more nodes at runtime. Give vito some credit, he is an incredibly talented engineer who built a product that you love. Maybe he saw in Dagger the potential to build something that you will love too :) He blogged about his thought process here: https://dev.to/vito/why-i-joined-dagger-43gb https://dev.to/vito/why-i-joined-dagger-43gb I will concede that Dagger’s clustering capabilities are not great yet. Which is why piggyback on CI infrastructure for that part… for now!
Huggernaut 3y ago
Less Pivotal and more VMWare post acquisition dismantling I'd say. There was a lot of love internally for Concourse (I left before the acquisition though). SSH debugging and one off tasks absolutely dreamy.
- toastal 3y ago
  Even SourceHut’s Spartan CI supports SSH debugging

lawnchair 3y ago

solatic -- I have an existing solution that accounts for a lot of these issues you bring up. Would it be possible to pick your brain? Can you share your email or shoot me an email? lawnchair@lawnchair.net.

bryandollery 3y ago

Have you seen Tekton? (https://tekton.dev/ https://tekton.dev/)

couchand 3y ago

> Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one. Looks like I can move on that "build caching mysteriously broken" issue now. Thanks for the heads up!

txtsd 3y ago

Try the sourcehut build server

solatic 3y ago
It's a public alpha and clearly targeted for small hobbyists / FOSS work as a result. I'm looking for something where the creator has more confidence in its reliability...

speed_spread 3y ago

Concourse rocks. I didn't know the team had been dismantled, this sucks. Zito's communication style was the best.

Huggernaut 3y ago
Vito is at dagger.io now so hopefully we can expect some good stuff in the CI space there.
- solatic 3y ago
  Sadly, Dagger doesn't get it either. It's so focused on portability between the underlying infrastructure providers, on not being the underlying infrastructure provider, and therefore it doesn't solve the real problem, which is the underlying infrastructure provider. (a) Consider Dagger's integration with GitHub Actions: https://docs.dagger.io/cookbook#github-actions https://docs.dagger.io/cookbook#github-actions where you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline. So Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point - it's unavoidable. Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it? (b) Consider a usecase where I want to parallelize the computation onto a number of machines dynamically chosen at run-time. Maybe I want to allow a test suite to run on an increasing number of machines without needing to periodically manually increase the number of machines in a configuration file, or maybe I'm using Terraform workspaces where I want to run terraform apply for each workspace on a different VM to let the number of workspaces scale horizontally. This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish. This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines, and running the underlying Concourse workers as an autoscaling Kubernetes statefulset/deployment, combined with other Kubernetes implements like cluster autoscaler.
  - shykes 3y ago
    Hi! Dagger co-founder here. I thought I’d share a few clarifying points - and acknowledge that we should explain certain aspects of Dagger’s design better, to avoid having to clarify in the first place! > you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline You do need to write CI configuration to run a dagger pipeline, but it’s a very small and standardized snippet (“install dagger, the run this command”) and it typically replaces a much larger custom mess of yaml and shell scripts. The main benefit though is that your pipeline logic is decoupled from CI altogether. You can run the same Dagger pipeline post-push (in CI) but also pre-push. Similar to running a makefile or shell script, except it’s real code. > * Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point* Dagger most definitely does do that :) We use Dagger and Github Actions ourselves, and have completely stopped using GHA’s caching system. Why bother, when Dagger caches everything automatically? > Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it? I think that’s your Stockholm syndrome talking. The terrible experience that is CI - the “push and pray” development loop; the drift between post-push yaml and pre-push scripts; the primitive caching system; the lack of good composition system; the total impossibility of testing your pipelines - that pain is avoidable, and you shouldn’t have to embrace it. You deserve better! > This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish. This is possible with Dagger. It certainly shouldn’t be a core feature of the engine, but the beauty of a programmable system is that you can build infinite capabilities on top of it. You do need a truly programmable system though, which Github Actions is not. > This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines Dagger pipelines can dynamically run new pipelines, at arbitrary depth. In other words nodes in the DAG can add more nodes at runtime. Give vito some credit, he is an incredibly talented engineer who built a product that you love. Maybe he saw in Dagger the potential to build something that you will love too :) He blogged about his thought process here: https://dev.to/vito/why-i-joined-dagger-43gb https://dev.to/vito/why-i-joined-dagger-43gb I will concede that Dagger’s clustering capabilities are not great yet. Which is why piggyback on CI infrastructure for that part… for now!

Huggernaut 3y ago

Less Pivotal and more VMWare post acquisition dismantling I'd say. There was a lot of love internally for Concourse (I left before the acquisition though). SSH debugging and one off tasks absolutely dreamy.

toastal 3y ago
Even SourceHut’s Spartan CI supports SSH debugging