3 ms·

What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can

What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results. While I'm dreaming about my perfect CI, I'd also like the ability to download benchmark results from previous commits so that I can generate trend graphs and publish them in the build results. To do this right the CI system would have to have an API using REST, GraphQL, gRPC or some such API format that generates clients in many languages. That way they don't have to maintain bindings in every language. [1] https://nuke.build/ https://nuke.build/

gravypod 5y ago
I think the main issue is that most build systems are not expressive enough to do complex packaging, code gen, etc that most projects need. I've been using Bazel for a few projects/teams and it's worked really well. Once your build system is as expressive as you need you get a lot of freedom. Bazel also let's you execute completely local builds and remote builds triggered locally. If you set this up debugging CI issues is amazingly simple.
- octopoc 5y ago
  Totally agree. I've never used Bazel; it sounds a lot like Nuke. I had a coworker whose job was to maintain the CI. His commit messages would look like this sometimes: > Fix CI issue with blah blah blah > Hmm that didn't work lets try something from stackoverflow > Build fix > Build fix please work > Please > I hate my life > I am tired and hungry, I want to go home > Stupid yaml The CTO would call him like a week later to make sure he was okay. This happened often before we switched to Nuke. It rarely happened afterwards. It's awesome being able to debug your CI locally.
  - gravypod 5y ago
    Yep, and if you get everything working smoothly enough developers can use the same very polished tooling for their own workflow.
  - nly 5y ago
    The problem in C++ land (my land) is all your dependencies are probably using different build systems. Do you wan to migrate all your third party libs to Bazel, or just hack up a few lines of CI yaml to call the authors build system? I lean on gitlab yaml quite heavily and it feels quite effortless
    - vsgzusnex 5y ago
      We vendored all of our third party dependencies into our Bazel build. There were a lot and it was a giant PITA but over a multi-year horizon it was worth it. That doesn't mean it's the right choice for every business but it paid off for us.
      - nly 5y ago
        Vendoring all your third party dependencies is fine, but you still have to build them. Personally I use the native build system that upstream uses for most of them, and just hack in my compiler flags/options where necessary. Then I use CI triggers to rebuild my projects when the dependencies are rebuilt.
initplus 5y ago
GCP takes a really flexible approach with their cloudbuild CICD offering. Each build step is just a docker image URL, entrypoint, and args. Want to do something fancy and complex? Just pull in a python image and call your python script. But you can still implement simple build steps directly in the build file. No need for a multitude of language bindings, everything you want to do is probably implemented with simple shell commands.
AlphaSite 5y ago
I think this is more of an issue with people misusing CI systems, your gitlab-CI/Jenkins/concourse/etc shouldn’t have any real logic in it, it’s just glue code. Bazel/Gradle/Maven/etc tend push you in the right direction, bash, etc dont. Or to put it succinctly, you need both a build system and a CI system, they are not the same thing.
qbasic_forever 5y ago
Check out tekton CI (https://tekton.dev/ https://tekton.dev/), it's a Kubernetes operator to run a CI pipeline that's defined as commands running inside any container. Use any language, any commands, etc--as long as you can get a container image, you're good to go. There's a growing set of community created and curated actions to do common things too: https://github.com/tektoncd/catalog https://github.com/tektoncd/catalog Yeah you need a k8s cluster, but even a simple kind dev cluster that you spin up in 30 seconds with one command on your laptop will work. I like it a lot because it enforces very little structure on you and doesn't reinvent everything. Stuff like storage (either ephermeral or existing volumes), secrets, configuration, etc. are already modeled and supported by Kubernetes and tekton can use all of that natively. And since it's all k8s native stuff you have all of the power of k8s, like its entire API for manipulating and managing execution, exposing services, etc. There is very little cognitive overhead or new things to learn once you know k8s. If you're really averse to k8s though, check out drone. It has a local execution mode that is similar and just runs whatever pipeline commands you want in docker containers. https://github.com/drone/drone https://github.com/drone/drone Batect is another even more minimal tool that's effectively just a docker-based workflow system: https://github.com/batect/batect https://github.com/batect/batect
- theptip 5y ago
  I’ve been meaning to give Tekton a try, but how is the visualization (pipeline graphs ets) and Github/Gitlab/CI integration for returning results and viewing build output?
  - qbasic_forever 5y ago
    There's a super basic dashboard: https://github.com/tektoncd/dashboard https://github.com/tektoncd/dashboard Tekton is really more of a lower level engine for CI though and doesn't focus on building UI, etc. You might check out jenkinsx which uses tekton but builds a whole new jenkins experience on it: https://jenkins-x.io/ https://jenkins-x.io/
- dustym 5y ago
  We use Tekton to manage our CI pipeline and I agree that the way it enforces very little structure is a strength. On the other hand it's new enough that if you need to stretch its capabilities you are going to have to get creative. The primitives it has are nice, but they have their limits. For instance, if running a bunch of parallel tasks, collating results on a PV is out the window unless your cluster supports multiple writer volume types, which GKE does not. You have to bring in NFS volume types or something like that for it. In the early days of tekton they had a results primitive which synchronized an output dir to GCS, but they decommissioned that. So you are left pushing that logic into your task command. Running gsutil is easy enough, but it means you are pushing logic into your scripts and not declaring steps in the pipeline definition. You could make that command a step but I see little benefit in that. Additionally there is no way to loop in the configuration to generate tasks, much less loop with an ordinal value. We end up just programmatically generating the resource definitions with ruby erb templates. All of our pipeline specs (including task runs, etc) creates a 2MB yaml file. We push dozens and dozens of these into k8s daily. It works but at the same time our usage of Tekton is more or less as a glorified alternative to batch jobs which works because batch jobs _still_ don't have a proper sidecar capability and also because we rely on the DAG to order dependent taskruns. If your pipeline is simple, look at Tekton. But if your pipeline is complex... still look at Tekton but expect to do some work. Once you get a good workflow though you can you can scale your pipelines as easily as you can a deployment in k8s. We use node autoscaling and preemptibles (Tekton can retry if a task disappears due to node reclamation) to manage our CI costs quite effectively.
rosineygp 5y ago
My main problem is running the job/step locally and not wait to CI steps to validate some change. My solution is: https://github.com/rosineygp/mkdkr https://github.com/rosineygp/mkdkr I can write somethings like this: py: @$(dkr) instance: python:3.8 run: pip install requests export MKDKR_SHELL=python run: 'import requests r = requests.get(f"https://api.isevenapi.xyz/api/iseven/2/") print(r.json())'
globular-toast 5y ago
> What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Every tool I've used allows you to do this. Just write the script in the language of your choice, then run it in the pipeline. A pipeline should be considered like "markup" around your scripts. Everything should work independently and on any system. The pipeline config just tells the scheduler which order to run it in, which bits can run in parallel, what artifacts to keep etc.
- pydry 5y ago
  In practice the YAML seems to be used to hook things up - uploading assets, sending messages on slack, etc. If gitlab/GitHub could just provide hooks to do this stuff for a range of scripting languages that could be easily dry run this would solve the problem.

gravypod 5y ago

I think the main issue is that most build systems are not expressive enough to do complex packaging, code gen, etc that most projects need. I've been using Bazel for a few projects/teams and it's worked really well. Once your build system is as expressive as you need you get a lot of freedom. Bazel also let's you execute completely local builds and remote builds triggered locally. If you set this up debugging CI issues is amazingly simple.

octopoc 5y ago
Totally agree. I've never used Bazel; it sounds a lot like Nuke. I had a coworker whose job was to maintain the CI. His commit messages would look like this sometimes: > Fix CI issue with blah blah blah > Hmm that didn't work lets try something from stackoverflow > Build fix > Build fix please work > Please > I hate my life > I am tired and hungry, I want to go home > Stupid yaml The CTO would call him like a week later to make sure he was okay. This happened often before we switched to Nuke. It rarely happened afterwards. It's awesome being able to debug your CI locally.
- gravypod 5y ago
  Yep, and if you get everything working smoothly enough developers can use the same very polished tooling for their own workflow.
- nly 5y ago
  The problem in C++ land (my land) is all your dependencies are probably using different build systems. Do you wan to migrate all your third party libs to Bazel, or just hack up a few lines of CI yaml to call the authors build system? I lean on gitlab yaml quite heavily and it feels quite effortless
  - vsgzusnex 5y ago
    We vendored all of our third party dependencies into our Bazel build. There were a lot and it was a giant PITA but over a multi-year horizon it was worth it. That doesn't mean it's the right choice for every business but it paid off for us.
    - nly 5y ago
      Vendoring all your third party dependencies is fine, but you still have to build them. Personally I use the native build system that upstream uses for most of them, and just hack in my compiler flags/options where necessary. Then I use CI triggers to rebuild my projects when the dependencies are rebuilt.

initplus 5y ago

GCP takes a really flexible approach with their cloudbuild CICD offering. Each build step is just a docker image URL, entrypoint, and args. Want to do something fancy and complex? Just pull in a python image and call your python script. But you can still implement simple build steps directly in the build file. No need for a multitude of language bindings, everything you want to do is probably implemented with simple shell commands.

AlphaSite 5y ago

I think this is more of an issue with people misusing CI systems, your gitlab-CI/Jenkins/concourse/etc shouldn’t have any real logic in it, it’s just glue code. Bazel/Gradle/Maven/etc tend push you in the right direction, bash, etc dont. Or to put it succinctly, you need both a build system and a CI system, they are not the same thing.

qbasic_forever 5y ago

Check out tekton CI (https://tekton.dev/ https://tekton.dev/), it's a Kubernetes operator to run a CI pipeline that's defined as commands running inside any container. Use any language, any commands, etc--as long as you can get a container image, you're good to go. There's a growing set of community created and curated actions to do common things too: https://github.com/tektoncd/catalog https://github.com/tektoncd/catalog Yeah you need a k8s cluster, but even a simple kind dev cluster that you spin up in 30 seconds with one command on your laptop will work. I like it a lot because it enforces very little structure on you and doesn't reinvent everything. Stuff like storage (either ephermeral or existing volumes), secrets, configuration, etc. are already modeled and supported by Kubernetes and tekton can use all of that natively. And since it's all k8s native stuff you have all of the power of k8s, like its entire API for manipulating and managing execution, exposing services, etc. There is very little cognitive overhead or new things to learn once you know k8s. If you're really averse to k8s though, check out drone. It has a local execution mode that is similar and just runs whatever pipeline commands you want in docker containers. https://github.com/drone/drone https://github.com/drone/drone Batect is another even more minimal tool that's effectively just a docker-based workflow system: https://github.com/batect/batect https://github.com/batect/batect

theptip 5y ago
I’ve been meaning to give Tekton a try, but how is the visualization (pipeline graphs ets) and Github/Gitlab/CI integration for returning results and viewing build output?
- qbasic_forever 5y ago
  There's a super basic dashboard: https://github.com/tektoncd/dashboard https://github.com/tektoncd/dashboard Tekton is really more of a lower level engine for CI though and doesn't focus on building UI, etc. You might check out jenkinsx which uses tekton but builds a whole new jenkins experience on it: https://jenkins-x.io/ https://jenkins-x.io/
dustym 5y ago
We use Tekton to manage our CI pipeline and I agree that the way it enforces very little structure is a strength. On the other hand it's new enough that if you need to stretch its capabilities you are going to have to get creative. The primitives it has are nice, but they have their limits. For instance, if running a bunch of parallel tasks, collating results on a PV is out the window unless your cluster supports multiple writer volume types, which GKE does not. You have to bring in NFS volume types or something like that for it. In the early days of tekton they had a results primitive which synchronized an output dir to GCS, but they decommissioned that. So you are left pushing that logic into your task command. Running gsutil is easy enough, but it means you are pushing logic into your scripts and not declaring steps in the pipeline definition. You could make that command a step but I see little benefit in that. Additionally there is no way to loop in the configuration to generate tasks, much less loop with an ordinal value. We end up just programmatically generating the resource definitions with ruby erb templates. All of our pipeline specs (including task runs, etc) creates a 2MB yaml file. We push dozens and dozens of these into k8s daily. It works but at the same time our usage of Tekton is more or less as a glorified alternative to batch jobs which works because batch jobs _still_ don't have a proper sidecar capability and also because we rely on the DAG to order dependent taskruns. If your pipeline is simple, look at Tekton. But if your pipeline is complex... still look at Tekton but expect to do some work. Once you get a good workflow though you can you can scale your pipelines as easily as you can a deployment in k8s. We use node autoscaling and preemptibles (Tekton can retry if a task disappears due to node reclamation) to manage our CI costs quite effectively.

rosineygp 5y ago

My main problem is running the job/step locally and not wait to CI steps to validate some change. My solution is: https://github.com/rosineygp/mkdkr https://github.com/rosineygp/mkdkr I can write somethings like this: py: @$(dkr) instance: python:3.8 run: pip install requests export MKDKR_SHELL=python run: 'import requests r = requests.get(f"https://api.isevenapi.xyz/api/iseven/2/") print(r.json())'

globular-toast 5y ago

> What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc. Every tool I've used allows you to do this. Just write the script in the language of your choice, then run it in the pipeline. A pipeline should be considered like "markup" around your scripts. Everything should work independently and on any system. The pipeline config just tells the scheduler which order to run it in, which bits can run in parallel, what artifacts to keep etc.

pydry 5y ago
In practice the YAML seems to be used to hook things up - uploading assets, sending messages on slack, etc. If gitlab/GitHub could just provide hooks to do this stuff for a range of scripting languages that could be easily dry run this would solve the problem.