3 ms·

Building a CI/CD Pipeline Runner from Scratch in Python

68 points by mr_o47 11mo ago

systemerror 11mo ago
Why does air-gapped environment require rolling your own CI/CD solution? Plenty of examples of air-gapped Jenkins and/or Argo Workflows. Was this just an educational exercise?
- mr_o47 11mo ago
  This was more like an educational exercise
  - esafak 11mo ago
    Since you're exercising, you can take it to the next level where you don't specify the next step but the inputs to each task, allowing you to infer the DAG and implement caching...
    - verdverm 11mo ago
      You can do this with cue/flow, but have not turned it into a full CI system. The building blocks are there
      - mr_o47 11mo ago
        Never heard of cue/flow will definitely check it out
        verdverm 11mo ago
        cue/flow is a Go package and what powers the scripting / custom tools part of CUE. It's built on top of the language and a graph resolver. I have another custom flow implementation that I find more ergonomic: https://hofstadter.io/getting-started/task-engine/ https://hofstadter.io/getting-started/task-engine/
- verdverm 11mo ago
  Jenkins sucks but is insanely reliable Argo Workflows does not live up to what they advertise, it is so much more complex to setup and then build workflows for. Helm + Argo is pain (both use the same template delimiters...)
  - bigstrat2003 11mo ago
    Jenkins, like many tools with extreme flexibility, sucks as much as you make it suck. You can pretty easily turn Jenkins into a monstrosity that makes everyone afraid to ever try to update it. On the other hand, you can also have a pretty solid setup that is easy to work on. The trouble is that the tool itself doesn't guide you much to the good path, so unless you've seen a pleasant Jenkins instance before you're likely to have a worse time than necessary.
    - IshKebab 11mo ago
      Are you sure, because last time I used Jenkins it actively sucked. The interface was a total mess and it doesn't surface results in any useful way.
      - fowlie 11mo ago
        When was the last time you used Jenkins? I don't get the hate. Not only from you, but lots of people on the internet. What makes Jenkins stand out IMO is the community and the core maintainers, they are perhaps moving slow, but they are moving in the right directions. The interface looks really nice now, they've done a lot of ux improvements lately.
        verdverm 11mo ago
        both the old & new interfaces to Jenkins are riddled with bugs, work seems to be maintenance mode, across the plugin ecosystem too If you are talking about Jenkins-X, that is a different story, it's basically a rewrite to Kubernetes. I haven't talked to anyone actually using it, if you go k8s, you are far more likely to go argo
- piker 11mo ago
  It seems like a simple CI/CD in an airgapped environment might be simpler to implement than to (1) learn and (2) onboard an off-the-shelf solution when your airgapped requirement limits your ability to leverage the off-the-shelf ecosystem.
deleted 11mo ago
[deleted]
halfcat 11mo ago
> We need to: > Build a dependency graph (which jobs need which other jobs) > Execute jobs in topological order (respecting dependencies) For what it’s worth, Python has graphlib.TopologicalSorter in the standard library that can do this, including grouping tasks that can be run in parallel: https://docs.python.org/3/library/graphlib.html https://docs.python.org/3/library/graphlib.html
- skylurk 11mo ago
  One of the best real "batteries" added in recent years.
max-privatevoid 11mo ago
Why use Docker as a build job execution engine? It seems terribly unsuited for this.
- mr_o47 11mo ago
  It's widely used among DevOps Engineers so hence I picked Docker as it makes it easier to understand
- ramon156 11mo ago
  > terribly unsuited Care to elaborate? If you already deploy in docker then wouldn't this be nice?
  - max-privatevoid 11mo ago
    Docker is unusable for build tools that use namespaces (of which Docker itself is one), unless you use privileged mode and throw away much more security than you'd need to. Docker images are difficult to reproduce with conventional Docker tools, and using a non-reproducible base image for your build environment seems like a rather bad idea.
codeonline 11mo ago
I like the iterative approach that you took with the post and codebase. It really takes the reader on a journey with you and helps understand the decision points and process of software development, it's so important compared to just showing a final polished solution, after all, we're all trying to replicate the software development process, not the product.

68 points by mr_o47 11mo ago

Why does air-gapped environment require rolling your own CI/CD solution? Plenty of examples of air-gapped Jenkins and/or Argo Workflows. Was this just an educational exercise?

mr_o47 11mo ago
This was more like an educational exercise
- esafak 11mo ago
  Since you're exercising, you can take it to the next level where you don't specify the next step but the inputs to each task, allowing you to infer the DAG and implement caching...
  - verdverm 11mo ago
    You can do this with cue/flow, but have not turned it into a full CI system. The building blocks are there
    - mr_o47 11mo ago
      Never heard of cue/flow will definitely check it out
      - verdverm 11mo ago
        cue/flow is a Go package and what powers the scripting / custom tools part of CUE. It's built on top of the language and a graph resolver. I have another custom flow implementation that I find more ergonomic: https://hofstadter.io/getting-started/task-engine/ https://hofstadter.io/getting-started/task-engine/
verdverm 11mo ago
Jenkins sucks but is insanely reliable Argo Workflows does not live up to what they advertise, it is so much more complex to setup and then build workflows for. Helm + Argo is pain (both use the same template delimiters...)
- bigstrat2003 11mo ago
  Jenkins, like many tools with extreme flexibility, sucks as much as you make it suck. You can pretty easily turn Jenkins into a monstrosity that makes everyone afraid to ever try to update it. On the other hand, you can also have a pretty solid setup that is easy to work on. The trouble is that the tool itself doesn't guide you much to the good path, so unless you've seen a pleasant Jenkins instance before you're likely to have a worse time than necessary.
  - IshKebab 11mo ago
    Are you sure, because last time I used Jenkins it actively sucked. The interface was a total mess and it doesn't surface results in any useful way.
    - fowlie 11mo ago
      When was the last time you used Jenkins? I don't get the hate. Not only from you, but lots of people on the internet. What makes Jenkins stand out IMO is the community and the core maintainers, they are perhaps moving slow, but they are moving in the right directions. The interface looks really nice now, they've done a lot of ux improvements lately.
      - verdverm 11mo ago
        both the old & new interfaces to Jenkins are riddled with bugs, work seems to be maintenance mode, across the plugin ecosystem too If you are talking about Jenkins-X, that is a different story, it's basically a rewrite to Kubernetes. I haven't talked to anyone actually using it, if you go k8s, you are far more likely to go argo
piker 11mo ago
It seems like a simple CI/CD in an airgapped environment might be simpler to implement than to (1) learn and (2) onboard an off-the-shelf solution when your airgapped requirement limits your ability to leverage the off-the-shelf ecosystem.

deleted 11mo ago

[deleted]

halfcat 11mo ago

> We need to: > Build a dependency graph (which jobs need which other jobs) > Execute jobs in topological order (respecting dependencies) For what it’s worth, Python has graphlib.TopologicalSorter in the standard library that can do this, including grouping tasks that can be run in parallel: https://docs.python.org/3/library/graphlib.html https://docs.python.org/3/library/graphlib.html

skylurk 11mo ago
One of the best real "batteries" added in recent years.

max-privatevoid 11mo ago

Why use Docker as a build job execution engine? It seems terribly unsuited for this.

mr_o47 11mo ago
It's widely used among DevOps Engineers so hence I picked Docker as it makes it easier to understand
ramon156 11mo ago
> terribly unsuited Care to elaborate? If you already deploy in docker then wouldn't this be nice?
- max-privatevoid 11mo ago
  Docker is unusable for build tools that use namespaces (of which Docker itself is one), unless you use privileged mode and throw away much more security than you'd need to. Docker images are difficult to reproduce with conventional Docker tools, and using a non-reproducible base image for your build environment seems like a rather bad idea.

codeonline 11mo ago

I like the iterative approach that you took with the post and codebase. It really takes the reader on a journey with you and helps understand the decision points and process of software development, it's so important compared to just showing a final polished solution, after all, we're all trying to replicate the software development process, not the product.