5 ms·

The CD Pipeline Manifesto

89 points by bullcitydev 2y ago

cdaringe 2y ago
Well written. Ive long held these values, and never could express them so concisely. Kudos! I look forward to seeing some matrix eval of impl strategies against these values
deleted 2y ago
[deleted]
mdaniel 2y ago
I'm guessing this is relevant: https://news.ycombinator.com/item?id=42267316 https://news.ycombinator.com/item?id=42267316 Show HN: Glu – Deployment pipeline framework as code - Nov, 2024 - 2 comments And, tellingly, it seems they still haven't provided a "why not ${other tool}" anywhere that I can readily spot
- esafak 2y ago
  You beat me to it: why not Dagger?
jiggawatts 2y ago
I had a look at the example glu deployment pipeline and I’m decidedly unimpressed. Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate! Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script every step is expected to fail in novel and interesting ways! This isn’t “normal code” where fallible external I/O interactions are few and far between, so error handling overhead is amortised over many lines of logic! Instead the code becomes all error handling with logic… in there… somewhere. Good luck even spotting it. Second, I don’t see the benefit of glu (specifically) over established IaC systems such as Pulumi — which is polyglot and allows the use of languages that aren’t mostly repetitive error handling ceremony. This seems like an internally developed tool that suits the purposes of a single org “thrown over the fence” in the hope that the open source community will contribute to their private tool.
- cdaringe 2y ago
  ocamlci is an OCaml Platform offered canned recipe, a la glu, and they really cut that boilerplate down. I almost never use, but i had the same vibes as you did and it made me think of an impl i thought glu may have something to learng from. https://github.com/ocurrent/ocaml-ci?tab=readme-ov-file https://github.com/ocurrent/ocaml-ci?tab=readme-ov-file
paweladamczuk 2y ago
> Without types, it is difficult to compose pipelines together. I would gladly hear this argument expanded. It's really not obvious to me that that's the case.
- wesselbindt 2y ago
  Suppose I give you two functions f, and g. Can you run f(g()) without breaking things? The honest answer is you don't know until you read the functions, which is a slow and difficult thing to do. Suppose I give you functions f and g of respective types int -> str and Nothing -> str. Can you compose them? No, and you see this immediately from the types. Types make reasoning about composability a lot easier. Of course, it's not a panacea, and it's less helpful the more side effects a function has. Can we compose pure int->int functions? Of course! Can we compose two of them where the second expects some image to exist in some docker registry? You'll need to read the first to be able to tell. Given the highly side effectful nature of pipelines, I'd think the applicability of types would be limited. But maybe that's just a lack of imagination on my part. Certainly information like "this pipeline expects these variables" and "this pipeline sets these variables" are susceptible to a typed approach, and it would make things easier. By how much, I don't know.
  - joshAg 2y ago
    What are side-effects but undocumented arguments and returns? Firstly, you want to ensure your functions are pure with respect to input. That is to say, they might reference a configuration or context object that is passed to them as an argument, but they'll never reference some global object/variable. So then the docker image inside some docker registry? Both the image and the registry are values in the config/context argument at the least. Maybe they're their own separate arguments depending on whether you prefer a single big object argument or a bunch of smaller more primitive arguments. So then the pure function that expects the docker image to exist in some registry is no longer Int -> Int It's now String -> String -> Int -> Int because it needs a registry and an image. Maybe it's String -> String -> String -> String -> Int -> Int because there's a username and password required to access the registry. Icky, but if we make a few types like data Registry { user :: String, password :: String, url :: String } that becomes Registry -> String -> Int But we could make it better by doing something like data Foo { reg:: Registry, image :: String } and now the function can be Foo -> Int -> Int This doesn't fix the image not actually existing in the registry, but at least now we know that the two functions aren't composable, and when it fails because the image doesn't exist we can hopefully trace through to see who the caller is that's giving it incorrect data. PS: sorry if i got the haskell typing wrong. I don't know haskell so that's the result of what i could cobble together from googling about haskell type syntax
sourishkrout 2y ago
I commend anyone who’s taking a hard look at our current CI/CD practices. Good work! Succinctly stating the problems is easier said than done. I believe https://dagger.io https://dagger.io checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.
- GrumpyCat42 2y ago
  My company is currently adopting this and I don't see the appeal yet - likely from a lack of knowing much about it. I added it to a side project just to get familiar and it added quite a few sdk files and folders to my project, and lots of decorators. It also required Docker and yadda yadda yadda. I just could not justify using it compared to just running some regular Typescript file with Bun (or, in a different project, `go run cmd/ci/main.go`)
  - sourishkrout 2y ago
    Simple problems require simple solutions. If Makefile, NPM run, or Rake gets the job done, stick with it. That's great. The problem that Dagger and similar efforts solve is for pipelines at scale, whether that's a sea of microservices maintained by an armada of teams (which never work the same) or your massive pipelines that should be decomposed into a more atomic pipeline that fed into one. I believe the latter is a big productivity hurdle even without org-scale. My release pipeline runs for 25min with a team <5 because it's multi-staged (testing pyramid) and includes end-to-end tests. I love my pipeline because it makes me feel safe releasing my software upon success. However, god forbid it fails with a non-obvious error of 20 minutes into exec. Lack of portability (Hi GHA vendor lock-in) and reproducibility (local-run = impossible) will make this feedback-loop hell. Now, wiseguys might tell me that pipelines shouldn't run for multiple minutes and only unit tests blah. That's divorced from reality. This sentiment won't not solve automation problems and won't optimize for velocity. It merely throws it over the fence to somebody else. If you have "the luxury" a QE/QA/Release team which I feel bad for. So the question to ask yourself is: how do I know I have outgrown `go run cmd/ci/main.go`?
deleted 2y ago
[deleted]
moltar 2y ago
Best pipeline I’ve had the pleasure to design is AWS CodePipeline via AWS CDK. Ticks all boxes. Uses pure TypeScript code.
deleted 2y ago
[deleted]
clvx 2y ago
I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.
- ttyprintk 2y ago
  This is important for SBOM (software bill-of-materials) which will soon be mandatory in regulated domains.
  - eddsolves 2y ago
    What will SBOM require in regulated domains?
    - ttyprintk 2y ago
      It's already required in Federal procurement and if you're seeking FDA approval, they've indicated you should prepare for it. I'd wager it reaches other regulated domains once the standards, experience, and tooling stabilize.
jas39 2y ago
I'm thinking before we build a CI/CI pipeline, make sure there is a Makefile. Why have they gone out of style?
- ericyd 2y ago
  Not sure but my guess is because they aren't a good fit for many languages. If you need a task runner then often languages will have a built in option or there are better alternatives than Make. If you need a build system then Make isn't a good fit for a lot of modern languages.
  - SOLAR_FIELDS 2y ago
    I’ve been using earthly a lot lately and its general value prop is simple: it turns out that if Buildkit is your primary build tool that Make targets can almost always be represented as OCI image layers. The killer feature IMO is that its syntax is familiar enough to end users of both Make and Dockerfiles that engineers tend to be willing to onboard to it. A lot of these other solutions that use proprietary DSL’s struggle to cover every use case, and the implementations in turing complete typical language SDK approach often forces you into analysis paralysis if there is no existing pattern.
- maccard 2y ago
  If I want to build and test a golang app and push it to a container repository, what value does a makefile provide over go build && docker push? All the tools do their own dependency tracking already (unfortunately).
  - drowsspa 2y ago
    Honestly I like it just to keep it as a command runner with the needed flags. Then in the unholy YAML there's just make build, make test, etc
    - pdimitar 2y ago
      The `just` tool is a better and much easier to understand command runner than `make`, however. Much less feature surface, too, which eliminates nasty surprises coming from the unnecessary complexity of `make`.
azeirah 2y ago
None of these are a problem anymore since the advent of Nix.
aarmenaa 2y ago
FTA: > The Fix: Use a full modern programming language, with its existing testing frameworks and tooling. I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not because I like Python (I'd rather Ruby) but because it is ubiquitous and completely sufficient for describing a CI/CD pipeline. I'm firmly of the opinion that once we start implementing "the power of real code: loops, conditionals, runtime logic, standard libraries, and more" in YAML then YAML was the wrong choice. I absolutely despise Ansible for the same reason and wish I could still write Chef cookbooks.
- mqus 2y ago
  I don't think I agree. I've now seen the 'language' approach in jenkins and the static yaml file approach in gitlab and drone. A lot of value is to be gained if the whole script can be analysed statically, before execution. E.g. UI Elements can be there and the whole pipeline is visible, before even starting it. It also serves as a natural sandbox for the "setup" part so we can always know that in a finite (and short) timeline, the script is interpreted and no weird stuff can ever happen. Of course, there are ways to combine it (e.g. gitlab can generate and then trigger downstream pipelines from within the running CI, but the default is the script. It also has the side effect that pipeline setup can't ever do stuff that cannot be debugged (because it's running _before_ the pipeline) But I concede that this is not that clear-cut. Both have advantages.
  - aarmenaa 2y ago
    If you manage to avoid scope creep then sure, static YAML has advantages. But that's not usually what happens, is it? The minute you allow users to execute an outside program -- which is strictly necessary for a CI/CD system -- you've already lost. But even if we ignore that, the number of features always grows over time: you add variables so certain elements can be re-used, then you add loops and conditionals because some things need to happen multiple times, and then you add the ability to do math, string manipulation is always useful, and so on. Before you know it you're trying to solve the halting problem because your "declarative markup" is a poorly specified turing-complete language that just happens to use a YAML parser as a tokenizer. This bespoke language will be strictly worse than Python in every way. My argument is that we should acknowledge that any CI/CD system intended for wide usage will eventually arrive here, and it's better that we go into that intentionally rather than accidentally.
rat87 2y ago
> Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort. So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub/gitlab/etc? I do agree with them about the need for a real programming language. I hate yaml in gitlabs config, it is very hard to interpret how it will be interpreted. Things were much easier when I was scripting Jenkins even though I didn't know or like groovy then with gitlab
- cdaringe 2y ago
  > So are they talking about … meta language Said kindly, no, they’re not. They’re just stating values here, imho, not impl detail
vergessenmir 2y ago
Is any of this reproducible? Not sure why that requirement has been quietly overlooked. I've worked in this space for a long time and can't make head or tail of what glue is. A motivating examplt would be help which I might have missed?
a1o 2y ago
I don't think this is reasonable if you have a cross platform (web, android, iOS, macOS, Windows, Linux, FreeBSD, ...) app, things won't be that clean, that only works if whatever you do is very simple, otherwise there will be some patchwork to build and test across all platforms - there's just no way to run all of them local in your single platform computer whatever that is. Honestly a lot of what is there is not that useful, I don't need types in the pipeline when scripting python for build integration logic, there's nothing that types brings that are a must in that case.

89 points by bullcitydev 2y ago

Well written. Ive long held these values, and never could express them so concisely. Kudos! I look forward to seeing some matrix eval of impl strategies against these values

deleted 2y ago

[deleted]

mdaniel 2y ago

I'm guessing this is relevant: https://news.ycombinator.com/item?id=42267316 https://news.ycombinator.com/item?id=42267316 Show HN: Glu – Deployment pipeline framework as code - Nov, 2024 - 2 comments And, tellingly, it seems they still haven't provided a "why not ${other tool}" anywhere that I can readily spot

esafak 2y ago
You beat me to it: why not Dagger?

jiggawatts 2y ago

I had a look at the example glu deployment pipeline and I’m decidedly unimpressed. Admittedly most of my criticism is related to the choice of Go as an implementation language: more than 80% of the code volume is error handling boilerplate! Before the lovers of Go start making the usual arguments consider that in a high-level pipeline script every step is expected to fail in novel and interesting ways! This isn’t “normal code” where fallible external I/O interactions are few and far between, so error handling overhead is amortised over many lines of logic! Instead the code becomes all error handling with logic… in there… somewhere. Good luck even spotting it. Second, I don’t see the benefit of glu (specifically) over established IaC systems such as Pulumi — which is polyglot and allows the use of languages that aren’t mostly repetitive error handling ceremony. This seems like an internally developed tool that suits the purposes of a single org “thrown over the fence” in the hope that the open source community will contribute to their private tool.

cdaringe 2y ago
ocamlci is an OCaml Platform offered canned recipe, a la glu, and they really cut that boilerplate down. I almost never use, but i had the same vibes as you did and it made me think of an impl i thought glu may have something to learng from. https://github.com/ocurrent/ocaml-ci?tab=readme-ov-file https://github.com/ocurrent/ocaml-ci?tab=readme-ov-file

paweladamczuk 2y ago

> Without types, it is difficult to compose pipelines together. I would gladly hear this argument expanded. It's really not obvious to me that that's the case.

wesselbindt 2y ago
Suppose I give you two functions f, and g. Can you run f(g()) without breaking things? The honest answer is you don't know until you read the functions, which is a slow and difficult thing to do. Suppose I give you functions f and g of respective types int -> str and Nothing -> str. Can you compose them? No, and you see this immediately from the types. Types make reasoning about composability a lot easier. Of course, it's not a panacea, and it's less helpful the more side effects a function has. Can we compose pure int->int functions? Of course! Can we compose two of them where the second expects some image to exist in some docker registry? You'll need to read the first to be able to tell. Given the highly side effectful nature of pipelines, I'd think the applicability of types would be limited. But maybe that's just a lack of imagination on my part. Certainly information like "this pipeline expects these variables" and "this pipeline sets these variables" are susceptible to a typed approach, and it would make things easier. By how much, I don't know.
- joshAg 2y ago
  What are side-effects but undocumented arguments and returns? Firstly, you want to ensure your functions are pure with respect to input. That is to say, they might reference a configuration or context object that is passed to them as an argument, but they'll never reference some global object/variable. So then the docker image inside some docker registry? Both the image and the registry are values in the config/context argument at the least. Maybe they're their own separate arguments depending on whether you prefer a single big object argument or a bunch of smaller more primitive arguments. So then the pure function that expects the docker image to exist in some registry is no longer Int -> Int It's now String -> String -> Int -> Int because it needs a registry and an image. Maybe it's String -> String -> String -> String -> Int -> Int because there's a username and password required to access the registry. Icky, but if we make a few types like data Registry { user :: String, password :: String, url :: String } that becomes Registry -> String -> Int But we could make it better by doing something like data Foo { reg:: Registry, image :: String } and now the function can be Foo -> Int -> Int This doesn't fix the image not actually existing in the registry, but at least now we know that the two functions aren't composable, and when it fails because the image doesn't exist we can hopefully trace through to see who the caller is that's giving it incorrect data. PS: sorry if i got the haskell typing wrong. I don't know haskell so that's the result of what i could cobble together from googling about haskell type syntax

sourishkrout 2y ago

I commend anyone who’s taking a hard look at our current CI/CD practices. Good work! Succinctly stating the problems is easier said than done. I believe https://dagger.io https://dagger.io checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.

GrumpyCat42 2y ago
My company is currently adopting this and I don't see the appeal yet - likely from a lack of knowing much about it. I added it to a side project just to get familiar and it added quite a few sdk files and folders to my project, and lots of decorators. It also required Docker and yadda yadda yadda. I just could not justify using it compared to just running some regular Typescript file with Bun (or, in a different project, `go run cmd/ci/main.go`)
- sourishkrout 2y ago
  Simple problems require simple solutions. If Makefile, NPM run, or Rake gets the job done, stick with it. That's great. The problem that Dagger and similar efforts solve is for pipelines at scale, whether that's a sea of microservices maintained by an armada of teams (which never work the same) or your massive pipelines that should be decomposed into a more atomic pipeline that fed into one. I believe the latter is a big productivity hurdle even without org-scale. My release pipeline runs for 25min with a team <5 because it's multi-staged (testing pyramid) and includes end-to-end tests. I love my pipeline because it makes me feel safe releasing my software upon success. However, god forbid it fails with a non-obvious error of 20 minutes into exec. Lack of portability (Hi GHA vendor lock-in) and reproducibility (local-run = impossible) will make this feedback-loop hell. Now, wiseguys might tell me that pipelines shouldn't run for multiple minutes and only unit tests blah. That's divorced from reality. This sentiment won't not solve automation problems and won't optimize for velocity. It merely throws it over the fence to somebody else. If you have "the luxury" a QE/QA/Release team which I feel bad for. So the question to ask yourself is: how do I know I have outgrown `go run cmd/ci/main.go`?

deleted 2y ago

[deleted]

moltar 2y ago

Best pipeline I’ve had the pleasure to design is AWS CodePipeline via AWS CDK. Ticks all boxes. Uses pure TypeScript code.

deleted 2y ago

[deleted]

clvx 2y ago

I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.

ttyprintk 2y ago
This is important for SBOM (software bill-of-materials) which will soon be mandatory in regulated domains.
- eddsolves 2y ago
  What will SBOM require in regulated domains?
  - ttyprintk 2y ago
    It's already required in Federal procurement and if you're seeking FDA approval, they've indicated you should prepare for it. I'd wager it reaches other regulated domains once the standards, experience, and tooling stabilize.

jas39 2y ago

I'm thinking before we build a CI/CI pipeline, make sure there is a Makefile. Why have they gone out of style?

ericyd 2y ago
Not sure but my guess is because they aren't a good fit for many languages. If you need a task runner then often languages will have a built in option or there are better alternatives than Make. If you need a build system then Make isn't a good fit for a lot of modern languages.
- SOLAR_FIELDS 2y ago
  I’ve been using earthly a lot lately and its general value prop is simple: it turns out that if Buildkit is your primary build tool that Make targets can almost always be represented as OCI image layers. The killer feature IMO is that its syntax is familiar enough to end users of both Make and Dockerfiles that engineers tend to be willing to onboard to it. A lot of these other solutions that use proprietary DSL’s struggle to cover every use case, and the implementations in turing complete typical language SDK approach often forces you into analysis paralysis if there is no existing pattern.
maccard 2y ago
If I want to build and test a golang app and push it to a container repository, what value does a makefile provide over go build && docker push? All the tools do their own dependency tracking already (unfortunately).
- drowsspa 2y ago
  Honestly I like it just to keep it as a command runner with the needed flags. Then in the unholy YAML there's just make build, make test, etc
  - pdimitar 2y ago
    The `just` tool is a better and much easier to understand command runner than `make`, however. Much less feature surface, too, which eliminates nasty surprises coming from the unnecessary complexity of `make`.

azeirah 2y ago

None of these are a problem anymore since the advent of Nix.

aarmenaa 2y ago

FTA: > The Fix: Use a full modern programming language, with its existing testing frameworks and tooling. I was reading the article and thinking myself "a lot of this is fixed if the pipeline is just a Python script." And really, if I was to start building a new CI/CD tool today the "user facing" portion would be a Python library that contains helper functions for interfacing with with the larger CI/CD system. Not because I like Python (I'd rather Ruby) but because it is ubiquitous and completely sufficient for describing a CI/CD pipeline. I'm firmly of the opinion that once we start implementing "the power of real code: loops, conditionals, runtime logic, standard libraries, and more" in YAML then YAML was the wrong choice. I absolutely despise Ansible for the same reason and wish I could still write Chef cookbooks.

mqus 2y ago
I don't think I agree. I've now seen the 'language' approach in jenkins and the static yaml file approach in gitlab and drone. A lot of value is to be gained if the whole script can be analysed statically, before execution. E.g. UI Elements can be there and the whole pipeline is visible, before even starting it. It also serves as a natural sandbox for the "setup" part so we can always know that in a finite (and short) timeline, the script is interpreted and no weird stuff can ever happen. Of course, there are ways to combine it (e.g. gitlab can generate and then trigger downstream pipelines from within the running CI, but the default is the script. It also has the side effect that pipeline setup can't ever do stuff that cannot be debugged (because it's running _before_ the pipeline) But I concede that this is not that clear-cut. Both have advantages.
- aarmenaa 2y ago
  If you manage to avoid scope creep then sure, static YAML has advantages. But that's not usually what happens, is it? The minute you allow users to execute an outside program -- which is strictly necessary for a CI/CD system -- you've already lost. But even if we ignore that, the number of features always grows over time: you add variables so certain elements can be re-used, then you add loops and conditionals because some things need to happen multiple times, and then you add the ability to do math, string manipulation is always useful, and so on. Before you know it you're trying to solve the halting problem because your "declarative markup" is a poorly specified turing-complete language that just happens to use a YAML parser as a tokenizer. This bespoke language will be strictly worse than Python in every way. My argument is that we should acknowledge that any CI/CD system intended for wide usage will eventually arrive here, and it's better that we go into that intentionally rather than accidentally.

rat87 2y ago

> Pipeline definitions are scattered across multiple tools—GitHub Actions, Jenkins, ArgoCD, Kubernetes—and environments. This fragmentation leads to confusion, configuration drift, and duplicated effort. So are they talking about some sort of meta language compiling into multiple yaml configs for the different environments or a single separate CI tool that has plugins and integrates with GitHub/gitlab/etc? I do agree with them about the need for a real programming language. I hate yaml in gitlabs config, it is very hard to interpret how it will be interpreted. Things were much easier when I was scripting Jenkins even though I didn't know or like groovy then with gitlab

cdaringe 2y ago
> So are they talking about … meta language Said kindly, no, they’re not. They’re just stating values here, imho, not impl detail

vergessenmir 2y ago

Is any of this reproducible? Not sure why that requirement has been quietly overlooked. I've worked in this space for a long time and can't make head or tail of what glue is. A motivating examplt would be help which I might have missed?

a1o 2y ago

I don't think this is reasonable if you have a cross platform (web, android, iOS, macOS, Windows, Linux, FreeBSD, ...) app, things won't be that clean, that only works if whatever you do is very simple, otherwise there will be some patchwork to build and test across all platforms - there's just no way to run all of them local in your single platform computer whatever that is. Honestly a lot of what is there is not that useful, I don't need types in the pipeline when scripting python for build integration logic, there's nothing that types brings that are a must in that case.