4 ms·

Having built CI/CD pipelines in CircleCI, Jenkins, GitLab CI, Concourse, Semaphore, and GitHub Actions, I'm not entirely sure what this offers, other than being

Having built CI/CD pipelines in CircleCI, Jenkins, GitLab CI, Concourse, Semaphore, and GitHub Actions, I'm not entirely sure what this offers, other than being FOSS. That's not to say that it being FOSS is not valuable – for some users it most certainly is – but that I can't see another way that this would be valuable. The config language is nice, but honestly I've never been bothered by YAML for pipelines. I want my pipelines to be declarative anyway. I'm a little disappointed that Cicada didn't go with one of the existing options such as Jsonnet, Cue, Dhall, etc. Lastly, there's a feature I've always found useful and in many cases necessary, that many CI systems don't have – enforced serial execution. When you're doing continuous delivery it's often critical to make sure that only one release is going out at a time and that the version being released strictly increases. I've seen outages because of release jobs racing and an unintended downgrade happening. This requires CI system support to achieve. Last time I checked, Jenkins, Concourse, and GitHub Actions all provided mechanisms for this, Semaphore may have as well. Circle and GitLab did not (the former after endless discussion with our account manager over it!) and I found it hard to trust the platforms as a result. Cicada does not appear to have this, which is a shame. It suggests to me a lack of hands-on production experience with continuous delivery. Arguably this is not a CD system, but there's no reason why a CI system shouldn't be CD as well, it's not until a much larger team/product size that a dedicated CD system becomes truly necessary and they take a lot of work to set up well.

digi59404 3y ago
Maybe I’m misunderstanding? For GitLab this should be doable by two different ways. The first is merge trains, which merges requests one by one to prevent this exact outcome. You just have to force all deployments to be done via a merge request. That’s the downside. The second being forcing a GitLab Runner to run one job at a time. Tag it as “deployer” then ensure all deployment jobs are marked as “deployer”. That runner will pick up deployment jobs one by one in order of first creation.
- danpalmer 3y ago
  I think merge trains weren't a feature when I worked with GitLab, it was a while ago. This does certainly solve it, but at the cost of quite a different process. If what you're optimising for is time to release, merge trains add an overhead. At some point that overhead is worth it, but it depends on the team/product/etc. Having just one runner be the deployer is an option too. I think we used hosted runners so not sure if this is possible in that setup? This would also make pipelines harder to optimise. Often there are many parts in pipelines that are safe to do in parallel, and only a few "critical sections" around which you want locking. This would solve simultaneous releases, but not the general case of the problem (which at least Jenkins and GitHub Actions manage ok). GitLab always felt to me like Travis++, whereas systems developed later felt like they were built on fundamentally better primitives. Jenkins is a weird one because it has all of the features, can do all of these things, really quite well in many cases, but has a pretty bad developer experience and required a lot of maintenance to run a performant and secure install.
- tsak 3y ago
  There's also a third way called resource groups [1]. We use that to ensure that we only run the newest job if we have multiple deployment jobs waiting for execution. This way even if we have multiple pipelines racing each other, only the last deployment job wins. [1] https://docs.gitlab.com/ee/ci/resource_groups/index.html https://docs.gitlab.com/ee/ci/resource_groups/index.html
  - imp0cat 3y ago
    Yeah, we use that instead of dedicated runners, too. But it is a relatively new Gitlab concept, isn't it?
    - tsak 3y ago
      We've been using it for a year, as we often had the problem that somebody would push an update after an initial publish of a branch, not stopping the first branch and then two or more deploy jobs racing each other with unpredictable results.
  - ranguna 3y ago
    I was a bit confused when I first saw this in the context of deployment jobs. If you go with oldest first and two or more prod jobs want to run, you are gonna have to wait for all the old deployments to finish unless you go to each one and cancel them, leaving only the latest one. This does prevent unorderd deployments from overwriting each other, but it's a pita. If you go with newest first, old deployments will overwrite new ones unless you toggle the setting to prevent outdated jobs from running, by which case you'll be locked out of doing a rollback unless you regenerate a whole old pipeline. It would make sense if they had a "newest only", where if you have one deployment running and 10 pending, by the time the current deployment finishes, gitlab would cancel all deployments except for the latest one. This way you don't have to wait for old deployments and you're free to do a rollback at any time. Bonus points if the cancelled jobs display a link to the latest job that was chosen from the queue.
    - tsak 3y ago
      I've just checked our code, and we use `oldest_first`, not `newest_first`.
matrss 3y ago
> I want my pipelines to be declarative anyway. Is there any CI system that allows one to write declarative pipelines? What would that even mean? GitHub Actions, GitLab Pipelines, etc. are all effectively just shell scripts disguised as more or less verbose yaml, with some trigger conditions added. I guess nix's hydra is the closest to a declarative CI system that exists, because nix does the hard work of abstracting the imperative build steps into a declarative interface. Even then, if you want to do anything outside of nix derivations you would be writing something imperative.
- pydry 3y ago
  it's github actions but you keep it short and put everything you can in a script that gets shelled out to.
  - matrss 3y ago
    So, only use the CI system for it's dispatch/scheduling part and put the imperative steps into a dedicated script? Reasonable, and what I have done with some GitLab Pipelines as well, but I don't see how that would be considered more "declarative".
  - summarity 3y ago
    Buildkite was once that.
- tonyhb 3y ago
  Check out https://dagger.io/ https://dagger.io/. Write declarative pipelines in code, reproducibly run anywhere.
- danpalmer 3y ago
  I'm not looking for literally everything to be declarative, but ideally each snippet of shell script would be entirely independent and run effectively reproducibly. GitHub/GitLab do this well enough, although not perfectly. Concourse does this really well. Bazel is also great at this.
  - 3y ago
dosisod 3y ago
Creator of Cicada here. Thank you for the feedback! I've mentioned this in a few threads already, but the reason for making a new DSL for writing the workflows is that YAML makes it hard/cumbersome to express more complex workflows. Using a programming language though gives you more control over how your workflows execute. While there are already plenty of tools that use existing programming languages (ie, Python/TypeScript) to configure workflows, having a custom DSL allows you to make some of the more abstract terms like caching, conditional execution, permissions, etc. more explicit. To your last point, I have experience with using CD in production, but not to the scale where I have builds stepping over each other and causing issues. I agree that serial builds are important in this case, and is something that I will need to look into (conceptually it sounds pretty simple).
- melezhik 3y ago
  You might want to check out the SparrowCI - it has combination of yaml AND standard programming languages ( Python/Ruby/GoLang/Powershell/Bash/Raku) giving you the best of two worlds - declarative style and extremely flexible flows when required
- danpalmer 3y ago
  Awesome! I think serial builds should be fairly simple to implement if you can tie it back to a transaction in a data store. I get the DSL desire, and I feel I've already lost the "YAML is fine" battle elsewhere so that's not a problem. I think a language like jsonnet or cue would have been a better choice simply because they don't involve users learning a new language or you from implementing a new language. Both would have allowed plugging in your own standard library of functions and abstractions.
deleted 3y ago
[deleted]

digi59404 3y ago

Maybe I’m misunderstanding? For GitLab this should be doable by two different ways. The first is merge trains, which merges requests one by one to prevent this exact outcome. You just have to force all deployments to be done via a merge request. That’s the downside. The second being forcing a GitLab Runner to run one job at a time. Tag it as “deployer” then ensure all deployment jobs are marked as “deployer”. That runner will pick up deployment jobs one by one in order of first creation.

danpalmer 3y ago
I think merge trains weren't a feature when I worked with GitLab, it was a while ago. This does certainly solve it, but at the cost of quite a different process. If what you're optimising for is time to release, merge trains add an overhead. At some point that overhead is worth it, but it depends on the team/product/etc. Having just one runner be the deployer is an option too. I think we used hosted runners so not sure if this is possible in that setup? This would also make pipelines harder to optimise. Often there are many parts in pipelines that are safe to do in parallel, and only a few "critical sections" around which you want locking. This would solve simultaneous releases, but not the general case of the problem (which at least Jenkins and GitHub Actions manage ok). GitLab always felt to me like Travis++, whereas systems developed later felt like they were built on fundamentally better primitives. Jenkins is a weird one because it has all of the features, can do all of these things, really quite well in many cases, but has a pretty bad developer experience and required a lot of maintenance to run a performant and secure install.
tsak 3y ago
There's also a third way called resource groups [1]. We use that to ensure that we only run the newest job if we have multiple deployment jobs waiting for execution. This way even if we have multiple pipelines racing each other, only the last deployment job wins. [1] https://docs.gitlab.com/ee/ci/resource_groups/index.html https://docs.gitlab.com/ee/ci/resource_groups/index.html
- imp0cat 3y ago
  Yeah, we use that instead of dedicated runners, too. But it is a relatively new Gitlab concept, isn't it?
  - tsak 3y ago
    We've been using it for a year, as we often had the problem that somebody would push an update after an initial publish of a branch, not stopping the first branch and then two or more deploy jobs racing each other with unpredictable results.
- ranguna 3y ago
  I was a bit confused when I first saw this in the context of deployment jobs. If you go with oldest first and two or more prod jobs want to run, you are gonna have to wait for all the old deployments to finish unless you go to each one and cancel them, leaving only the latest one. This does prevent unorderd deployments from overwriting each other, but it's a pita. If you go with newest first, old deployments will overwrite new ones unless you toggle the setting to prevent outdated jobs from running, by which case you'll be locked out of doing a rollback unless you regenerate a whole old pipeline. It would make sense if they had a "newest only", where if you have one deployment running and 10 pending, by the time the current deployment finishes, gitlab would cancel all deployments except for the latest one. This way you don't have to wait for old deployments and you're free to do a rollback at any time. Bonus points if the cancelled jobs display a link to the latest job that was chosen from the queue.
  - tsak 3y ago
    I've just checked our code, and we use `oldest_first`, not `newest_first`.

matrss 3y ago

> I want my pipelines to be declarative anyway. Is there any CI system that allows one to write declarative pipelines? What would that even mean? GitHub Actions, GitLab Pipelines, etc. are all effectively just shell scripts disguised as more or less verbose yaml, with some trigger conditions added. I guess nix's hydra is the closest to a declarative CI system that exists, because nix does the hard work of abstracting the imperative build steps into a declarative interface. Even then, if you want to do anything outside of nix derivations you would be writing something imperative.

pydry 3y ago
it's github actions but you keep it short and put everything you can in a script that gets shelled out to.
- matrss 3y ago
  So, only use the CI system for it's dispatch/scheduling part and put the imperative steps into a dedicated script? Reasonable, and what I have done with some GitLab Pipelines as well, but I don't see how that would be considered more "declarative".
- summarity 3y ago
  Buildkite was once that.
tonyhb 3y ago
Check out https://dagger.io/ https://dagger.io/. Write declarative pipelines in code, reproducibly run anywhere.
danpalmer 3y ago
I'm not looking for literally everything to be declarative, but ideally each snippet of shell script would be entirely independent and run effectively reproducibly. GitHub/GitLab do this well enough, although not perfectly. Concourse does this really well. Bazel is also great at this.
- 3y ago

dosisod 3y ago

Creator of Cicada here. Thank you for the feedback! I've mentioned this in a few threads already, but the reason for making a new DSL for writing the workflows is that YAML makes it hard/cumbersome to express more complex workflows. Using a programming language though gives you more control over how your workflows execute. While there are already plenty of tools that use existing programming languages (ie, Python/TypeScript) to configure workflows, having a custom DSL allows you to make some of the more abstract terms like caching, conditional execution, permissions, etc. more explicit. To your last point, I have experience with using CD in production, but not to the scale where I have builds stepping over each other and causing issues. I agree that serial builds are important in this case, and is something that I will need to look into (conceptually it sounds pretty simple).

melezhik 3y ago
You might want to check out the SparrowCI - it has combination of yaml AND standard programming languages ( Python/Ruby/GoLang/Powershell/Bash/Raku) giving you the best of two worlds - declarative style and extremely flexible flows when required
danpalmer 3y ago
Awesome! I think serial builds should be fairly simple to implement if you can tie it back to a transaction in a data store. I get the DSL desire, and I feel I've already lost the "YAML is fine" battle elsewhere so that's not a problem. I think a language like jsonnet or cue would have been a better choice simply because they don't involve users learning a new language or you from implementing a new language. Both would have allowed plugging in your own standard library of functions and abstractions.

deleted 3y ago

[deleted]