3 ms·

YAML is fine for what it is: a markup language. I have no problem with it being used in simple configuration files, for instance. However, CI is not "configure

by deng 2y ago

YAML is fine for what it is: a markup language. I have no problem with it being used in simple configuration files, for instance. However, CI is not "configured", it is coded. It is simply the wrong tool. YAML was continuously extended to deal with that, so it developed into much more than just "markup", but it grew into this terrible chimera. Once you start using advanced features in GitLab's YAML like anchors and references to avoid writing the same stuff again and again, you'll notice that the whole tooling around YAML is simply not there. How does the resulting YAML look like? How do you run this stuff locally? How do you debug this? Just don't go there. You will not be able to avoid YAML completely, obviously, but use it the way it was originally intended to.

LeonM 2y ago
> CI is not "configured", it is coded. Finally! I was always struggling to explain to others why YAML is OK-ish as a language, but then never seems to work well for the things people tried doing with it. Especially stuff that needs to run commands, such as CI. > How does the resulting YAML look like? How do you run this stuff locally? How do you debug this? Just don't go there. Agreed. GitHub actions, or any remote CI runner for that matter, makes the problem even worse. The whole cycle of having to push CI code, wait 10 minutes while praying for it to work, still getting an error, trying to figure out the mistake, fixing one subtle syntax error, then pushing the code again in the hope that that works is just a terrible workflow. Massive waste of time. > You will not be able to avoid YAML completely, obviously, but use it the way it was originally intended to. Even for configurations YAML remains a pain, unfortunately. It could have been great for configs, but in my experience the whole strict whitespace (tabs-vs-spaces) part ruined it. It isn't a problem when you work from an IDE that protects you from accidentally using tabs (also, auto-formatting for the win!) but when you have to write YAML configuration (for example: Netplan) on a remote server using just an editor it quickly becomes a game of whack-a-mole.
- motorest 2y ago
  > Especially stuff that needs to run commands, such as CI. I don't understand what problem you could possibly be experiencing. What exactly do you find hard about running commands in, say, GitLab CICD?
  - cmsj 2y ago
    So, I'm not interested in the debate about the correctness (or otherwise) of yaml as a declarative programming language, but I will say this... iterating a GitHub Actions workflow is a gigantic pain in the ass. Capturing all of the important logic in a script/makefile/whatever means I can iterate it locally way faster and then all I need github to do is provision an environment and call my scripts in the order I require.
    - motorest 2y ago
      > iterating a GitHub Actions workflow is a gigantic pain in the ass. Capturing all of the important logic in a script/makefile/whatever means I can iterate it locally way faster and then all I need github to do is provision an environment and call my scripts in the order I require. What's wrong with this? https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/adding-scripts-to-your-workflow https://docs.github.com/en/actions/writing-workflows/choosin...
      - kbolino 2y ago
        When it gets realistic, with conditions, variable substitutions, etc., it ends up being 20 steps in a language that isn't shell but is calling shell over and over again, and can't be run outside of CI. Whereas, if you just wrote one shell script, it could've done all of those things in one language and been runnable locally too.
        motorest 2y ago
        > When it gets realistic, with conditions, variable substitutions, etc., What exactly do you find hard in writing your own scripts with a scripting language? Surely you are not a software developer who feels conditionals and variable substitutions are hard. > it ends up being 20 steps in a language that isn't shell but is calling shell over and over again, and can't be run outside of CI. Why are you writing your CICD scripts in a way that you cannot run them outside of a CICD pipeline? I mean, you're writing them yourself, aren't you? Why are you failing to meet your own requirements? If you have a requirement to run your own scripts outside of a pipeline, how come you're not writing them like that? It's CICD 101 that those scripts should be runnable outside of the pipeline. From your description, you're failing to even follow the most basic recommendations and best practices. Why? That doesn't sound like a YAML problem, does it?
motorest 2y ago
> However, CI is not "configured", it is coded. No, it really isn't. I'll clarify why. Pretty much all pipeline services share the same architecture pattern: * A pipeline run is comprised of one or more build jobs, * Pipeline runs are triggered by external events * Build jobs have contexts and can output artifacts, * Build jobs are grouped into stages, * Stages are organized as a directed graph, * Transitions between stages in the directed graph is ruled by a set of rules, some supported by default (i.e., if a job fails then the stage fails) complemented by custom rules (manual or automatic approvals, API tests, baking periods, etc). This is the textbook scenario ideal for DSLs. You already are bound to an architecture pattern, this there is no point of reinventing the wheel each time. Just specify your stages and which jobs run as part of each stage, manage artifacts and promotion logic, and you're done. You do not need to take my word for it. Take a look at GitLab CICD for a pipeline with build, test, and delivery stage. See what a mess you will put together if you support the same feature set with whatever scripting language you choose. There is no discussion or debate.
- baq 2y ago
  I can’t understand how you can say DSL and YAML in the same sentence and say it’s fine. YAML is a serialization format. A bad DSL would be a welcome improvement over GHA pipelines in YAML. You’re fundamentally confusing concepts here, you want to restrict flexibility (I agree with that btw) by using a simplistic language, but what it actually does is increase complexity of the code comprising the pipeline with zero hard restrictions.
  - motorest 2y ago
    [flagged]
- deng 2y ago
  We don't disagree here. There are tools which support you in doing this, and I mentioned a few of them in my post (Make, Just, doit, mage). There are many more. I also think that re-inventing these tools is a waste of time, but it is still better than shoehorning this into YAML. You seem to think YAML is some kind of DSL for pipelines. It really is not.
adolph 2y ago
> However, CI is not "configured", it is coded. . . . YAML was continuously extended to deal with that, so it developed into much more than just "markup", but it grew into this terrible chimera. Brings to mind the classic "Kingdom of Nouns" [0] parable, which I read to my kid just last week. The multi-line "run" nodes in GitHub actions give me the heebie-jeebies, like how MUMPS data validation was maintained in metadata of VA-Fileman [1]. 0. https://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html https://steve-yegge.blogspot.com/2006/03/execution-in-kingdo... 1. https://www.hardhats.org/fileman/pm/gfs_frm.htm https://www.hardhats.org/fileman/pm/gfs_frm.htm
bastardoperator 2y ago
I've created multiple actions, reusable, composite, along with multiple Jenkins plugins and CircleCI Orbs. I disagree, code your actions, your jenkins plugins, your orbs, whatever. Those are just code wrappers that expose configuration via YAML or Pipeline DSL. Agreed, coding in YAML is pretty bad, but ultimately it's a choice. I will take the Actions path 100% of the time. Building your own action is so insanely simple it makes me wonder if the people complaining about YAML understand the tooling because it's entirely avoidable. It also coincides with top comments about coding your own CI, if you're just "using" YAML you're barely touching the surface.
Lammy 2y ago
> YAML is fine for what it is: a markup language. Pardon my pedantry, but the meaning of YAML's name was changed from the original “Yet Another Markup Language” to “YAML Ain't Markup Language” in a 2002 draft spec because YAML is, in fact, not a markup language :) Compare: https://yaml.org/spec/history/2001-12-10.html https://yaml.org/spec/history/2001-12-10.html https://yaml.org/spec/history/2002-04-07.html https://yaml.org/spec/history/2002-04-07.html
ruuda 2y ago
If you have to deal with tools that need to be configured with yaml, give https://rcl-lang.org/ https://rcl-lang.org/ a try! It can be "coded" to avoid duplication, with real variables, functions, and loops. It can show you the result that it evaluates to. It can do debug tracing, and it has a built-in build command to generate files like GitHub Actions workflows from an RCL file.

by deng 2y ago

LeonM 2y ago

> CI is not "configured", it is coded. Finally! I was always struggling to explain to others why YAML is OK-ish as a language, but then never seems to work well for the things people tried doing with it. Especially stuff that needs to run commands, such as CI. > How does the resulting YAML look like? How do you run this stuff locally? How do you debug this? Just don't go there. Agreed. GitHub actions, or any remote CI runner for that matter, makes the problem even worse. The whole cycle of having to push CI code, wait 10 minutes while praying for it to work, still getting an error, trying to figure out the mistake, fixing one subtle syntax error, then pushing the code again in the hope that that works is just a terrible workflow. Massive waste of time. > You will not be able to avoid YAML completely, obviously, but use it the way it was originally intended to. Even for configurations YAML remains a pain, unfortunately. It could have been great for configs, but in my experience the whole strict whitespace (tabs-vs-spaces) part ruined it. It isn't a problem when you work from an IDE that protects you from accidentally using tabs (also, auto-formatting for the win!) but when you have to write YAML configuration (for example: Netplan) on a remote server using just an editor it quickly becomes a game of whack-a-mole.

motorest 2y ago
> Especially stuff that needs to run commands, such as CI. I don't understand what problem you could possibly be experiencing. What exactly do you find hard about running commands in, say, GitLab CICD?
- cmsj 2y ago
  So, I'm not interested in the debate about the correctness (or otherwise) of yaml as a declarative programming language, but I will say this... iterating a GitHub Actions workflow is a gigantic pain in the ass. Capturing all of the important logic in a script/makefile/whatever means I can iterate it locally way faster and then all I need github to do is provision an environment and call my scripts in the order I require.
  - motorest 2y ago
    > iterating a GitHub Actions workflow is a gigantic pain in the ass. Capturing all of the important logic in a script/makefile/whatever means I can iterate it locally way faster and then all I need github to do is provision an environment and call my scripts in the order I require. What's wrong with this? https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/adding-scripts-to-your-workflow https://docs.github.com/en/actions/writing-workflows/choosin...
    - kbolino 2y ago
      When it gets realistic, with conditions, variable substitutions, etc., it ends up being 20 steps in a language that isn't shell but is calling shell over and over again, and can't be run outside of CI. Whereas, if you just wrote one shell script, it could've done all of those things in one language and been runnable locally too.
      - motorest 2y ago
        > When it gets realistic, with conditions, variable substitutions, etc., What exactly do you find hard in writing your own scripts with a scripting language? Surely you are not a software developer who feels conditionals and variable substitutions are hard. > it ends up being 20 steps in a language that isn't shell but is calling shell over and over again, and can't be run outside of CI. Why are you writing your CICD scripts in a way that you cannot run them outside of a CICD pipeline? I mean, you're writing them yourself, aren't you? Why are you failing to meet your own requirements? If you have a requirement to run your own scripts outside of a pipeline, how come you're not writing them like that? It's CICD 101 that those scripts should be runnable outside of the pipeline. From your description, you're failing to even follow the most basic recommendations and best practices. Why? That doesn't sound like a YAML problem, does it?

motorest 2y ago

> However, CI is not "configured", it is coded. No, it really isn't. I'll clarify why. Pretty much all pipeline services share the same architecture pattern: * A pipeline run is comprised of one or more build jobs, * Pipeline runs are triggered by external events * Build jobs have contexts and can output artifacts, * Build jobs are grouped into stages, * Stages are organized as a directed graph, * Transitions between stages in the directed graph is ruled by a set of rules, some supported by default (i.e., if a job fails then the stage fails) complemented by custom rules (manual or automatic approvals, API tests, baking periods, etc). This is the textbook scenario ideal for DSLs. You already are bound to an architecture pattern, this there is no point of reinventing the wheel each time. Just specify your stages and which jobs run as part of each stage, manage artifacts and promotion logic, and you're done. You do not need to take my word for it. Take a look at GitLab CICD for a pipeline with build, test, and delivery stage. See what a mess you will put together if you support the same feature set with whatever scripting language you choose. There is no discussion or debate.

baq 2y ago
I can’t understand how you can say DSL and YAML in the same sentence and say it’s fine. YAML is a serialization format. A bad DSL would be a welcome improvement over GHA pipelines in YAML. You’re fundamentally confusing concepts here, you want to restrict flexibility (I agree with that btw) by using a simplistic language, but what it actually does is increase complexity of the code comprising the pipeline with zero hard restrictions.
- motorest 2y ago
  [flagged]
deng 2y ago
We don't disagree here. There are tools which support you in doing this, and I mentioned a few of them in my post (Make, Just, doit, mage). There are many more. I also think that re-inventing these tools is a waste of time, but it is still better than shoehorning this into YAML. You seem to think YAML is some kind of DSL for pipelines. It really is not.

adolph 2y ago

> However, CI is not "configured", it is coded. . . . YAML was continuously extended to deal with that, so it developed into much more than just "markup", but it grew into this terrible chimera. Brings to mind the classic "Kingdom of Nouns" [0] parable, which I read to my kid just last week. The multi-line "run" nodes in GitHub actions give me the heebie-jeebies, like how MUMPS data validation was maintained in metadata of VA-Fileman [1]. 0. https://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html https://steve-yegge.blogspot.com/2006/03/execution-in-kingdo... 1. https://www.hardhats.org/fileman/pm/gfs_frm.htm https://www.hardhats.org/fileman/pm/gfs_frm.htm

bastardoperator 2y ago

I've created multiple actions, reusable, composite, along with multiple Jenkins plugins and CircleCI Orbs. I disagree, code your actions, your jenkins plugins, your orbs, whatever. Those are just code wrappers that expose configuration via YAML or Pipeline DSL. Agreed, coding in YAML is pretty bad, but ultimately it's a choice. I will take the Actions path 100% of the time. Building your own action is so insanely simple it makes me wonder if the people complaining about YAML understand the tooling because it's entirely avoidable. It also coincides with top comments about coding your own CI, if you're just "using" YAML you're barely touching the surface.

Lammy 2y ago

> YAML is fine for what it is: a markup language. Pardon my pedantry, but the meaning of YAML's name was changed from the original “Yet Another Markup Language” to “YAML Ain't Markup Language” in a 2002 draft spec because YAML is, in fact, not a markup language :) Compare: https://yaml.org/spec/history/2001-12-10.html https://yaml.org/spec/history/2001-12-10.html https://yaml.org/spec/history/2002-04-07.html https://yaml.org/spec/history/2002-04-07.html

ruuda 2y ago

If you have to deal with tools that need to be configured with yaml, give https://rcl-lang.org/ https://rcl-lang.org/ a try! It can be "coded" to avoid duplication, with real variables, functions, and loops. It can show you the result that it evaluates to. It can do debug tracing, and it has a built-in build command to generate files like GitHub Actions workflows from an RCL file.