6 ms·

I don't like YAML and would like to move on, but I hope we don't move onto this. I think it's crazy that when I add a string to an inline list, I may need to c

I don't like YAML and would like to move on, but I hope we don't move onto this. I think it's crazy that when I add a string to an inline list, I may need to convert that inline list to a list because this string needs different handling. I think it's crazy that "convert an inline list to a list" is a coherent statement, but that is the nomenclature that they chose. I don't like that a truncated document is a complete and valid document. But what is most unappealing is their whitespace handling. I couldn't even figure out how to encode a string with CR line endings. So, I downloaded their python client to see how it did it. Turns out, they couldn't figure it out either: >>> nt.loads(nt.dumps("\r"),top="str") '\n'

ljm 5y ago
It would be nice if YAML wasn't horrendously abused the way it is. You have CI pipelines that let you construct DAGs to represent your builds, but you need several thousand lines of YAML and a load of custom parsing to get programming constructs in the string types, for example. And then each provider has its own way of providing those. I don't have to re-read manuals describing how to do if/else in Ruby or Java or Lisp, but as soon as yaml and some 'devops' tooling is involved, I have to constantly jump back and forth between the reference and my config. The main point being that the problem isn't the file format but the products that continue to push it, presumably because hacking stuff on top of `YAML.parse` is less effort than designing something that fits the purpose.
- Lhiw 5y ago
  I'm pretty sure the format is the issue. I still don't know how to do arrays in yaml. Is it new line, tab to same indent or same indent + one space, do I need a dash? Does a dash make it an array or an object? It's just simply not that big of a deal to add a few quotes and braces to make everything make sense. The only real issue with Json is the lack of comments and strictness about extra commas.
- donmcronald 5y ago
  Yeah. A lot of times I find myself thinking YAML is like a really awful programming language. You can sort of do conditional logic and loops, but usually I find it hard to follow what's going on. For build systems, I always liked the idea of Gradle where the core functionality was simple and declarative, but with the option to use a real programming language for things that weren't simple. For example, integrating installers or form builders (pre-processing) into a build are things I would consider non-trivial if there aren't official plugins, but it was still relatively easy to do with Gradle. The biggest problem I always had with Gradle was that I didn't like Groovy and I always though there was a missed opportunity to have a statically typed build system with a solid API/contract and all the fancy tooling like auto-complete that you get with statically typed languages. I see JSON5 mentioned a lot in the comments. In terms of CI / build systems, I feel like something built with JSON5/TypeScript could be really good. I'd be really happy using TypeScript for configuring things like build systems where there shouldn't really be an argument for needing it to be usable by non-programmers. Personally I feel like I've spent way to much of my life debugging YAML syntax issues.
  - TheCycoONE 5y ago
    Does the Kotlin support in Gradle solve your problem? https://docs.gradle.org/current/userguide/kotlin_dsl.html https://docs.gradle.org/current/userguide/kotlin_dsl.html
    - spuz 5y ago
      My experience with Kotlin gradle scripts is worse than Groovy. For example, given the following valid groovy/kotlin gradle program: dependencies { } What would you expect to see between the curly braces? IntelliJ IDEA which supposedly has full support for the gradle DSL both for Groovy and Kotlin offers only generic suggestions. Common function calls such "implementation()" or "testImplementation()" are not suggested. If you do use those functions, no suggestion is made for their parameters. Because Gradle's DSL is built on top of a general purpose language, it loses the benefits of a DSL (constraining the set of possible configurations and guiding the user towards valid configurations).
- theamk 5y ago
  I really like approach of buildkite CI -- they use yaml, but this yaml can be produced by an executable script. So you write yaml by hand for trivial cases, but once it get complex, you can just drop back to shell/python/ruby/node/whatever, implement any complex logic, and serialize results to plain yaml.
- int_19h 5y ago
  What's really sad is that XML had a much better ecosystem around this for ages. I'd very much rather deal with XQuery or even XSLT to construct build trees, than the current crop of ad-hoc YAML preprocessors. At least the XML stuff had a consistent type system underneath!
  - tdeck 5y ago
    XSLT is an absolute horror and not something I would want to deal with again. It feels like some weird academic experiment in an XML declarative programming language that should never have made it to print. If something needs the flexibility of a programming language, why not use a real one that's been well tested for writing other programs? These various config file programming systems always end up creating something notorious that everyone tries to avoid having to work on.
    - int_19h 5y ago
      XQuery is, in many ways, XSLT with better syntax. It doesn't have the pattern-matching transforms that are the T in XSLT - but for configs, I don't think it makes a big difference. Also, I don't think many realize that the stack has evolved since early 00s. XSLT 1.0 was a very limiting language, requiring extensions for many advanced scenarios. But there's XSLT v3.0 these days, and XPath & XQuery v3.1, with some major new features - e.g. maps and lambdas. Granted, this doesn't fix the most basic complaint about XSLT - its insanely verbose syntax - but even then, I'd still take XSLT over ad-hoc YAML-based loops and conditionals.
  - dnautics 5y ago
    XML died because too many configurations turned what should be a 'prop' into an inner tag -- and it doesn't help that XML doesn't really give guidance as to when to use which. And, of course, when you deserialize XML, the innerText is always in a very strange place as to not really being clear "what the right way to handle it" is.
  - stjohnswarts 5y ago
    I will take the verbosity of XML any day over YAML wrestling (complex YAML configs of course). There is simply too many "implicit rules" for YAML. it's why I prefer python over ruby and perl. Generally though TOML has been good enough for me to do lots of fairly large config files that are easy for humans and machines to parse.
- thayne 5y ago
  Honestly, I think using an embedded scripting language, like lua or even javascript, would be a much better fit for these use cases than trying to make yaml do something it wasn't designed for.
  - ljm 5y ago
    Ironically, having used cdk8s[1] for dealing with kubernetes infrastructure, that's the one thing where I've actually preferred yaml. That said, k8s resource definitions are pure config so there's no need to try and hack extra bits on top of a serialized data structure. [1]https://cdk8s.io/ https://cdk8s.io/
- ilyash 5y ago
  It's always the same trajectory with declarative programming. It starts with "it's just configuration, we need something simple". Then users come with use cases which are more complex. Then you have programming language on top of configuration language syntax. * https://ilya-sher.org/2018/06/30/terraform-becomes-a-programming-language/ https://ilya-sher.org/2018/06/30/terraform-becomes-a-program... * https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a-programming-language/ https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a...
  - silon42 5y ago
    Some even start with a programming language and pretend it's declarative... IMO, it's only declarative when it's a data model which is easily parsed by multiple languages/systems where it's needed.
  - stevekemp 5y ago
    Very much so. A good few years ago I got annoyed that I couldn't have change mutt configuration the way that I wanted, because it has a built in configuration language which doesn't allow complicated conditionals etc. (There are workarounds, and off-hand I can't think of a great example, but bear with me.) In the end I wrote a simple console-based mail-client, which used a Lua configuration file. That would generally default to using hashes, and key=value settings, but over the time I used it things got really quite configurable via user-defined callbacks, and functions to return various settings. For example I wrote a hook called `on_reply_to`, and if you defined that function in your configuration file it would be invoked when you triggered the Reply function. This kind of flexibility was very self-consistent, and easy to add using an embedded real language. Later I added some hacks to a local fork of GNU Screen, there I just said: * If the ~/.screenrc file is executable, then execute it, and parse the output. That let me say "If hostname == foo; do this ; otherwise do this .." and get conditionals and some other things easily. Another example was unbinding all keys, and then only allowing some actions to be bound. (I later submitted "unbindall" upstream, to remove the need for that.)
- shusaku 5y ago
  This is the crux of the issue right? YAML is fine for most projects, but THE project for using YAML is CI configuration, for which it doesn’t match.
jrochkind1 5y ago
> So, I downloaded their python client to see how it did it. Who are you suggesting the python client belongs to, who is 'they' in 'their'?
- detaro 5y ago
  ... the people that made it and wrote the submission we are discussing? how is this even a question?
  - deleted 5y ago
    [deleted]
tialaramex 5y ago
> I don't like that a truncated document is a complete and valid document. Me either. If your documents have this property you're likely to tempt people to start trying to process partial documents. When they do that, they violate Full Recognition Before Processing and likely there's a latent security bug as a result.
nanis 5y ago
I wish people would stop trying to write programs for which there are no interpreters, compilers, or linters: name: Install dependencies run: > python -m pip install --upgrade pip > pip install pytest > if [ -f 'requirements.txt' ]; then pip install -r requirements.txt; fi That is a program that is hiding in the bowels of a "nestedtext" document ... It is no better than a program that is hiding in the bowels a JSON or YAML document. We all have to deal with this, but it is beyond stupid. [Install Dependencies] run=/path/to/install-script Then, write `install-script` in whatever language you want ... verify it works. It should have tests. etc etc etc.
- theamk 5y ago
  I don't think it matters much if this is inline or in separate file. If you want to test your tests, "yq -r .run input.yaml | sh -e" works as well. In fact, if I really wanted to test my tests, I'd say that directly testing the corresponding clause is the more comprehensive approach. For example, what if someone accidentally changes the line to read: run=/path/to/install-scriptq ? then your test of "install-script" will not catch anything. But if your test runs "yq -r .run | sh -e", then it will catch that error. And you can still forward to a script if you wanted to. So let's keep inline scripts, they are very reasonable methods for just a few commands.
  - galkk 5y ago
    Depending on the source control tool you may lose syntax highlightning, you, most likely, lose linters and even copying those multi-line commands to shell becomes cumbersome. I consider inlining example from GP's comment awful
  - kitd 5y ago
    V minor point: new yq versions use 'e' instead of '-r'.

ljm 5y ago

It would be nice if YAML wasn't horrendously abused the way it is. You have CI pipelines that let you construct DAGs to represent your builds, but you need several thousand lines of YAML and a load of custom parsing to get programming constructs in the string types, for example. And then each provider has its own way of providing those. I don't have to re-read manuals describing how to do if/else in Ruby or Java or Lisp, but as soon as yaml and some 'devops' tooling is involved, I have to constantly jump back and forth between the reference and my config. The main point being that the problem isn't the file format but the products that continue to push it, presumably because hacking stuff on top of `YAML.parse` is less effort than designing something that fits the purpose.

Lhiw 5y ago
I'm pretty sure the format is the issue. I still don't know how to do arrays in yaml. Is it new line, tab to same indent or same indent + one space, do I need a dash? Does a dash make it an array or an object? It's just simply not that big of a deal to add a few quotes and braces to make everything make sense. The only real issue with Json is the lack of comments and strictness about extra commas.
donmcronald 5y ago
Yeah. A lot of times I find myself thinking YAML is like a really awful programming language. You can sort of do conditional logic and loops, but usually I find it hard to follow what's going on. For build systems, I always liked the idea of Gradle where the core functionality was simple and declarative, but with the option to use a real programming language for things that weren't simple. For example, integrating installers or form builders (pre-processing) into a build are things I would consider non-trivial if there aren't official plugins, but it was still relatively easy to do with Gradle. The biggest problem I always had with Gradle was that I didn't like Groovy and I always though there was a missed opportunity to have a statically typed build system with a solid API/contract and all the fancy tooling like auto-complete that you get with statically typed languages. I see JSON5 mentioned a lot in the comments. In terms of CI / build systems, I feel like something built with JSON5/TypeScript could be really good. I'd be really happy using TypeScript for configuring things like build systems where there shouldn't really be an argument for needing it to be usable by non-programmers. Personally I feel like I've spent way to much of my life debugging YAML syntax issues.
- TheCycoONE 5y ago
  Does the Kotlin support in Gradle solve your problem? https://docs.gradle.org/current/userguide/kotlin_dsl.html https://docs.gradle.org/current/userguide/kotlin_dsl.html
  - spuz 5y ago
    My experience with Kotlin gradle scripts is worse than Groovy. For example, given the following valid groovy/kotlin gradle program: dependencies { } What would you expect to see between the curly braces? IntelliJ IDEA which supposedly has full support for the gradle DSL both for Groovy and Kotlin offers only generic suggestions. Common function calls such "implementation()" or "testImplementation()" are not suggested. If you do use those functions, no suggestion is made for their parameters. Because Gradle's DSL is built on top of a general purpose language, it loses the benefits of a DSL (constraining the set of possible configurations and guiding the user towards valid configurations).
theamk 5y ago
I really like approach of buildkite CI -- they use yaml, but this yaml can be produced by an executable script. So you write yaml by hand for trivial cases, but once it get complex, you can just drop back to shell/python/ruby/node/whatever, implement any complex logic, and serialize results to plain yaml.
int_19h 5y ago
What's really sad is that XML had a much better ecosystem around this for ages. I'd very much rather deal with XQuery or even XSLT to construct build trees, than the current crop of ad-hoc YAML preprocessors. At least the XML stuff had a consistent type system underneath!
- tdeck 5y ago
  XSLT is an absolute horror and not something I would want to deal with again. It feels like some weird academic experiment in an XML declarative programming language that should never have made it to print. If something needs the flexibility of a programming language, why not use a real one that's been well tested for writing other programs? These various config file programming systems always end up creating something notorious that everyone tries to avoid having to work on.
  - int_19h 5y ago
    XQuery is, in many ways, XSLT with better syntax. It doesn't have the pattern-matching transforms that are the T in XSLT - but for configs, I don't think it makes a big difference. Also, I don't think many realize that the stack has evolved since early 00s. XSLT 1.0 was a very limiting language, requiring extensions for many advanced scenarios. But there's XSLT v3.0 these days, and XPath & XQuery v3.1, with some major new features - e.g. maps and lambdas. Granted, this doesn't fix the most basic complaint about XSLT - its insanely verbose syntax - but even then, I'd still take XSLT over ad-hoc YAML-based loops and conditionals.
- dnautics 5y ago
  XML died because too many configurations turned what should be a 'prop' into an inner tag -- and it doesn't help that XML doesn't really give guidance as to when to use which. And, of course, when you deserialize XML, the innerText is always in a very strange place as to not really being clear "what the right way to handle it" is.
- stjohnswarts 5y ago
  I will take the verbosity of XML any day over YAML wrestling (complex YAML configs of course). There is simply too many "implicit rules" for YAML. it's why I prefer python over ruby and perl. Generally though TOML has been good enough for me to do lots of fairly large config files that are easy for humans and machines to parse.
thayne 5y ago
Honestly, I think using an embedded scripting language, like lua or even javascript, would be a much better fit for these use cases than trying to make yaml do something it wasn't designed for.
- ljm 5y ago
  Ironically, having used cdk8s[1] for dealing with kubernetes infrastructure, that's the one thing where I've actually preferred yaml. That said, k8s resource definitions are pure config so there's no need to try and hack extra bits on top of a serialized data structure. [1]https://cdk8s.io/ https://cdk8s.io/
ilyash 5y ago
It's always the same trajectory with declarative programming. It starts with "it's just configuration, we need something simple". Then users come with use cases which are more complex. Then you have programming language on top of configuration language syntax. * https://ilya-sher.org/2018/06/30/terraform-becomes-a-programming-language/ https://ilya-sher.org/2018/06/30/terraform-becomes-a-program... * https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a-programming-language/ https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a...
- silon42 5y ago
  Some even start with a programming language and pretend it's declarative... IMO, it's only declarative when it's a data model which is easily parsed by multiple languages/systems where it's needed.
- stevekemp 5y ago
  Very much so. A good few years ago I got annoyed that I couldn't have change mutt configuration the way that I wanted, because it has a built in configuration language which doesn't allow complicated conditionals etc. (There are workarounds, and off-hand I can't think of a great example, but bear with me.) In the end I wrote a simple console-based mail-client, which used a Lua configuration file. That would generally default to using hashes, and key=value settings, but over the time I used it things got really quite configurable via user-defined callbacks, and functions to return various settings. For example I wrote a hook called `on_reply_to`, and if you defined that function in your configuration file it would be invoked when you triggered the Reply function. This kind of flexibility was very self-consistent, and easy to add using an embedded real language. Later I added some hacks to a local fork of GNU Screen, there I just said: * If the ~/.screenrc file is executable, then execute it, and parse the output. That let me say "If hostname == foo; do this ; otherwise do this .." and get conditionals and some other things easily. Another example was unbinding all keys, and then only allowing some actions to be bound. (I later submitted "unbindall" upstream, to remove the need for that.)
shusaku 5y ago
This is the crux of the issue right? YAML is fine for most projects, but THE project for using YAML is CI configuration, for which it doesn’t match.

jrochkind1 5y ago

> So, I downloaded their python client to see how it did it. Who are you suggesting the python client belongs to, who is 'they' in 'their'?

detaro 5y ago
... the people that made it and wrote the submission we are discussing? how is this even a question?
- deleted 5y ago
  [deleted]

tialaramex 5y ago

> I don't like that a truncated document is a complete and valid document. Me either. If your documents have this property you're likely to tempt people to start trying to process partial documents. When they do that, they violate Full Recognition Before Processing and likely there's a latent security bug as a result.

nanis 5y ago

I wish people would stop trying to write programs for which there are no interpreters, compilers, or linters: name: Install dependencies run: > python -m pip install --upgrade pip > pip install pytest > if [ -f 'requirements.txt' ]; then pip install -r requirements.txt; fi That is a program that is hiding in the bowels of a "nestedtext" document ... It is no better than a program that is hiding in the bowels a JSON or YAML document. We all have to deal with this, but it is beyond stupid. [Install Dependencies] run=/path/to/install-script Then, write `install-script` in whatever language you want ... verify it works. It should have tests. etc etc etc.

theamk 5y ago
I don't think it matters much if this is inline or in separate file. If you want to test your tests, "yq -r .run input.yaml | sh -e" works as well. In fact, if I really wanted to test my tests, I'd say that directly testing the corresponding clause is the more comprehensive approach. For example, what if someone accidentally changes the line to read: run=/path/to/install-scriptq ? then your test of "install-script" will not catch anything. But if your test runs "yq -r .run | sh -e", then it will catch that error. And you can still forward to a script if you wanted to. So let's keep inline scripts, they are very reasonable methods for just a few commands.
- galkk 5y ago
  Depending on the source control tool you may lose syntax highlightning, you, most likely, lose linters and even copying those multi-line commands to shell becomes cumbersome. I consider inlining example from GP's comment awful
- kitd 5y ago
  V minor point: new yq versions use 'e' instead of '-r'.