6 ms·

If this article is any indication of the current state of the industry then the devops movement has busted. We're back to dev and ops.

TechBro8615 4y ago
The problem is YAML, but it's too late to stop that momentum now. Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.
- tristan957 4y ago
  I would be more interested in a stripped down JavaScript runtime for configuration. Some tooling already supports this paradigm. Embed something like QuickJS or a Lua interpreter and have the script export a config object. It feels too much like we want these plaintext config files to turn into fully-fledged scripts.
  - andrewstuart2 4y ago
    It's not stripped down, but Pulumi is a version of this. Regular code, and all the benefits and drawbacks of that approach, in your language(s) of choice to create infrastructure.
    - tristan957 4y ago
      I haven't used Pulumi, but I have used Terraform (years ago). That is pretty much exactly what I am talking about, but instead of creating a new language, just use an already established language meant for scripting.
      - mieubrisse 4y ago
        We're trying to do something very similar to what the OP proposes at our current company (Kurtosis), and we explored YAML, Jsonnet, Dhall, CUE, and general-purpose languages and eventually settled on Starlark: https://docs.kurtosis.com/explanations/starlark#why-did-kurtosis-choose-starlark-for-its-environment-definitions https://docs.kurtosis.com/explanations/starlark#why-did-kurt... BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
      - elcritch 4y ago
        I've started using this setup for my Nim projects. I created a project that lets me use Nim instead of YAML for a statically checked configuration system. It can be run and output json, yaml, or run as is in Nim code. Its been much nicer than pure YAML! https://github.com/elcritch/ants/tree/main https://github.com/elcritch/ants/tree/main Not satisfied I used a simple source filter to get a YAML compatible variant (WIP): https://github.com/elcritch/ants/blob/main/tests/cimport_example.yaml https://github.com/elcritch/ants/blob/main/tests/cimport_exa... Our concept of configs languages needs to keep evolving.
  - brodouevencode 4y ago
    Like a CDK (cloud development kit)? Those tools already exist - like AWS CDK, Pulumi, CDK-TF - but they are hardly stripped down. AWS CDK for instance transpiles everything down to Typescript and those layers of abstractions make debugging painful.
    - tristan957 4y ago
      Yes kind of, but instead of creating a new language (like HCL), just use a language people already know, but strip the runtime to a minimal set of features.
  - TechBro8615 4y ago
    With Lua, you don't even need to export a config object, the Lua script is the config object. I do think there is some promise to this - see Neovim for example. There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
  - baq 4y ago
    Lua has been created as a config language. I love to see the world coming full circle.
    - tristan957 4y ago
      Born in 1996, so I have no clue on the history of Lua, but good to know I am following in the footsteps of gods! :)
- zamalek 4y ago
  I share your, and all Norwegians', distaste for YAML. One saving grace I have found is that YAML is a superset of JSON. JSON isn't great, but it's certainly better than YAML. I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling. Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
  - TechBro8615 4y ago
    Upon further reflection, it's not actually YAML I have a problem with, so much as the (lack of) composability of YAML. As soon as you're dealing with multiple files and want to "include" something in one way or another, it seems like every DevOps software has its own semi-standard way of allowing that. As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them. I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
    - zamalek 4y ago
      My main issue is that YAML is used as a [bad] programming language. To make matters worse, in the domains where it's typically used (DevOps) the "developer inner loop" is usually minutes long (e.g. GitHub actions). Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
  - brodouevencode 4y ago
    > JSON isn't great, but it's certainly better than YAML. This is the first time I've ever seen someone say this. Why do you think it's better?
    - TechBro8615 4y ago
- edganiukov 4y ago
  We do - https://foundation.namespace.so/docs https://foundation.namespace.so/docs. Some feature of "Namespace" are kind of similar to Ergomake, but we use own format with CUE instead of docker-compose. Here is comparison to docker-compose - https://namespace.so/blog/from-docker-compose-to-namespace https://namespace.so/blog/from-docker-compose-to-namespace.
- yourapostasy 4y ago
  CUE [1] is kinda-sorta "transpiled" into by Pulumi's language support facility [2]. In general the greater the domain space being expressed the more I'm leery of this kind of approach; leaky abstractions creep in faster the larger the volume of concepts embraced. Infrastructure is a pretty big concepts volume. Whatever the source system (Pulumi in this case) does not provide a hook for that the target system (CUE in this case) supports for example, hacks tend to start to emerge. If there was both this layer of abstraction and a "break glass" layer available down to something analogous to an Abstract Syntax Tree level of primitives, then the target systems at least have hooks to compose new abstractions via some kind of DSL into the source system perhaps. There be dragons there too, though; no panaceas, but at least it is better than all sorts of hacks. I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE. Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale. One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code. [1] https://cuelang.org/docs/about/ https://cuelang.org/docs/about/ [2] https://www.pulumi.com/blog/extending-pulumi-languages-with-yaml-cue-jsonnet-rust/ https://www.pulumi.com/blog/extending-pulumi-languages-with-... [3] https://dhall-lang.org/ https://dhall-lang.org/ [4] https://news.ycombinator.com/item?id=32102203 https://news.ycombinator.com/item?id=32102203
poolopolopolo 4y ago
If anything, situation is much worse than when we used to have "System/Network administrators", the cloud was a lie.
- brodouevencode 4y ago
  Don't know that I'd go so far as to blame the cloud (maybe the marketing materials but not the actual cloud) but I do share the sentiment.
- bradwood 4y ago
  Yeah. Everything then was a "layer 9" problem.
bradwood 4y ago
100%. It's not hard to learn a bit of k8s yaml if you're an appdev FFS.
mieubrisse 4y ago
Little bit of self-promotion here, but seeing this comment is exactly what we felt at Kurtosis: https://docs.kurtosis.com/explanations/what-is-kurtosis https://docs.kurtosis.com/explanations/what-is-kurtosis We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.
- morelisp 4y ago
  "Progress" is that now also ops gets to throw half-functional operators, underdocumented CRDs, and pluggable CRI/CNI/CSI all not quite configured to work together properly back over the wall to devs who have no hope to debug anything anymore!

TechBro8615 4y ago

The problem is YAML, but it's too late to stop that momentum now. Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.

tristan957 4y ago
I would be more interested in a stripped down JavaScript runtime for configuration. Some tooling already supports this paradigm. Embed something like QuickJS or a Lua interpreter and have the script export a config object. It feels too much like we want these plaintext config files to turn into fully-fledged scripts.
- andrewstuart2 4y ago
  It's not stripped down, but Pulumi is a version of this. Regular code, and all the benefits and drawbacks of that approach, in your language(s) of choice to create infrastructure.
  - tristan957 4y ago
    I haven't used Pulumi, but I have used Terraform (years ago). That is pretty much exactly what I am talking about, but instead of creating a new language, just use an already established language meant for scripting.
    - mieubrisse 4y ago
      We're trying to do something very similar to what the OP proposes at our current company (Kurtosis), and we explored YAML, Jsonnet, Dhall, CUE, and general-purpose languages and eventually settled on Starlark: https://docs.kurtosis.com/explanations/starlark#why-did-kurtosis-choose-starlark-for-its-environment-definitions https://docs.kurtosis.com/explanations/starlark#why-did-kurt... BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
    - elcritch 4y ago
      I've started using this setup for my Nim projects. I created a project that lets me use Nim instead of YAML for a statically checked configuration system. It can be run and output json, yaml, or run as is in Nim code. Its been much nicer than pure YAML! https://github.com/elcritch/ants/tree/main https://github.com/elcritch/ants/tree/main Not satisfied I used a simple source filter to get a YAML compatible variant (WIP): https://github.com/elcritch/ants/blob/main/tests/cimport_example.yaml https://github.com/elcritch/ants/blob/main/tests/cimport_exa... Our concept of configs languages needs to keep evolving.
- brodouevencode 4y ago
  Like a CDK (cloud development kit)? Those tools already exist - like AWS CDK, Pulumi, CDK-TF - but they are hardly stripped down. AWS CDK for instance transpiles everything down to Typescript and those layers of abstractions make debugging painful.
  - tristan957 4y ago
    Yes kind of, but instead of creating a new language (like HCL), just use a language people already know, but strip the runtime to a minimal set of features.
- TechBro8615 4y ago
  With Lua, you don't even need to export a config object, the Lua script is the config object. I do think there is some promise to this - see Neovim for example. There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
- baq 4y ago
  Lua has been created as a config language. I love to see the world coming full circle.
  - tristan957 4y ago
    Born in 1996, so I have no clue on the history of Lua, but good to know I am following in the footsteps of gods! :)
zamalek 4y ago
I share your, and all Norwegians', distaste for YAML. One saving grace I have found is that YAML is a superset of JSON. JSON isn't great, but it's certainly better than YAML. I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling. Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
- TechBro8615 4y ago
  Upon further reflection, it's not actually YAML I have a problem with, so much as the (lack of) composability of YAML. As soon as you're dealing with multiple files and want to "include" something in one way or another, it seems like every DevOps software has its own semi-standard way of allowing that. As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them. I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
  - zamalek 4y ago
    My main issue is that YAML is used as a [bad] programming language. To make matters worse, in the domains where it's typically used (DevOps) the "developer inner loop" is usually minutes long (e.g. GitHub actions). Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
- brodouevencode 4y ago
  > JSON isn't great, but it's certainly better than YAML. This is the first time I've ever seen someone say this. Why do you think it's better?
  - TechBro8615 4y ago
edganiukov 4y ago
We do - https://foundation.namespace.so/docs https://foundation.namespace.so/docs. Some feature of "Namespace" are kind of similar to Ergomake, but we use own format with CUE instead of docker-compose. Here is comparison to docker-compose - https://namespace.so/blog/from-docker-compose-to-namespace https://namespace.so/blog/from-docker-compose-to-namespace.
yourapostasy 4y ago
CUE [1] is kinda-sorta "transpiled" into by Pulumi's language support facility [2]. In general the greater the domain space being expressed the more I'm leery of this kind of approach; leaky abstractions creep in faster the larger the volume of concepts embraced. Infrastructure is a pretty big concepts volume. Whatever the source system (Pulumi in this case) does not provide a hook for that the target system (CUE in this case) supports for example, hacks tend to start to emerge. If there was both this layer of abstraction and a "break glass" layer available down to something analogous to an Abstract Syntax Tree level of primitives, then the target systems at least have hooks to compose new abstractions via some kind of DSL into the source system perhaps. There be dragons there too, though; no panaceas, but at least it is better than all sorts of hacks. I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE. Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale. One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code. [1] https://cuelang.org/docs/about/ https://cuelang.org/docs/about/ [2] https://www.pulumi.com/blog/extending-pulumi-languages-with-yaml-cue-jsonnet-rust/ https://www.pulumi.com/blog/extending-pulumi-languages-with-... [3] https://dhall-lang.org/ https://dhall-lang.org/ [4] https://news.ycombinator.com/item?id=32102203 https://news.ycombinator.com/item?id=32102203

poolopolopolo 4y ago

If anything, situation is much worse than when we used to have "System/Network administrators", the cloud was a lie.

brodouevencode 4y ago
Don't know that I'd go so far as to blame the cloud (maybe the marketing materials but not the actual cloud) but I do share the sentiment.
bradwood 4y ago
Yeah. Everything then was a "layer 9" problem.

bradwood 4y ago

100%. It's not hard to learn a bit of k8s yaml if you're an appdev FFS.

mieubrisse 4y ago

Little bit of self-promotion here, but seeing this comment is exactly what we felt at Kurtosis: https://docs.kurtosis.com/explanations/what-is-kurtosis https://docs.kurtosis.com/explanations/what-is-kurtosis We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.

morelisp 4y ago
"Progress" is that now also ops gets to throw half-functional operators, underdocumented CRDs, and pluggable CRI/CNI/CSI all not quite configured to work together properly back over the wall to devs who have no hope to debug anything anymore!