5 ms·

> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never ag

> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.

hbogert 2y ago
Man I'd love some real war story sharing here! This might save me some time. B/c I have to admit i'm still in my honeymoon period with CUE
zbentley 2y ago
I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick another one). Like, people are correct that it is not great to allow (restricted at runtime or not) arbitrary code to define configs. I think "our config language is Python"-type statements are concerning in a whole lot of contexts, and for good reasons. But holy shit at what cost. The landscape formed by the extremity of that no-real-scripting-languages-allowed sentiment is bleak. We've all dealt with it: the proliferation of handfuls of arcane tools that have to be present at specific versions in order to render a string. The triple-templated hellscapes written in Helm/Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell. Indentation hell. Brace hell. The endless, endless bikesheds over which data language to render to (which are never really resolved and always just end up with JSON-interpolated-in-YAML-interpolated-in-TOML and transformed to the "right" format at render time). The flavor-of-the-month XKCD#927 creation of each iteration of a data+templating language that will solve all of the problems, but for real this time. This really sucks. Not because its tedious; tedium is not necessarily a bad thing, with apologies to the automate-all-the-things crowd. But because it's expensive: in time, training, bugs, incident MTTR, and so on. I dunno. Maybe we should accept that past some complexity threshold, you should just write your configs in a general purpose scripting language. Not because it's a good practice, but because our best attempts at an alternative are so much worse that it's probably the best practice to just do it in fucking Python. Starting from there and adding restrictions (i.e. Starlark) can't possibly be worse than the currently blessed way.
- rockwotj 2y ago
  Agreed - I actually think TypeScript is easier to craft JSON output in tho: https://news.ycombinator.com/item?id=39242314 https://news.ycombinator.com/item?id=39242314 The Starlark approach is good too - I just like types if it's going to get narly and find Python typing to be suboptimal compared to TypeScript in terms of LSP and tooling (and only Facebook's Rust starlark interpreter supports types)
- andrewaylett 2y ago
  I don't object to the desired state being represented in YAML, I object to generating that YAML using something that's not a "real" programming language. I don't particularly care if you generate your data structures from schemas, or generate your schemas from data structures, but I do care if you don't generate your output YAML using anything that's not type-safe. If you're generating YAML to deploy, you can do that during build. If you're needing to combine configuration to generate new YAML at runtime, that's what controllers are for.
  - hosh 2y ago
    I generally agree. I use Ruby for this. Duck typing is sufficient. I’ve also found that coupling manifests generation with application deployment to close a lot of doors. It’s better to modify manifests to change image references (or something similar), and have a separate process for applying manifest changes.
    - FridgeSeal 2y ago
      I’ve had great success with Flux and a small bit of CI for this. We currently have configs in raw K8s yaml in our monorepo in kustomize base+overlay style setup. It’s super duper obvious what’s in each env, and straightforward to change them. Images are swapped in using Kustomize’s “newImage” functionality. Flux watches git and deploys to K8s. At any point it’s super clear what the configs are, what image we were using at the time, flux just pulls in what it finds in git, so there complete separation between “make them” and “deploy them”. Nicest CICD experience ever. Zero helm tempting, zero pain.

hbogert 2y ago

Man I'd love some real war story sharing here! This might save me some time. B/c I have to admit i'm still in my honeymoon period with CUE

zbentley 2y ago

I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick another one). Like, people are correct that it is not great to allow (restricted at runtime or not) arbitrary code to define configs. I think "our config language is Python"-type statements are concerning in a whole lot of contexts, and for good reasons. But holy shit at what cost. The landscape formed by the extremity of that no-real-scripting-languages-allowed sentiment is bleak. We've all dealt with it: the proliferation of handfuls of arcane tools that have to be present at specific versions in order to render a string. The triple-templated hellscapes written in Helm/Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell. Indentation hell. Brace hell. The endless, endless bikesheds over which data language to render to (which are never really resolved and always just end up with JSON-interpolated-in-YAML-interpolated-in-TOML and transformed to the "right" format at render time). The flavor-of-the-month XKCD#927 creation of each iteration of a data+templating language that will solve all of the problems, but for real this time. This really sucks. Not because its tedious; tedium is not necessarily a bad thing, with apologies to the automate-all-the-things crowd. But because it's expensive: in time, training, bugs, incident MTTR, and so on. I dunno. Maybe we should accept that past some complexity threshold, you should just write your configs in a general purpose scripting language. Not because it's a good practice, but because our best attempts at an alternative are so much worse that it's probably the best practice to just do it in fucking Python. Starting from there and adding restrictions (i.e. Starlark) can't possibly be worse than the currently blessed way.

rockwotj 2y ago
Agreed - I actually think TypeScript is easier to craft JSON output in tho: https://news.ycombinator.com/item?id=39242314 https://news.ycombinator.com/item?id=39242314 The Starlark approach is good too - I just like types if it's going to get narly and find Python typing to be suboptimal compared to TypeScript in terms of LSP and tooling (and only Facebook's Rust starlark interpreter supports types)
andrewaylett 2y ago
I don't object to the desired state being represented in YAML, I object to generating that YAML using something that's not a "real" programming language. I don't particularly care if you generate your data structures from schemas, or generate your schemas from data structures, but I do care if you don't generate your output YAML using anything that's not type-safe. If you're generating YAML to deploy, you can do that during build. If you're needing to combine configuration to generate new YAML at runtime, that's what controllers are for.
- hosh 2y ago
  I generally agree. I use Ruby for this. Duck typing is sufficient. I’ve also found that coupling manifests generation with application deployment to close a lot of doors. It’s better to modify manifests to change image references (or something similar), and have a separate process for applying manifest changes.
  - FridgeSeal 2y ago
    I’ve had great success with Flux and a small bit of CI for this. We currently have configs in raw K8s yaml in our monorepo in kustomize base+overlay style setup. It’s super duper obvious what’s in each env, and straightforward to change them. Images are swapped in using Kustomize’s “newImage” functionality. Flux watches git and deploys to K8s. At any point it’s super clear what the configs are, what image we were using at the time, flux just pulls in what it finds in git, so there complete separation between “make them” and “deploy them”. Nicest CICD experience ever. Zero helm tempting, zero pain.