8 ms·

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just m

His take on text interpolation is very right. I'm a SWE turned SRE because as a developer I really enjoyed using K8s. But as a full-time SRE where I work just means YAML juggling. It's mind numbing that everybody is okay with this, this really is our domain's assembly era, albeit with whitespace, colons, dashes and brackets. I've found solace in CUE which I just run locally to catch all the small errors everybody makes on a daily basis. Putting the CUE validation in our pipeline is too confronting for others, yet they're constantly making up best practices adhoc during reviews which could've easily been codified with CUE (or some other serious config language).

jauntywundrkind 2y ago
There's a ton of discussion of alternative ideas as comments. Many look amazing. But I keep wondering how many would be useful in giving us the modularity to make something like the shareable charts Helm has. That bitnami and others have these massive troves of charts is such a superpower for Kubernetes. We all seem to agree it's not the string templating that's excellent, but figuring out how to package & make modular the templates/generators, and how to manage the outputs of the generators: that's "the rest of the owl" that tools such as Kustomzie and languages such as Dhall and starlark don't really buy us. There is MetaController, which seems like some way to operationalize running generators and managing outputs. But I'm not sure what good examples there are of its use and specifically what examples are intended for distribution alike Helm Charts sometimes are.
galdosdi 2y ago
> Putting the CUE validation in our pipeline is too confronting for others Sad. Can you get away with boiling the frog by getting it initially added but configured to check very, very little? Maybe some specific rare category of mistake that doesn't happen enough to annoy your colleagues when it catches it but is implicated in a recent outage management still remembers? Then slowly adding rules over time as feasible (exploiting each outage as an opportunity to enlist stakeholder support for adding more rules that would have caught that particular misconfiguration event) Sometimes I think figuring out how to stage and phase gradually making the changes you need to improve the system in light of social inertia against it is the most complex part of corporate enterprise software work. I definitely remember a time I wrote a whole perl-based wheel-reinventing crappy puppet knockoff just for a set of nagios plugins, entirely not for technical reasons, but for the political reason that this way we would control it instead of the other department which refused to get with the program and let us do staged canary rollouts. It was wrong technically if you assume frictionless corporate politics, but it was the right and only practical way to achieve the goal of ending the drumbeat of customer outages.
yourapostasy 2y ago
> ...just means YAML juggling. Perhaps I'm missing something, but even the "YAML juggling" still seems to be in an immature stage. As an example, a problem I've been wrestling with recently where I could not find a solution with yq or other YAML wrangling tooling is the following. Say I have a YAML fragment like this: rules: - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - get - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - list I want to condense it to this: rules: - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - get - list More generally, I want to find the smallest most compact representation possible between the apiGroups, resources and verbs terms, combining everywhere possible. In the example, I combined the verbs, but the same combining can take place for the apiGroups and resources terms. This comes in really handy when cleaning up a litter trail of these constructs built from figuring out what resources to add to enable more complex Helm Chart installs (the iterative nature of that exercise is another rant in itself) in reasonably restrictive environments. I resorted to writing a script to solve just this problem, but I keep thinking I must have missed some solution to this everyone else seems to have solved but me. Apparently the solution eludes my local K8S experts as well, because they condense it by hand, which I refuse to accept as the proper solution.
doctorpangloss 2y ago
> which I just run locally to catch all the small errors everybody makes on a daily basis What small errors?
hosh 2y ago
The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby. The problem is that it is written in Ruby. It's great for shops that already have Ruby expertise. Wherever I have implemented this, it's worked out great. I have my doubts about it in other shops. The community went a different way -- starting with templated JSON (not even JSON transforms). It was the least common denominator. This is an issue with the ecosystem rather than design flaws of Kubernetes itself.
- lmm 2y ago
  Sort of. Kubernetes is lowest common denominator by design. Specific ecosystems had much more efficient solutions to these problems (e.g. rather than having to deploy a whole docker container to do a version upgrade you deploy just your specific web application) but the problem was those solutions only worked for those specific ecosystems.
  - hosh 2y ago
    I disagree. Kubernetes is composed from primitives that can be applied in versatile ways. Those primitives can be swapped out for even greater extensibility. Kubernetes is not the lowest common denominator, and more of a toolkit that can broadly and deeply applied to many setups. Templated yaml or json is a step backwards. Those are not composable. If we wanted something like that, then we would need to at least be able to do merges and transforms. Something like jquery or css for yaml that allows to transform fields based upon selectors. Manifest generators using a real language, on the other hand, start with data structures that can be merged and transformed before its final output as yaml. They can take advantage of language features. The one I wrote in Ruby can take advantage of class inheritance and mixins, allowing me to define common configs across resources. Someone who can distill the algebra of merging Kubernetes manifests can do that for a functional programming language.
- jamesfinlayson 2y ago
  > The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby. I've seen something similar at a previous company - some Ruby DSL written to generate CloudFormation. It was used in another department though so not sure how well it worked in practice.
candiddevmike 2y ago
Ksonnet being deprecated was one of the worst things to happen with the ecosystem IMO. Some kind of Jsonnet tool should have been integrated with kubectl like kustomize. Shameless plug for Etcha, a configuration management tool (that works with kube!) built around Jsonnet manifests: https://etcha.dev https://etcha.dev
deleted 2y ago
[deleted]
FridgeSeal 2y ago
> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.
- hbogert 2y ago
  Man I'd love some real war story sharing here! This might save me some time. B/c I have to admit i'm still in my honeymoon period with CUE
- zbentley 2y ago
  I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick another one). Like, people are correct that it is not great to allow (restricted at runtime or not) arbitrary code to define configs. I think "our config language is Python"-type statements are concerning in a whole lot of contexts, and for good reasons. But holy shit at what cost. The landscape formed by the extremity of that no-real-scripting-languages-allowed sentiment is bleak. We've all dealt with it: the proliferation of handfuls of arcane tools that have to be present at specific versions in order to render a string. The triple-templated hellscapes written in Helm/Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell. Indentation hell. Brace hell. The endless, endless bikesheds over which data language to render to (which are never really resolved and always just end up with JSON-interpolated-in-YAML-interpolated-in-TOML and transformed to the "right" format at render time). The flavor-of-the-month XKCD#927 creation of each iteration of a data+templating language that will solve all of the problems, but for real this time. This really sucks. Not because its tedious; tedium is not necessarily a bad thing, with apologies to the automate-all-the-things crowd. But because it's expensive: in time, training, bugs, incident MTTR, and so on. I dunno. Maybe we should accept that past some complexity threshold, you should just write your configs in a general purpose scripting language. Not because it's a good practice, but because our best attempts at an alternative are so much worse that it's probably the best practice to just do it in fucking Python. Starting from there and adding restrictions (i.e. Starlark) can't possibly be worse than the currently blessed way.
andix 2y ago
I don't think that the k8s yamls/jsons are bad. It's just bad that we write and edit them as text files My take on the k8s yaml mess is, that we are lacking a modern replacement for Helm. The yaml templating on text file level is just pure crazyness. I think we would need an approach that is more like React/JSX or even better TSX (typescript JSX). Some possibility to get away from simple templating and more into structured descriptions, with typing and control flows. I think there are some approaches into this direction (like the k8s Terraform module, or the defunct ksonnet), but none of them got it perfectly right yet.
- ruuda 2y ago
  RCL has control flow and types: https://rcl-lang.org/ https://rcl-lang.org/. No record types yet though, so they are not yet very useful to validate e.g. a Kubernetes manifest against a schema.
  - andix 2y ago
    Looks a lot like Jsonnet. This is going into the direction I'm talking about. Is there also some typing-checking/code-completion available for Kubernetes resources? I think this would be an essential part to improve developer experience and automatic checks. Some linter to validate the k8s manifests must be a part of a solution. It's possible to check the output, but a perfect solution would also lint the RCL source code and highlight the sections that generate invalid manifests.
    - ruuda 2y ago
      At the moment nothing like that exists. Eventually it should be possible to generate RCL types like https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes does for Dhall.
  - andix 2y ago
    I was just comparing different approaches and stumbled upon KCL. Might be worth a look: https://www.kcl-lang.io/docs/user_docs/getting-started/intro https://www.kcl-lang.io/docs/user_docs/getting-started/intro
reedjosh 2y ago
KCL is amazing and growing quickly. Can use Crossplane function KCL too.
yen223 2y ago
Something curious I noticed: while this comment was posted "6 hours ago", when I click reply, it instead says that the comment was posted "3 days ago". Is this happening to others as well? From digging around it looks like the comment really was posted 3 days ago, looking at the `title="2024-07-29T08:41:07"` attribute in the timestamp tag.
- Jtsummers 2y ago
  Submissions get renewed sometimes and the time stamps are set to a newer time (this is to get it ranked high enough to hit the front page). All comments have a false (newer) time stamp for a while if they predate the renewed time stamp. After a few hours the original time is restored for the submission and comments. These usually come from the second chance pool (click lists at the bottom to get to the pool and some other lists).
carlsborg 2y ago
Did you look at CDK8s?

jauntywundrkind 2y ago

There's a ton of discussion of alternative ideas as comments. Many look amazing. But I keep wondering how many would be useful in giving us the modularity to make something like the shareable charts Helm has. That bitnami and others have these massive troves of charts is such a superpower for Kubernetes. We all seem to agree it's not the string templating that's excellent, but figuring out how to package & make modular the templates/generators, and how to manage the outputs of the generators: that's "the rest of the owl" that tools such as Kustomzie and languages such as Dhall and starlark don't really buy us. There is MetaController, which seems like some way to operationalize running generators and managing outputs. But I'm not sure what good examples there are of its use and specifically what examples are intended for distribution alike Helm Charts sometimes are.

galdosdi 2y ago

> Putting the CUE validation in our pipeline is too confronting for others Sad. Can you get away with boiling the frog by getting it initially added but configured to check very, very little? Maybe some specific rare category of mistake that doesn't happen enough to annoy your colleagues when it catches it but is implicated in a recent outage management still remembers? Then slowly adding rules over time as feasible (exploiting each outage as an opportunity to enlist stakeholder support for adding more rules that would have caught that particular misconfiguration event) Sometimes I think figuring out how to stage and phase gradually making the changes you need to improve the system in light of social inertia against it is the most complex part of corporate enterprise software work. I definitely remember a time I wrote a whole perl-based wheel-reinventing crappy puppet knockoff just for a set of nagios plugins, entirely not for technical reasons, but for the political reason that this way we would control it instead of the other department which refused to get with the program and let us do staged canary rollouts. It was wrong technically if you assume frictionless corporate politics, but it was the right and only practical way to achieve the goal of ending the drumbeat of customer outages.

yourapostasy 2y ago

> ...just means YAML juggling. Perhaps I'm missing something, but even the "YAML juggling" still seems to be in an immature stage. As an example, a problem I've been wrestling with recently where I could not find a solution with yq or other YAML wrangling tooling is the following. Say I have a YAML fragment like this: rules: - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - get - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - list I want to condense it to this: rules: - apiGroups: - rbac.authorization.k8s.io resources: -clusterroles verbs: - get - list More generally, I want to find the smallest most compact representation possible between the apiGroups, resources and verbs terms, combining everywhere possible. In the example, I combined the verbs, but the same combining can take place for the apiGroups and resources terms. This comes in really handy when cleaning up a litter trail of these constructs built from figuring out what resources to add to enable more complex Helm Chart installs (the iterative nature of that exercise is another rant in itself) in reasonably restrictive environments. I resorted to writing a script to solve just this problem, but I keep thinking I must have missed some solution to this everyone else seems to have solved but me. Apparently the solution eludes my local K8S experts as well, because they condense it by hand, which I refuse to accept as the proper solution.

doctorpangloss 2y ago

> which I just run locally to catch all the small errors everybody makes on a daily basis What small errors?

hosh 2y ago

The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby. The problem is that it is written in Ruby. It's great for shops that already have Ruby expertise. Wherever I have implemented this, it's worked out great. I have my doubts about it in other shops. The community went a different way -- starting with templated JSON (not even JSON transforms). It was the least common denominator. This is an issue with the ecosystem rather than design flaws of Kubernetes itself.

lmm 2y ago
Sort of. Kubernetes is lowest common denominator by design. Specific ecosystems had much more efficient solutions to these problems (e.g. rather than having to deploy a whole docker container to do a version upgrade you deploy just your specific web application) but the problem was those solutions only worked for those specific ecosystems.
- hosh 2y ago
  I disagree. Kubernetes is composed from primitives that can be applied in versatile ways. Those primitives can be swapped out for even greater extensibility. Kubernetes is not the lowest common denominator, and more of a toolkit that can broadly and deeply applied to many setups. Templated yaml or json is a step backwards. Those are not composable. If we wanted something like that, then we would need to at least be able to do merges and transforms. Something like jquery or css for yaml that allows to transform fields based upon selectors. Manifest generators using a real language, on the other hand, start with data structures that can be merged and transformed before its final output as yaml. They can take advantage of language features. The one I wrote in Ruby can take advantage of class inheritance and mixins, allowing me to define common configs across resources. Someone who can distill the algebra of merging Kubernetes manifests can do that for a functional programming language.
jamesfinlayson 2y ago
> The better way is to use manifest generators with a Turing-complete language. I wrote such a tool in Ruby. I've seen something similar at a previous company - some Ruby DSL written to generate CloudFormation. It was used in another department though so not sure how well it worked in practice.

candiddevmike 2y ago

Ksonnet being deprecated was one of the worst things to happen with the ecosystem IMO. Some kind of Jsonnet tool should have been integrated with kubectl like kustomize. Shameless plug for Etcha, a configuration management tool (that works with kube!) built around Jsonnet manifests: https://etcha.dev https://etcha.dev

deleted 2y ago

[deleted]

FridgeSeal 2y ago

> I've found solace in CUE We rebuilt our Kustomization layers to be built via cue, so that we could theoretically work with less mistakes. Oh my god, never again. Incredibly confusing and horrible errors. Writing new modules was painful and unclear, trying to compose them together was even worse. I still get flashbacks to that “Invalid bytes” error.

hbogert 2y ago
Man I'd love some real war story sharing here! This might save me some time. B/c I have to admit i'm still in my honeymoon period with CUE
zbentley 2y ago
I think it's worth re-examining the widespread opposition to writing configs in an interpreted "real" programming language, rather than a string templating language (if you're looking to split hairs, in this taxonomy PHP and JSX are real languages, but they're close to the line. Python is prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH string expansion are not. If the word "real" bothers you, pick another one). Like, people are correct that it is not great to allow (restricted at runtime or not) arbitrary code to define configs. I think "our config language is Python"-type statements are concerning in a whole lot of contexts, and for good reasons. But holy shit at what cost. The landscape formed by the extremity of that no-real-scripting-languages-allowed sentiment is bleak. We've all dealt with it: the proliferation of handfuls of arcane tools that have to be present at specific versions in order to render a string. The triple-templated hellscapes written in Helm/Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell. Indentation hell. Brace hell. The endless, endless bikesheds over which data language to render to (which are never really resolved and always just end up with JSON-interpolated-in-YAML-interpolated-in-TOML and transformed to the "right" format at render time). The flavor-of-the-month XKCD#927 creation of each iteration of a data+templating language that will solve all of the problems, but for real this time. This really sucks. Not because its tedious; tedium is not necessarily a bad thing, with apologies to the automate-all-the-things crowd. But because it's expensive: in time, training, bugs, incident MTTR, and so on. I dunno. Maybe we should accept that past some complexity threshold, you should just write your configs in a general purpose scripting language. Not because it's a good practice, but because our best attempts at an alternative are so much worse that it's probably the best practice to just do it in fucking Python. Starting from there and adding restrictions (i.e. Starlark) can't possibly be worse than the currently blessed way.

andix 2y ago

I don't think that the k8s yamls/jsons are bad. It's just bad that we write and edit them as text files My take on the k8s yaml mess is, that we are lacking a modern replacement for Helm. The yaml templating on text file level is just pure crazyness. I think we would need an approach that is more like React/JSX or even better TSX (typescript JSX). Some possibility to get away from simple templating and more into structured descriptions, with typing and control flows. I think there are some approaches into this direction (like the k8s Terraform module, or the defunct ksonnet), but none of them got it perfectly right yet.

ruuda 2y ago
RCL has control flow and types: https://rcl-lang.org/ https://rcl-lang.org/. No record types yet though, so they are not yet very useful to validate e.g. a Kubernetes manifest against a schema.
- andix 2y ago
  Looks a lot like Jsonnet. This is going into the direction I'm talking about. Is there also some typing-checking/code-completion available for Kubernetes resources? I think this would be an essential part to improve developer experience and automatic checks. Some linter to validate the k8s manifests must be a part of a solution. It's possible to check the output, but a perfect solution would also lint the RCL source code and highlight the sections that generate invalid manifests.
  - ruuda 2y ago
    At the moment nothing like that exists. Eventually it should be possible to generate RCL types like https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes does for Dhall.
- andix 2y ago
  I was just comparing different approaches and stumbled upon KCL. Might be worth a look: https://www.kcl-lang.io/docs/user_docs/getting-started/intro https://www.kcl-lang.io/docs/user_docs/getting-started/intro

reedjosh 2y ago

KCL is amazing and growing quickly. Can use Crossplane function KCL too.

yen223 2y ago

Something curious I noticed: while this comment was posted "6 hours ago", when I click reply, it instead says that the comment was posted "3 days ago". Is this happening to others as well? From digging around it looks like the comment really was posted 3 days ago, looking at the `title="2024-07-29T08:41:07"` attribute in the timestamp tag.

Jtsummers 2y ago
Submissions get renewed sometimes and the time stamps are set to a newer time (this is to get it ranked high enough to hit the front page). All comments have a false (newer) time stamp for a while if they predate the renewed time stamp. After a few hours the original time is restored for the submission and comments. These usually come from the second chance pool (click lists at the bottom to get to the pool and some other lists).

carlsborg 2y ago

Did you look at CDK8s?