6 ms·

I think this might be a step in the right direction, but my main problem with Kubernetes package management today might not be fixable by a package manager, sad

I think this might be a step in the right direction, but my main problem with Kubernetes package management today might not be fixable by a package manager, sadly. The biggest issue I have in my daily life is handling the multiple levels of nested YAML and the unpredictability of the results. Think of an ArgoCD ApplicationSet that generates a bunch of Applications. Those Applications render a bunch of Helm charts, and inside those charts there are CRDs used by some random operator like Strimzi, Grafana or Vector. Given YAML's lack of syntax and the absense of any sort of standard for rendering templates, it's practically impossible to know what are the actual YAML being injected in the Kubernetes API when you make a top-level change. It's trial and error, expensive blue-green deployments and hundreds of debugging minutes all the way, every month.

pmig 2y ago
This is actually a problem we want to focus on with Glasskube Cloud (https://glasskube.cloud/ https://glasskube.cloud/) where our glasskube[bot] will comment on your GitOps Pull request with an exact diff of resources that will get changed across all connected clusters. This diff will be performed by controller running inside your cluster. Think of it as codecov analysis, but just for resource changes.
- ForHackernews 2y ago
  This sounds like terraform. Is this TF for k8s?
  - pmig 2y ago
    That's an interesting analogy, but I thinks it's a stretch.
- granra 2y ago
  IMO the pull request should be the diff. https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/
sandwitches 2y ago
The solution is to not use Kubernetes.
theLiminator 2y ago
The wide adoption of YAML for devops adjacent tooling was a mistake. I think proper programming language support is the way to go. Ideally a static type system that isn't turing complete and guaranteed to terminate. So something like starlark with types.
- dventimi 2y ago
  > I think proper programming language support is the way to go. Personally, I would prefer a SQLite database. Ok I'll show myself out.
- verdverm 2y ago
  Have you looked at CUE? (https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/) CUE is also pragmatic in that it has integrations with yaml, json, jsonschema, openapi, protobuf
  - dewbrite 2y ago
    I've tried out Pkl which is similar in spirit, and I think it's a real solution for k8s manifests. The only thing holding it back is industry adoption imo. It's leagues better than Helm, and mostly better than Kustomize. See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
    - verdverm 2y ago
      I don't expect a winner personally, rather that there will be dozens of alternatives always. Like build systems, deployments are quite bespoke to organizations and legacy has a way of sticking around for a long time Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
      - pmig 2y ago
        So whats your take on https://github.com/stripe/skycfg https://github.com/stripe/skycfg do you also have experience with it?
zikohh 2y ago
Have you tried the rendered manifests pattern ? https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/
- granra 2y ago
  I read this article a while ago and it seems like the most sane way of dealing with this. Which tool you use to render the manifests doesn't even matter anymore.
- appplication 2y ago
  Interesting, we have a system (different context, though it does use yaml) that allows nested configurations, and arrived at a similar solution, where nested configs (implicit/human interface) are compiled to fully qualified specifications (explicit/machine interface). It works quite well for managing e.g. batch configurations with plenty of customization. I was unaware there was a name for this pattern, thank you.
- nijave 2y ago
  This pattern is powerful since you can pick arbitrary tooling and easily make modifications with your own tooling. For instance substituting variables/placeholders or applying static analysis.
- rorychatt 2y ago
  While I agree generally with the pattern (dynamically generating manifests, and using pipelines to co-ordinate pattern change), I could never quite figure out the value of using Branches instead of Folders (with CODEOWNER restrictions) or repositories (to enforce other types of rules if needed). I can't quite put my finger on it, but having multiple, orphaned commit histories inside a single repository sounds off, even if technically feasible.
  - theptip 2y ago
    I believe the idea is that it makes it very explicit to track provenance of code between environments, eg merge staging->master is a branch merge operation. And all the changes are explicitly tracked in CI as a diff. With directories you need to resort to diffing to spot any changes between files in folders. That said there are some merge conflict scenarios that make it a little annoying to do in practice. The author doesn’t seem to mention this one, but if you have a workflow where hotfixes can get promoted from older versions (eg prod runs 1.0.0, staging is running 1.1.0, and you need to cut 1.0.1) then you can hit merge conflicts and the dream of a simple “click to release” workflow evaporates.

pmig 2y ago

This is actually a problem we want to focus on with Glasskube Cloud (https://glasskube.cloud/ https://glasskube.cloud/) where our glasskube[bot] will comment on your GitOps Pull request with an exact diff of resources that will get changed across all connected clusters. This diff will be performed by controller running inside your cluster. Think of it as codecov analysis, but just for resource changes.

ForHackernews 2y ago
This sounds like terraform. Is this TF for k8s?
- pmig 2y ago
  That's an interesting analogy, but I thinks it's a stretch.
granra 2y ago
IMO the pull request should be the diff. https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/

sandwitches 2y ago

The solution is to not use Kubernetes.

theLiminator 2y ago

The wide adoption of YAML for devops adjacent tooling was a mistake. I think proper programming language support is the way to go. Ideally a static type system that isn't turing complete and guaranteed to terminate. So something like starlark with types.

dventimi 2y ago
> I think proper programming language support is the way to go. Personally, I would prefer a SQLite database. Ok I'll show myself out.
verdverm 2y ago
Have you looked at CUE? (https://cuelang.org/docs/concept/the-logic-of-cue/ https://cuelang.org/docs/concept/the-logic-of-cue/) CUE is also pragmatic in that it has integrations with yaml, json, jsonschema, openapi, protobuf
- dewbrite 2y ago
  I've tried out Pkl which is similar in spirit, and I think it's a real solution for k8s manifests. The only thing holding it back is industry adoption imo. It's leagues better than Helm, and mostly better than Kustomize. See also: KCL, which is very similar and might _actually_ be "the winner". Time will tell.
  - verdverm 2y ago
    I don't expect a winner personally, rather that there will be dozens of alternatives always. Like build systems, deployments are quite bespoke to organizations and legacy has a way of sticking around for a long time Having used CUE, mainly outside of Kubernetes, I cannot see myself switching to KCL. I really like having a configuration language that isn't so tied to a specific system and which I can use with the Go SDK
    - pmig 2y ago
      So whats your take on https://github.com/stripe/skycfg https://github.com/stripe/skycfg do you also have experience with it?

zikohh 2y ago

Have you tried the rendered manifests pattern ? https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/

granra 2y ago
I read this article a while ago and it seems like the most sane way of dealing with this. Which tool you use to render the manifests doesn't even matter anymore.
appplication 2y ago
Interesting, we have a system (different context, though it does use yaml) that allows nested configurations, and arrived at a similar solution, where nested configs (implicit/human interface) are compiled to fully qualified specifications (explicit/machine interface). It works quite well for managing e.g. batch configurations with plenty of customization. I was unaware there was a name for this pattern, thank you.
nijave 2y ago
This pattern is powerful since you can pick arbitrary tooling and easily make modifications with your own tooling. For instance substituting variables/placeholders or applying static analysis.
rorychatt 2y ago
While I agree generally with the pattern (dynamically generating manifests, and using pipelines to co-ordinate pattern change), I could never quite figure out the value of using Branches instead of Folders (with CODEOWNER restrictions) or repositories (to enforce other types of rules if needed). I can't quite put my finger on it, but having multiple, orphaned commit histories inside a single repository sounds off, even if technically feasible.
- theptip 2y ago
  I believe the idea is that it makes it very explicit to track provenance of code between environments, eg merge staging->master is a branch merge operation. And all the changes are explicitly tracked in CI as a diff. With directories you need to resort to diffing to spot any changes between files in folders. That said there are some merge conflict scenarios that make it a little annoying to do in practice. The author doesn’t seem to mention this one, but if you have a workflow where hotfixes can get promoted from older versions (eg prod runs 1.0.0, staging is running 1.1.0, and you need to cut 1.0.1) then you can hit merge conflicts and the dream of a simple “click to release” workflow evaporates.