15 ms·

Pitfalls of Helm – Insights from 3 years with the leading K8s package manager

137 points by louis_w_gk 3y ago

aschleck 3y ago
I found that using "helm template" to convert every Helm chart into yaml, and then using Pulumi to track changes and update my clusters (with Python transformation functions to get per-cluster configuration) made my life so much better than using Helm. Watching Pulumi or Terraform watch Helm watch Kubernetes update a deployment felt pointlessly complicated.
- jpgvm 3y ago
  I do the same with Tanka + Jsonnet, definitely a million times better than dealing with Helm itself or god forbid, letting it apply manifests.
  - personomas 3y ago
    [dead]
    - jpgvm 3y ago
      The main thing is that it's still kicking, Ksonnet is sadly dead. In addition to that it supports importing Helm charts, has a blessed convention for multiple environments and several native Jsonnet functions that make things a bit nicer.
      - personomas 3y ago
        Interesting, thanks.
- notnmeyer 3y ago
  i am hearing this more and more from folks.
- personomas 3y ago
  What about using terraform instead of Pulumi? Why did you pick Pulumi for this?
  - aschleck 3y ago
    This is both a good thing and a bad thing, but Pulumi is way more flexible than Terraform. I wanted to have a cloud-provider-specific submodule that created resources (like EKS and GKE) that then exported output values (think kubeconfig), and then I wanted the parent module to pass those in as inputs to a cloud-provider-independent submodule. Terraform couldn't do it without needing to duplicate a ton of code, or without something heinous like Terragrunt (not sure it even would have worked.) Pulumi makes it trivial and in a language I like writing. Additionally, our applications consume our cloud configuration (eg something that launches pods on heterogenous GPUs needs to know which clusters support which GPUs, our colo cluster has H100s but our Google cluster has A100s etc.) Writing in the same language in the same monorepo makes it very easy to share that state.
  - totallywrong 3y ago
    With Pulumi you really are programming infrastructure in a language of your choice. HCL is a bad joke in comparison.
empath-nirvana 3y ago
I think helm is at it's best when you need to _publicly distribute_ a complex application to a large number of people in a way that's configurable through parameters. For internal applications, it's in an awkward place of being both too complex and too simple, and in a lot of cases what you really want to do is just write your own operator for the complex cases and use kustomize for the simple cases. Most of the problems with updating and installing helm charts go away if you manage it with something like argocd to automatically keep everything up to date.
- numbsafari 3y ago
  Personally much prefer kustomize for the “ship an app” business. Probably even better is to ship a controller and a CRD for the config. Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.
  - mountainriver 3y ago
    Controller + CRD is the way to go and seems more in line with how k8s was intended to be used. The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
    - arccy 3y ago
      operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
      - mountainriver 3y ago
        Fair enough, the UX is just so much better that I'd gamble it in most use cases
  - 3y ago
deathanatos 3y ago
> See, there is no general schema for what goes and doesn't go inside a values.yaml file. Thus, your development environment cannot help you beyond basic YAML syntax highlighting. … this is just an odd complaint. Naturally, there isn't a schema — there inherently cannot be one. Values are the options for the app at hand; they're naturally dependent on the app. > but without any schema validation! I have seen people supply JSON schemas for values with the chart. I appreciate that. Of all the pitfalls … the clunky stringly-typed "manipulate YAML with unsafe string templating" is the biggest pitfall, to me…
- everforward 3y ago
  A lot of those values end up being used in places that do have schemas. I think they're asking for what is basically inferred types. They want Helm to recognize that the cpuLimit value is used as a CPU limit for a Pod and throw errors for any cpuLimit that isn't a valid CPU limit. Agreed that the user will have to write their own schema for CLI arguments.
jpgvm 3y ago
Helm makes me sad. What I do to remediate this sadness is use Helm from Tanka. There is still sadness but now it's wrapped in a nice Jsonnet wrapper and I can easily mutate the output using Jsonnet features without having to mess with nasty Go templating. I've said it a million times before but it's always worth saying again: Don't use string templating for structured data.
- ivan4th 3y ago
  Yep. Many complain that with Lisp, you need to count parentheses (spoiler: you don't need to). And then proceed to count spaces for indent/nindent in the charts... That's somehow ok with almost everyone
  - morelisp 3y ago
    It's absolutely crazy to me how many tools are in common use for k8s templating which would all be wiped away with any decent macro system.
    - speedgoose 3y ago
      The template engine is not specific to Kubernetes but Golang. I wish they used something more adapted. https://pkg.go.dev/text/template https://pkg.go.dev/text/template
      - morelisp 3y ago
        Not just helm. There are probably a half dozen tools for rendering manifests in our company, only some use text/template, and they all suck. Text replacements are bad. Declarative structured patches are bad. Control flow in JSON is bad. We've had a language for dealing with generating complex nested structured data for years!
        anonacct37 3y ago
        text/template is probably ok... For some version of text. ditto with jinja and most templating languages. The cardinal sin of DevOps is using text macros to produce structured data. It only exists because unfortunately there is no other lowest common denominator for every config file syntax.
clvx 3y ago
Another one, when you upgrade your cluster and there's an API that is candidate for removal, helm doesn't have a way to update the Kind reference in their metadata which causes the inability to delete and update the release. I personally like cuelang's philosophy but it could become a little messy when you have to iterate and handle user inputs in large codebases.
- jpgvm 3y ago
  Cue/Jsonnet/friends are definitely the right tools for the job. It's a shame they aren't more popular.
notnmeyer 3y ago
i generally don't mind helm but im not sure i agree with every point. for the really simple stateless app situation, its trivial to create a chart with all the important or unique bits extracted to a values file. the crd shit is borderline untenable. i learned about it during an absolutely cursed calico upgrade. oops. since kustomize integrates tightly with kubectl these days though, i just use that for new things. i want fewer, simpler tools.
markbnj 3y ago
Over seven years of using a variety of deployment tooling including helm (2 and 3), kustomize and our own scripting we concluded that helm's strength is as a package manager, akin to dpkg. Writing good packages is complex, but the tool is quite powerful if you take the time to do that. For our own deployments what we typically want to to do is: build, test and push an image, plug some context specific things into yaml, send the yaml to the control plane and maybe monitor the result for pod readiness. We have some custom tooling that does this in gitlab pipelines, relying on kustomize for the yaml-spattering bits. We still do use a lot of our own and third-party helm charts but for us there's a clear distinction between installing packages (which tend to be longer-term stable infra things) and rapidly iterating on deployments of our own stuff.
- degenerate 3y ago
  Any advice/ideas/articles/references on using kustomize efficiently? I love the idea of using a tool bundled with kubectl for zero dependencies, but their examples and tutorials are horrible. I can't figure out how to use it correctly to have 1 copy of YAML that would deploy to 5 different environments. It seems I would need multiple copies of kustomization.yaml in multiple folders, if I have multiple namespaces/pods/etc...
  - leetrout 3y ago
    Not a kustomize expert - but yes, you likely would have a folder for each thing you target. It wasn't bad once I got through the docs / examples. They just assume so much existing knowledge I didn't have.
  - Hamuko 3y ago
    We use kustomize with multiple copies of kustomization.yaml and I don't know if there is a way to do it without that. Basically, there's a base kustomization.yaml and then there's test/kustomization.yaml, prod1/kustomization.yaml, prod2/kustomization.yaml, and so on.
  - markbnj 3y ago
    The model is base yaml with patches applied to it results in final yaml that get sent to the api, so the typical structure for us is to have the base yaml live with the service source, be maintained by the service owners and include all environment-agnostic properties. We then have one folder per targeted environment for that service which includes any patches and the kustomization.yaml manifest. Basically in line with what other replies have mentioned.
LittleChimera 3y ago
It's a good list, although I think there's more to it even. I wrote a bit more about helm design a while ago [0]. Nowadays, I use Helm from kustomize quite a lot because some projects don't provide any other way of deploying. However, you still need to check what helm is actually generating, especially if there's any hooks that need to be replaced with something declarative. [0]: https://littlechimera.com/posts/helm-design/ https://littlechimera.com/posts/helm-design/
cortesoft 3y ago
Isn't the last point wrong? You can query the kubernetes environment in your templates to customize the output based on cluster specific things
- BossingAround 3y ago
  The point isn't that you can never query the API, but that you can't really use helm chart as a controller (and, e.g. restart a pod under a certain condition, which is trivial for an operator).
  - cortesoft 3y ago
    Oh... why wouldn't you just write an operator then? Seems like a different requirement.
renlo 3y ago
https://archive.is/grNy4 https://archive.is/grNy4
throwawaaarrgh 3y ago
...that's it? What about hooks being an anti pattern? What about upgrades potentially resulting in blowing away your whole deployment without warning? What about a lack of diffs or planning changes? Or the complexity/kludginess of go template logic? Or the lack of ability to order installation of resources or processing of subcharts? Or waiting until a resource is done before continuing to the next one? Or the difficulty of generating and sharing dynamic values between subcharts? Or just a dry run (template) that can reference the K8s api? There's a ton of pitfalls and limits. It's still a genuinely useful tool and provides value. But it's mostly useful if you use it in the simplest possible ways with small charts. I just wish the "operation engine" were decoupled from the "generation engine", and pluggable. I like how it watches the deployment, has atomic upgrades, can do rollbacks. But if you want a complex deployment, you currently have to DIY it.
dijit 3y ago
I got lazy and just wrote scripts that output k8s manifests. The development story is much better (breakpoints! WHAT!?, loops and control flow!?), you can catch common issues quicker by adding tests, there's one "serialise" step so you don't have to deal with YAML's quirks and you can version/diff your generated manifests. It's dumb, and stupid, but it works and it's far less cognitive load. Now: handling mildly dynamic content outside of those generated manifests... that's a massive pain, releasing a new version of a container and avoiding to touch the generated manifests: not working for me.
- leetrout 3y ago
  I do the same with Terraform sometimes. I appreciate that TF has loops and dynamic blocks, etc etc, but sometimes it's just a lot easier to look at a Jinja2 template and run a script to generate the TF.
- temp_praneshp 3y ago
  at my current place, we started off with kustomize. I rewrote everything into helm, which was good initially (at least you can force inject some common params, and others can include this in their charts). But people (including me) were unhappy at yaml reading; I also grew to hate it with a passion because it's neither go nor yaml, and super difficult to read in general. We are a typescript company, and https://cdk8s.io/ https://cdk8s.io/ has been great for us. We can unit test parts of charts without rendering the whole thing, distribute canonical pod/deployment/service definitions, etc. In all of the cases, we combined this with config outputted by terraform, for env specific overrides, etc.
- imglorp 3y ago
  Found the workaround confession thread. Because you effectively CAN'T dynamically configure subcharts with templating that's done in your main chart, see eg https://github.com/helm/helm/pull/6876 https://github.com/helm/helm/pull/6876 here comes the hack. We run helm in helm. The top chart runs post-install and post-upgrade hook job which runs helm in a pod with a lot of permissions. The outer helm creates values override yaml for the subchart into a ConfigMap, using liberal templating, which gets mounted in the helm runner pod. Then helm runs in there with the custom values and does its own thing. Not proud but it lets us do a lot of dynamic things straight helm can't.
francoismassot 3y ago
While I really enjoy helm when playing with k8s or kickstarting projects, I never feel "safe" when using it in the long run for updates/upgrades. "values.yaml" files and templating YAML files are too error-prone...
baq 3y ago
Helm is a tool to use Jinja to write an object tree (or dag), in yaml. This is not endorsement. This is to point out that it makes hardly any sense! Use a proper programming language and serialize the object tree/network to whatever format is necessary.
- BossingAround 3y ago
  I think that's where the landscape is heading--language frameworks that output YAML on one end, and operators that control YAMLs through the K8s control loop on the other end.
- jen20 3y ago
  > tool to use Jinja It's not really that. Jinja is python, Helm is written in Go and uses one of the Go template languages, which has a passing similarity to Jinja. The rest of your comment is spot on, of course.
- pas 3y ago
  https://cdk8s.io/docs/latest/plus/ https://cdk8s.io/docs/latest/plus/ ... fast easy and useful check as you type feedback/validation (of course because it's using the TS LSP)
jimbobimbo 3y ago
I'm not buying the example of using the operator to figure out things dynamically. Especially that detection of the cloud in the example is done by looking at some random labels or other attributes specific to a cloud provider. This is what values and templates are for: no need to guess where you are deployed, I'll tell you that via values, template will make sense and adjustments of how resources will look like.
btown 3y ago
On top of what the OP mentions, Helm still doesn't have a way to forward logs from its pre/post-install hooks to the system calling helm upgrade (such as a Github Action) - a feature first requested in 2017 and still stuck in RFC stage. https://github.com/helm/helm/issues/2298 https://github.com/helm/helm/issues/2298 https://github.com/helm/helm/pull/10309 https://github.com/helm/helm/pull/10309 https://github.com/helm/community/pull/301 https://github.com/helm/community/pull/301 I can understand moving cautiously, but it's at a point where it almost feels like allowing users to understand what Helm is doing seems not to be a priority for Helm's developers.
galenmarchetti 3y ago
points #3 and #4; "user-friendly helm chart creation" and "values.yaml is an antipattern"...I think we're just all stuck in this horrible middle ground between "need static declarative configurations for simplicity of change management/fewest chances to mess it up" and "need dynamic, sometimes even imperative logic for flexibility, configurability, and ease of development" several commenters have mentioned Cue/Jsonnet/friends as great alternatives, others find them limiting / prefer pulumi with a general purpose language our solution at kurtosis is another, and tilt.dev took the same route we did...adopt starlark as a balanced middle-ground between general-purpose languages and static configs. you do get the lovely experience of writing in something pythonic, but without the "oops this k8s deployment is not runnable/reproducible in other clusters because I had non-deterministic evaluation / relied on external, non-portable devices"
ithkuil 3y ago
A few years I go I tried out an alternative approach to "templating". Basically the idea starts from a world without templates where you would distribute the k8s YAML in a form that is ready to be directly applied, with whatever sensible defaults you want directly present in the YAML The the user would then just change the values in their copy of the file to suit their needs and apply that. We all recoil in horror to such a thought, but let's stop a moment to think about why we do: The user effectively "forked" the YAML by placing their values there and what a nightmare would that be once the user would get a new version of the upstream file, potentially completely overhauled . If the changes are very small, a simple three way merge like you'd do with git would suffice to handle that. But what about larger changes? Most of the conflicts in the simple cases stem from the fact that text based diff/merge tools are oblivious to the structure of the YAML file and can only so a so-so job with many of the changes. Unfortunately most people are familiar only with text based merge tools and so they have been primed the hard way to assume that the merges only rarely work. Structural merges otoh so work much much better. But still if the upstream refractors the application in a significant way (e.g. changes a deployment into a stateful set or moves pieces of config from a configmap into a secret!) not even a structural merge can save you. My idea was to bring the manifest author into play and make them "annotate" the pieces of the manifest forest that contain configuration that has a high level meaning to the application and that would be moved around in the YAML forest as it gets reshaped. Another realization was that often such configuration snippets are deeply embedded in other internal "languages" wrapped inside string fields, subject to escaping and encodings (e.g. base64). E.g. a JSON snippet inside a TOML string value inside # base64 encoded annotation value (if you haven't seen these abominations I'm so happy for you you innocent child) So I implemented a tool that uses neated bidirectional parsers ("lenses") that can perform in-place editing of structured files. The edits preserve formatting, comments, quoting styles, etc. Even steing fields that are normally thought of as just strings are actually better though if as nested "formats". For example the OCI image references are composed of multiple parts. If you want to just copy images to your private registry and "rebase" all you image references to the new base, you can do it with an update that understands the format of the OCI image references instead of just doing substring replacement. Knot8 is an opinionated tool meant to help manifest authors and users manage setting/diffing/pulling annotated YAML k8s manifest packages https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 I didn't have the time to evangelize this approach much so it didn't get any traction (and perhaps it would because it doesn't have enough merits). But I encourage to give it a go. It might inspire you I also pulled out the "lens" mechanism in a separate binary in case it could be useful to edit general purpose files: https://github.com/kubecfg/lensed https://github.com/kubecfg/lensed
nunez 3y ago
Despite its pitfalls, I've found that Helm is still the best way to distribute an app that's destined for Kubernetes. It's easy to use and understand and provides just enough for me to ship an app along with its dependencies. I use kapp from Project Carvel and the "helm template" subcommand to work around Helm's inability to control desired state. I've found that kapp does a pretty good job of converging whatever resources Helm installed.
- s0l1dsnak3123 3y ago
  +1 for Carvel suite. To me it hits the sweet spot by allowing you to compose your own pipelines. It's not too alien for other k8s folks, and UNIXy enough that you can cut out the stuff you don't care about.
brainzap 3y ago
we use helm to deploy our apps and it works. Actually to be honest the templates are generated and then deployed from disk, without uploading a chart. We try to avoid the template language. What Helm can't do currently is handle Ingress renames, and they do not allow loading files that are outside the chart.
Smaug123 3y ago
Unhygienic string templating of a whitespace-sensitive language is a truly new hell for those who have not experienced it before. Quite an astonishing decision.
- ahoka 3y ago
  Does not beat the “BIND zone files generated by m4 macros from a shell script”, I had the honor having to modify. Comes close second, though.
  - cheekibreeki2 3y ago
    No raw sendmail.cf? :)
    - doubled112 3y ago
      Thanks, that little twitch under my eye just started up. I worked at an org where an admin went a little rogue and updated the config without using the macro file. Another admin didn't realize. Well, didn't realize until it was too late.
    - rwmj 3y ago
      autoconf (configure.ac) files are also m4 hell, with some whitespace surprises.
- ljm 3y ago
  I don't think I'll ever understand why YAML was chosen as the language of choice for all things devops, and why so many startups have sought to augment yaml with syntactical hacks I mean, as soon as you template it and need to do `{{ something | indent 4 }}` or some shit to make the template work you know you're on a bad track.
  - taspeotis 3y ago
    Yelling At My Laptop
  - 3y ago
gtsteve 3y ago
This is relevant to a discussion I am having at work right now. I am not a fan of using a templating language as such to generate string templates, especially for a whitespace sensitive language. I would rather use Terraform's Kubernetes or Kubectl module for this. Are there any pros or cons I should consider? I think one of the key things I like about it is that Terraform will show me what it plans to change whereas Helm doesn't (last time I checked)
- noamchomsky1 3y ago
  it's a horrible experience with terraform too.
  - gtsteve 3y ago
    Do tell?
    - klooney 3y ago
      Terraform wants to be the only thing that owns a K8S object, but the way things work in reality is you have a dozen things that want to write back to this attribute, or that overwrite objects in other places, etc, and you're constantly fighting with TF about this or that triviality.
  - mdaniel 3y ago
    every time I hear someone suggest such a thing, I remind them that now you have two systems who believe they own the state of the world: .tfstate and etcd and let me assure you that no matter how much our dumbass TF friend thinks it knows what's going on, etcd wins hands down every time that's why I strongly suggest that if anyone is a "whole TF shop," they go the operator route, because trying any lower level management is the road to ruin
- willejs 3y ago
  The kubernetes provider, and kubectl works, but its not the nicest way of making changes. Its slow, quite clunky, and its not particularly intuitive. If your just getting started, and you know terraform its ok though. Its useful to bootstrap gitops tools like Argo or FluxCD though. Helm diff will show you a similar diff to terraform. Running Helmfile in CD isn't a bad move, its really simple, and its a pattern that is easy to grok by any engineer. I think this is still a valid approach in a simple setup, its what some people call "CD OPS". It's a push model instead of pull, and there are downsides, but its not the end of the world. Ultimately, at scale, i think gitops tooling like Flux and ArgoCD are some of the nicest patterns. Especially Flux's support for OCI artifacts as a source of truth. However then you will venture into the realm of kustomize, and much more complex tooling and concepts, which is not always worth doing.
octacat 3y ago
Needed to install pg/mysql using helm charts and needed to apply SQL schema. And passing schema is so complex... (like write it into values file, but you cannot use any logic in values, so have an external script which would template values files). At least for the most popular Helm charts I've seen so far. Kinda sad because it is trivial to do in Docker. And still possible to do in k8s with some configmaps.
aliasxneo 3y ago
For those not aware, there's https://timoni.sh https://timoni.sh. It's similar to Helm but used CUE instead of the Go templating language. We've recently switched to it and are not looking back. Would be great to get some more contributors on board.
Octabrain 3y ago
I have to admit that I would chose Kustomize any day of the week. I personally find it easier to debug, version, grasp what is going on behind the scenes and simpler to maintain and operate. Also, to be fair, I have to admit that I've had a few negative experiences with Helm that made me to develop a negative bias against it. The main one being, to make the story short: I, as a DevOps guy, had the unpleasant experience of having to take care of the deployment process of a HeLLm chart(s) pile of crap that a very opinionated yet not very knowledgeable (on DevOps practices) team of developers created in a company I worked for in the past. It was composed by a myriad of obscure charts, jammed together as a house of cards. If that wasn't enough, the main chart was being called from a bash script that called Ansible and it pulled some data from a repo and rendered values coming from a repo into the values.yaml. The process of deploying that thing implied, apart from a couple of good hours of your time, to also have to perform some debugging sorcery with kubectl (removing this, modifying that etc) in a meeting while screen sharing your terminal with the proud parents of that monster. Absolute nightmare that still giving me sweaty hand palms to this day. Anyways, I want to take the opportunity to ask other DevOps in the room something: What workflow do you use for performing CD for a bunch of Helm charts? I mean, I guess you version a bunch of values.yaml files in a repo but how do you manage the installation of the repository from the CD runner and so on? I'm curious because from the top of my head, Helm implies the runner (the actor who calls Helm to install a chart) to install a repository and then install the package from that repository passing to it the path to the values.yaml file, but on a CD pipeline, this actor is usually a disposable container. Do you install during one of the steps of the CD pipeline the repositories from an in-house created script perhaps every time the pipeline runs? Are you hopefully using a GitOps approach instead? Thanks in advance for any answer.
- totallywrong 3y ago
  That was just badly done as anything can be. The template engine takes some getting used to, specially without Go experience. But I'd much rather have that than a million lines of manifests as in with Kustomize. That said, you need either Argo or Flux and only deploy Helm charts with GitOps workflows.
sigwinch28 3y ago
Helm and Kustomize are absolutely atrocious tools. The job of these tools is ultimately to generate a bunch of data structures compatible with Kubernetes API schemas. That's it. Take some input, go brrr, and spit out some serialized data structure. If we replace "YAML" with "JSON", this all seems a bit absurd. Helm is a tool where you write JSON templates and JSON templating helper functions, then users can provide a values.json file which specifies inputs for the templates and maybe even can contain templated values themselves. But it's easy to get the contents of the values.json slightly wrong in ways that silently fail, so the JSON templates spit out something but it might still be semantically valid Kubernetes JSON. So you can include JSON schema files alongside your JSON templates package to specify the schema of the values.json file, which the tool will check. Then sometimes the values.json file is too verbose to write by hand, so the user chooses another tool (or god forbid writes a small script in a general purpose programming language) to generate their values.json file. Absurd. Kustomize is a tool where you give it a bunch of JSON files, or maybe tell it to use one or more of the hellish Helm charts, which is itself a glorified JSON templater. Then you write a JSON configuration file which looks declarative but is secretly somewhat imperative. In this JSON file you specify which transformations you'd like to apply, maybe even specify some inline patches which use the JSON path patch syntax or a strategic merge where you write some more JSON. Some bits of the tool allow you to generate more JSON from files on your disk, like .env files or data files. If the feature set doesn't work you can then use KRM functions as generators or transformers, which allow you to escape from JSON hell and use a general purpose programming language to emit or transform JSON, but the configuration to this KRM function is itself provided as JSON. Absurd. These YAML-obsessed tools like Helm and Kustomize purport to be easier because "it's just YAML!" but what you're really doing is writing all of the arguments for a computer program which you must understand, but in YAML. Kustomize is slightly less bad because it doesn't allow for much control flow, but in Helm templates you have control flow, loops, and helper functions. So it's really just a program anyway with a YAML syntax, but you're confined to a stringly-typed templating language. This exists with tools like Kyverno, too: "oh, the policies are just YAML!" but they introduce a special DSL for writing conditionals, so you're still writing a program anyway, but in a YAML inner platform. If you think of it as "just JSON" then the YAML smokescreen and cargo cult falls away. It then becomes much more tempting to write something specific to your needs in Python, TypeScript, Ruby, Haskell, OCaml, or another general purpose programming language, where you can do whatever it is you need to do without this leaky YAML purism. Kubewarden's scope overlaps a bit with Kyverno, but Kubewarden chooses WASM as the common language instead of YAML. However, there's an argument to be made about the security model of Helm and Kustomize. Assume that I trust that the authors of those tools are not going to smuggle my data to a malicious third party or install spyware on my computer and that they do not have critical security bugs. Then I can confidently run those tools on my computer with inputs (e.g. charts and kustomizations) that I do not trust. To say nothing of whether the tool will generate a deployment which compromises my cluster when applied. I think what I'd like to see from the community is tooling that targets Deno or WASM as the common ground. Then I can sufficiently sandbox "packages" like helm charts or kustomizations without worrying that they have general access to my computer or CI/CD runner as if I'd imported a random Python package. I've already started writing something in Deno that tries to emulate Grafana's Tanka in TypeScript. My next job is to write a tool which runs WASM-compiled KRM functions. But I might need a configuration language for that tool, so maybe I'll choose YAML...
sureglymop 3y ago
Maybe I am just daft but I really never understood why an infrastructure resource management platform needed a package manager.

137 points by louis_w_gk 3y ago

I found that using "helm template" to convert every Helm chart into yaml, and then using Pulumi to track changes and update my clusters (with Python transformation functions to get per-cluster configuration) made my life so much better than using Helm. Watching Pulumi or Terraform watch Helm watch Kubernetes update a deployment felt pointlessly complicated.

jpgvm 3y ago
I do the same with Tanka + Jsonnet, definitely a million times better than dealing with Helm itself or god forbid, letting it apply manifests.
- personomas 3y ago
  [dead]
  - jpgvm 3y ago
    The main thing is that it's still kicking, Ksonnet is sadly dead. In addition to that it supports importing Helm charts, has a blessed convention for multiple environments and several native Jsonnet functions that make things a bit nicer.
    - personomas 3y ago
      Interesting, thanks.
notnmeyer 3y ago
i am hearing this more and more from folks.
personomas 3y ago
What about using terraform instead of Pulumi? Why did you pick Pulumi for this?
- aschleck 3y ago
  This is both a good thing and a bad thing, but Pulumi is way more flexible than Terraform. I wanted to have a cloud-provider-specific submodule that created resources (like EKS and GKE) that then exported output values (think kubeconfig), and then I wanted the parent module to pass those in as inputs to a cloud-provider-independent submodule. Terraform couldn't do it without needing to duplicate a ton of code, or without something heinous like Terragrunt (not sure it even would have worked.) Pulumi makes it trivial and in a language I like writing. Additionally, our applications consume our cloud configuration (eg something that launches pods on heterogenous GPUs needs to know which clusters support which GPUs, our colo cluster has H100s but our Google cluster has A100s etc.) Writing in the same language in the same monorepo makes it very easy to share that state.
- totallywrong 3y ago
  With Pulumi you really are programming infrastructure in a language of your choice. HCL is a bad joke in comparison.

empath-nirvana 3y ago

I think helm is at it's best when you need to _publicly distribute_ a complex application to a large number of people in a way that's configurable through parameters. For internal applications, it's in an awkward place of being both too complex and too simple, and in a lot of cases what you really want to do is just write your own operator for the complex cases and use kustomize for the simple cases. Most of the problems with updating and installing helm charts go away if you manage it with something like argocd to automatically keep everything up to date.

numbsafari 3y ago
Personally much prefer kustomize for the “ship an app” business. Probably even better is to ship a controller and a CRD for the config. Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.
- mountainriver 3y ago
  Controller + CRD is the way to go and seems more in line with how k8s was intended to be used. The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
  - arccy 3y ago
    operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
    - mountainriver 3y ago
      Fair enough, the UX is just so much better that I'd gamble it in most use cases
- 3y ago

deathanatos 3y ago

> See, there is no general schema for what goes and doesn't go inside a values.yaml file. Thus, your development environment cannot help you beyond basic YAML syntax highlighting. … this is just an odd complaint. Naturally, there isn't a schema — there inherently cannot be one. Values are the options for the app at hand; they're naturally dependent on the app. > but without any schema validation! I have seen people supply JSON schemas for values with the chart. I appreciate that. Of all the pitfalls … the clunky stringly-typed "manipulate YAML with unsafe string templating" is the biggest pitfall, to me…

everforward 3y ago
A lot of those values end up being used in places that do have schemas. I think they're asking for what is basically inferred types. They want Helm to recognize that the cpuLimit value is used as a CPU limit for a Pod and throw errors for any cpuLimit that isn't a valid CPU limit. Agreed that the user will have to write their own schema for CLI arguments.

jpgvm 3y ago

Helm makes me sad. What I do to remediate this sadness is use Helm from Tanka. There is still sadness but now it's wrapped in a nice Jsonnet wrapper and I can easily mutate the output using Jsonnet features without having to mess with nasty Go templating. I've said it a million times before but it's always worth saying again: Don't use string templating for structured data.

ivan4th 3y ago
Yep. Many complain that with Lisp, you need to count parentheses (spoiler: you don't need to). And then proceed to count spaces for indent/nindent in the charts... That's somehow ok with almost everyone
- morelisp 3y ago
  It's absolutely crazy to me how many tools are in common use for k8s templating which would all be wiped away with any decent macro system.
  - speedgoose 3y ago
    The template engine is not specific to Kubernetes but Golang. I wish they used something more adapted. https://pkg.go.dev/text/template https://pkg.go.dev/text/template
    - morelisp 3y ago
      Not just helm. There are probably a half dozen tools for rendering manifests in our company, only some use text/template, and they all suck. Text replacements are bad. Declarative structured patches are bad. Control flow in JSON is bad. We've had a language for dealing with generating complex nested structured data for years!
      - anonacct37 3y ago
        text/template is probably ok... For some version of text. ditto with jinja and most templating languages. The cardinal sin of DevOps is using text macros to produce structured data. It only exists because unfortunately there is no other lowest common denominator for every config file syntax.

clvx 3y ago

Another one, when you upgrade your cluster and there's an API that is candidate for removal, helm doesn't have a way to update the Kind reference in their metadata which causes the inability to delete and update the release. I personally like cuelang's philosophy but it could become a little messy when you have to iterate and handle user inputs in large codebases.

jpgvm 3y ago
Cue/Jsonnet/friends are definitely the right tools for the job. It's a shame they aren't more popular.

notnmeyer 3y ago

i generally don't mind helm but im not sure i agree with every point. for the really simple stateless app situation, its trivial to create a chart with all the important or unique bits extracted to a values file. the crd shit is borderline untenable. i learned about it during an absolutely cursed calico upgrade. oops. since kustomize integrates tightly with kubectl these days though, i just use that for new things. i want fewer, simpler tools.

markbnj 3y ago

Over seven years of using a variety of deployment tooling including helm (2 and 3), kustomize and our own scripting we concluded that helm's strength is as a package manager, akin to dpkg. Writing good packages is complex, but the tool is quite powerful if you take the time to do that. For our own deployments what we typically want to to do is: build, test and push an image, plug some context specific things into yaml, send the yaml to the control plane and maybe monitor the result for pod readiness. We have some custom tooling that does this in gitlab pipelines, relying on kustomize for the yaml-spattering bits. We still do use a lot of our own and third-party helm charts but for us there's a clear distinction between installing packages (which tend to be longer-term stable infra things) and rapidly iterating on deployments of our own stuff.

degenerate 3y ago
Any advice/ideas/articles/references on using kustomize efficiently? I love the idea of using a tool bundled with kubectl for zero dependencies, but their examples and tutorials are horrible. I can't figure out how to use it correctly to have 1 copy of YAML that would deploy to 5 different environments. It seems I would need multiple copies of kustomization.yaml in multiple folders, if I have multiple namespaces/pods/etc...
- leetrout 3y ago
  Not a kustomize expert - but yes, you likely would have a folder for each thing you target. It wasn't bad once I got through the docs / examples. They just assume so much existing knowledge I didn't have.
- Hamuko 3y ago
  We use kustomize with multiple copies of kustomization.yaml and I don't know if there is a way to do it without that. Basically, there's a base kustomization.yaml and then there's test/kustomization.yaml, prod1/kustomization.yaml, prod2/kustomization.yaml, and so on.
- markbnj 3y ago
  The model is base yaml with patches applied to it results in final yaml that get sent to the api, so the typical structure for us is to have the base yaml live with the service source, be maintained by the service owners and include all environment-agnostic properties. We then have one folder per targeted environment for that service which includes any patches and the kustomization.yaml manifest. Basically in line with what other replies have mentioned.

LittleChimera 3y ago

It's a good list, although I think there's more to it even. I wrote a bit more about helm design a while ago [0]. Nowadays, I use Helm from kustomize quite a lot because some projects don't provide any other way of deploying. However, you still need to check what helm is actually generating, especially if there's any hooks that need to be replaced with something declarative. [0]: https://littlechimera.com/posts/helm-design/ https://littlechimera.com/posts/helm-design/

cortesoft 3y ago

Isn't the last point wrong? You can query the kubernetes environment in your templates to customize the output based on cluster specific things

BossingAround 3y ago
The point isn't that you can never query the API, but that you can't really use helm chart as a controller (and, e.g. restart a pod under a certain condition, which is trivial for an operator).
- cortesoft 3y ago
  Oh... why wouldn't you just write an operator then? Seems like a different requirement.

renlo 3y ago

https://archive.is/grNy4 https://archive.is/grNy4

throwawaaarrgh 3y ago

...that's it? What about hooks being an anti pattern? What about upgrades potentially resulting in blowing away your whole deployment without warning? What about a lack of diffs or planning changes? Or the complexity/kludginess of go template logic? Or the lack of ability to order installation of resources or processing of subcharts? Or waiting until a resource is done before continuing to the next one? Or the difficulty of generating and sharing dynamic values between subcharts? Or just a dry run (template) that can reference the K8s api? There's a ton of pitfalls and limits. It's still a genuinely useful tool and provides value. But it's mostly useful if you use it in the simplest possible ways with small charts. I just wish the "operation engine" were decoupled from the "generation engine", and pluggable. I like how it watches the deployment, has atomic upgrades, can do rollbacks. But if you want a complex deployment, you currently have to DIY it.

dijit 3y ago

I got lazy and just wrote scripts that output k8s manifests. The development story is much better (breakpoints! WHAT!?, loops and control flow!?), you can catch common issues quicker by adding tests, there's one "serialise" step so you don't have to deal with YAML's quirks and you can version/diff your generated manifests. It's dumb, and stupid, but it works and it's far less cognitive load. Now: handling mildly dynamic content outside of those generated manifests... that's a massive pain, releasing a new version of a container and avoiding to touch the generated manifests: not working for me.

leetrout 3y ago
I do the same with Terraform sometimes. I appreciate that TF has loops and dynamic blocks, etc etc, but sometimes it's just a lot easier to look at a Jinja2 template and run a script to generate the TF.
temp_praneshp 3y ago
at my current place, we started off with kustomize. I rewrote everything into helm, which was good initially (at least you can force inject some common params, and others can include this in their charts). But people (including me) were unhappy at yaml reading; I also grew to hate it with a passion because it's neither go nor yaml, and super difficult to read in general. We are a typescript company, and https://cdk8s.io/ https://cdk8s.io/ has been great for us. We can unit test parts of charts without rendering the whole thing, distribute canonical pod/deployment/service definitions, etc. In all of the cases, we combined this with config outputted by terraform, for env specific overrides, etc.
imglorp 3y ago
Found the workaround confession thread. Because you effectively CAN'T dynamically configure subcharts with templating that's done in your main chart, see eg https://github.com/helm/helm/pull/6876 https://github.com/helm/helm/pull/6876 here comes the hack. We run helm in helm. The top chart runs post-install and post-upgrade hook job which runs helm in a pod with a lot of permissions. The outer helm creates values override yaml for the subchart into a ConfigMap, using liberal templating, which gets mounted in the helm runner pod. Then helm runs in there with the custom values and does its own thing. Not proud but it lets us do a lot of dynamic things straight helm can't.

francoismassot 3y ago

While I really enjoy helm when playing with k8s or kickstarting projects, I never feel "safe" when using it in the long run for updates/upgrades. "values.yaml" files and templating YAML files are too error-prone...

baq 3y ago

Helm is a tool to use Jinja to write an object tree (or dag), in yaml. This is not endorsement. This is to point out that it makes hardly any sense! Use a proper programming language and serialize the object tree/network to whatever format is necessary.

BossingAround 3y ago
I think that's where the landscape is heading--language frameworks that output YAML on one end, and operators that control YAMLs through the K8s control loop on the other end.
jen20 3y ago
> tool to use Jinja It's not really that. Jinja is python, Helm is written in Go and uses one of the Go template languages, which has a passing similarity to Jinja. The rest of your comment is spot on, of course.
pas 3y ago
https://cdk8s.io/docs/latest/plus/ https://cdk8s.io/docs/latest/plus/ ... fast easy and useful check as you type feedback/validation (of course because it's using the TS LSP)

jimbobimbo 3y ago

I'm not buying the example of using the operator to figure out things dynamically. Especially that detection of the cloud in the example is done by looking at some random labels or other attributes specific to a cloud provider. This is what values and templates are for: no need to guess where you are deployed, I'll tell you that via values, template will make sense and adjustments of how resources will look like.

btown 3y ago

On top of what the OP mentions, Helm still doesn't have a way to forward logs from its pre/post-install hooks to the system calling helm upgrade (such as a Github Action) - a feature first requested in 2017 and still stuck in RFC stage. https://github.com/helm/helm/issues/2298 https://github.com/helm/helm/issues/2298 https://github.com/helm/helm/pull/10309 https://github.com/helm/helm/pull/10309 https://github.com/helm/community/pull/301 https://github.com/helm/community/pull/301 I can understand moving cautiously, but it's at a point where it almost feels like allowing users to understand what Helm is doing seems not to be a priority for Helm's developers.

galenmarchetti 3y ago

points #3 and #4; "user-friendly helm chart creation" and "values.yaml is an antipattern"...I think we're just all stuck in this horrible middle ground between "need static declarative configurations for simplicity of change management/fewest chances to mess it up" and "need dynamic, sometimes even imperative logic for flexibility, configurability, and ease of development" several commenters have mentioned Cue/Jsonnet/friends as great alternatives, others find them limiting / prefer pulumi with a general purpose language our solution at kurtosis is another, and tilt.dev took the same route we did...adopt starlark as a balanced middle-ground between general-purpose languages and static configs. you do get the lovely experience of writing in something pythonic, but without the "oops this k8s deployment is not runnable/reproducible in other clusters because I had non-deterministic evaluation / relied on external, non-portable devices"

ithkuil 3y ago

A few years I go I tried out an alternative approach to "templating". Basically the idea starts from a world without templates where you would distribute the k8s YAML in a form that is ready to be directly applied, with whatever sensible defaults you want directly present in the YAML The the user would then just change the values in their copy of the file to suit their needs and apply that. We all recoil in horror to such a thought, but let's stop a moment to think about why we do: The user effectively "forked" the YAML by placing their values there and what a nightmare would that be once the user would get a new version of the upstream file, potentially completely overhauled . If the changes are very small, a simple three way merge like you'd do with git would suffice to handle that. But what about larger changes? Most of the conflicts in the simple cases stem from the fact that text based diff/merge tools are oblivious to the structure of the YAML file and can only so a so-so job with many of the changes. Unfortunately most people are familiar only with text based merge tools and so they have been primed the hard way to assume that the merges only rarely work. Structural merges otoh so work much much better. But still if the upstream refractors the application in a significant way (e.g. changes a deployment into a stateful set or moves pieces of config from a configmap into a secret!) not even a structural merge can save you. My idea was to bring the manifest author into play and make them "annotate" the pieces of the manifest forest that contain configuration that has a high level meaning to the application and that would be moved around in the YAML forest as it gets reshaped. Another realization was that often such configuration snippets are deeply embedded in other internal "languages" wrapped inside string fields, subject to escaping and encodings (e.g. base64). E.g. a JSON snippet inside a TOML string value inside # base64 encoded annotation value (if you haven't seen these abominations I'm so happy for you you innocent child) So I implemented a tool that uses neated bidirectional parsers ("lenses") that can perform in-place editing of structured files. The edits preserve formatting, comments, quoting styles, etc. Even steing fields that are normally thought of as just strings are actually better though if as nested "formats". For example the OCI image references are composed of multiple parts. If you want to just copy images to your private registry and "rebase" all you image references to the new base, you can do it with an update that understands the format of the OCI image references instead of just doing substring replacement. Knot8 is an opinionated tool meant to help manifest authors and users manage setting/diffing/pulling annotated YAML k8s manifest packages https://github.com/mkmik/knot8 https://github.com/mkmik/knot8 I didn't have the time to evangelize this approach much so it didn't get any traction (and perhaps it would because it doesn't have enough merits). But I encourage to give it a go. It might inspire you I also pulled out the "lens" mechanism in a separate binary in case it could be useful to edit general purpose files: https://github.com/kubecfg/lensed https://github.com/kubecfg/lensed

nunez 3y ago

Despite its pitfalls, I've found that Helm is still the best way to distribute an app that's destined for Kubernetes. It's easy to use and understand and provides just enough for me to ship an app along with its dependencies. I use kapp from Project Carvel and the "helm template" subcommand to work around Helm's inability to control desired state. I've found that kapp does a pretty good job of converging whatever resources Helm installed.

s0l1dsnak3123 3y ago
+1 for Carvel suite. To me it hits the sweet spot by allowing you to compose your own pipelines. It's not too alien for other k8s folks, and UNIXy enough that you can cut out the stuff you don't care about.

brainzap 3y ago

we use helm to deploy our apps and it works. Actually to be honest the templates are generated and then deployed from disk, without uploading a chart. We try to avoid the template language. What Helm can't do currently is handle Ingress renames, and they do not allow loading files that are outside the chart.

Smaug123 3y ago

Unhygienic string templating of a whitespace-sensitive language is a truly new hell for those who have not experienced it before. Quite an astonishing decision.

ahoka 3y ago
Does not beat the “BIND zone files generated by m4 macros from a shell script”, I had the honor having to modify. Comes close second, though.
- cheekibreeki2 3y ago
  No raw sendmail.cf? :)
  - doubled112 3y ago
    Thanks, that little twitch under my eye just started up. I worked at an org where an admin went a little rogue and updated the config without using the macro file. Another admin didn't realize. Well, didn't realize until it was too late.
  - rwmj 3y ago
    autoconf (configure.ac) files are also m4 hell, with some whitespace surprises.
ljm 3y ago
I don't think I'll ever understand why YAML was chosen as the language of choice for all things devops, and why so many startups have sought to augment yaml with syntactical hacks I mean, as soon as you template it and need to do `{{ something | indent 4 }}` or some shit to make the template work you know you're on a bad track.
- taspeotis 3y ago
  Yelling At My Laptop
- 3y ago

gtsteve 3y ago

This is relevant to a discussion I am having at work right now. I am not a fan of using a templating language as such to generate string templates, especially for a whitespace sensitive language. I would rather use Terraform's Kubernetes or Kubectl module for this. Are there any pros or cons I should consider? I think one of the key things I like about it is that Terraform will show me what it plans to change whereas Helm doesn't (last time I checked)

noamchomsky1 3y ago
it's a horrible experience with terraform too.
- gtsteve 3y ago
  Do tell?
  - klooney 3y ago
    Terraform wants to be the only thing that owns a K8S object, but the way things work in reality is you have a dozen things that want to write back to this attribute, or that overwrite objects in other places, etc, and you're constantly fighting with TF about this or that triviality.
- mdaniel 3y ago
  every time I hear someone suggest such a thing, I remind them that now you have two systems who believe they own the state of the world: .tfstate and etcd and let me assure you that no matter how much our dumbass TF friend thinks it knows what's going on, etcd wins hands down every time that's why I strongly suggest that if anyone is a "whole TF shop," they go the operator route, because trying any lower level management is the road to ruin
willejs 3y ago
The kubernetes provider, and kubectl works, but its not the nicest way of making changes. Its slow, quite clunky, and its not particularly intuitive. If your just getting started, and you know terraform its ok though. Its useful to bootstrap gitops tools like Argo or FluxCD though. Helm diff will show you a similar diff to terraform. Running Helmfile in CD isn't a bad move, its really simple, and its a pattern that is easy to grok by any engineer. I think this is still a valid approach in a simple setup, its what some people call "CD OPS". It's a push model instead of pull, and there are downsides, but its not the end of the world. Ultimately, at scale, i think gitops tooling like Flux and ArgoCD are some of the nicest patterns. Especially Flux's support for OCI artifacts as a source of truth. However then you will venture into the realm of kustomize, and much more complex tooling and concepts, which is not always worth doing.

octacat 3y ago

Needed to install pg/mysql using helm charts and needed to apply SQL schema. And passing schema is so complex... (like write it into values file, but you cannot use any logic in values, so have an external script which would template values files). At least for the most popular Helm charts I've seen so far. Kinda sad because it is trivial to do in Docker. And still possible to do in k8s with some configmaps.

aliasxneo 3y ago

For those not aware, there's https://timoni.sh https://timoni.sh. It's similar to Helm but used CUE instead of the Go templating language. We've recently switched to it and are not looking back. Would be great to get some more contributors on board.

Octabrain 3y ago

I have to admit that I would chose Kustomize any day of the week. I personally find it easier to debug, version, grasp what is going on behind the scenes and simpler to maintain and operate. Also, to be fair, I have to admit that I've had a few negative experiences with Helm that made me to develop a negative bias against it. The main one being, to make the story short: I, as a DevOps guy, had the unpleasant experience of having to take care of the deployment process of a HeLLm chart(s) pile of crap that a very opinionated yet not very knowledgeable (on DevOps practices) team of developers created in a company I worked for in the past. It was composed by a myriad of obscure charts, jammed together as a house of cards. If that wasn't enough, the main chart was being called from a bash script that called Ansible and it pulled some data from a repo and rendered values coming from a repo into the values.yaml. The process of deploying that thing implied, apart from a couple of good hours of your time, to also have to perform some debugging sorcery with kubectl (removing this, modifying that etc) in a meeting while screen sharing your terminal with the proud parents of that monster. Absolute nightmare that still giving me sweaty hand palms to this day. Anyways, I want to take the opportunity to ask other DevOps in the room something: What workflow do you use for performing CD for a bunch of Helm charts? I mean, I guess you version a bunch of values.yaml files in a repo but how do you manage the installation of the repository from the CD runner and so on? I'm curious because from the top of my head, Helm implies the runner (the actor who calls Helm to install a chart) to install a repository and then install the package from that repository passing to it the path to the values.yaml file, but on a CD pipeline, this actor is usually a disposable container. Do you install during one of the steps of the CD pipeline the repositories from an in-house created script perhaps every time the pipeline runs? Are you hopefully using a GitOps approach instead? Thanks in advance for any answer.

totallywrong 3y ago
That was just badly done as anything can be. The template engine takes some getting used to, specially without Go experience. But I'd much rather have that than a million lines of manifests as in with Kustomize. That said, you need either Argo or Flux and only deploy Helm charts with GitOps workflows.

sigwinch28 3y ago

Helm and Kustomize are absolutely atrocious tools. The job of these tools is ultimately to generate a bunch of data structures compatible with Kubernetes API schemas. That's it. Take some input, go brrr, and spit out some serialized data structure. If we replace "YAML" with "JSON", this all seems a bit absurd. Helm is a tool where you write JSON templates and JSON templating helper functions, then users can provide a values.json file which specifies inputs for the templates and maybe even can contain templated values themselves. But it's easy to get the contents of the values.json slightly wrong in ways that silently fail, so the JSON templates spit out something but it might still be semantically valid Kubernetes JSON. So you can include JSON schema files alongside your JSON templates package to specify the schema of the values.json file, which the tool will check. Then sometimes the values.json file is too verbose to write by hand, so the user chooses another tool (or god forbid writes a small script in a general purpose programming language) to generate their values.json file. Absurd. Kustomize is a tool where you give it a bunch of JSON files, or maybe tell it to use one or more of the hellish Helm charts, which is itself a glorified JSON templater. Then you write a JSON configuration file which looks declarative but is secretly somewhat imperative. In this JSON file you specify which transformations you'd like to apply, maybe even specify some inline patches which use the JSON path patch syntax or a strategic merge where you write some more JSON. Some bits of the tool allow you to generate more JSON from files on your disk, like .env files or data files. If the feature set doesn't work you can then use KRM functions as generators or transformers, which allow you to escape from JSON hell and use a general purpose programming language to emit or transform JSON, but the configuration to this KRM function is itself provided as JSON. Absurd. These YAML-obsessed tools like Helm and Kustomize purport to be easier because "it's just YAML!" but what you're really doing is writing all of the arguments for a computer program which you must understand, but in YAML. Kustomize is slightly less bad because it doesn't allow for much control flow, but in Helm templates you have control flow, loops, and helper functions. So it's really just a program anyway with a YAML syntax, but you're confined to a stringly-typed templating language. This exists with tools like Kyverno, too: "oh, the policies are just YAML!" but they introduce a special DSL for writing conditionals, so you're still writing a program anyway, but in a YAML inner platform. If you think of it as "just JSON" then the YAML smokescreen and cargo cult falls away. It then becomes much more tempting to write something specific to your needs in Python, TypeScript, Ruby, Haskell, OCaml, or another general purpose programming language, where you can do whatever it is you need to do without this leaky YAML purism. Kubewarden's scope overlaps a bit with Kyverno, but Kubewarden chooses WASM as the common language instead of YAML. However, there's an argument to be made about the security model of Helm and Kustomize. Assume that I trust that the authors of those tools are not going to smuggle my data to a malicious third party or install spyware on my computer and that they do not have critical security bugs. Then I can confidently run those tools on my computer with inputs (e.g. charts and kustomizations) that I do not trust. To say nothing of whether the tool will generate a deployment which compromises my cluster when applied. I think what I'd like to see from the community is tooling that targets Deno or WASM as the common ground. Then I can sufficiently sandbox "packages" like helm charts or kustomizations without worrying that they have general access to my computer or CI/CD runner as if I'd imported a random Python package. I've already started writing something in Deno that tries to emulate Grafana's Tanka in TypeScript. My next job is to write a tool which runs WASM-compiled KRM functions. But I might need a configuration language for that tool, so maybe I'll choose YAML...

sureglymop 3y ago

Maybe I am just daft but I really never understood why an infrastructure resource management platform needed a package manager.