4 ms·

Helm is problematic in my view. To new Kubernetes users, Helm appears to be the "Official" package manager, and the "right way to do it". It's not. Many helm

Helm is problematic in my view. To new Kubernetes users, Helm appears to be the "Official" package manager, and the "right way to do it". It's not. Many helm charts are out of date, have subtle bugs, and are not actively maintained. Upstream projects often have their own way of customizing and rendering Manifests. The biggest flaw in Helm is the decision to use text-templating for semantic data. This leads to ridiculous situations like having to specify the number of spaces to indent a block, so the output YAML can be parsed properly. Promising alternatives are: * Jsonnet [0] See kube-prometheus [1] for a complex project that uses Jsonnet to accomplish things that would be much more difficult using pre-defined templating. * Cue [2] from the creator of BCL/GCL (Borg/Google Config Lang), which is what Jsonnet is essentially. He's written quite a bit about Cue and how he intends it to address the shortcomings of Jsonnet. [0] https://jsonnet.org/articles/kubernetes.html https://jsonnet.org/articles/kubernetes.html [1] https://github.com/coreos/kube-prometheus https://github.com/coreos/kube-prometheus [2] https://cuelang.org/docs/about/ https://cuelang.org/docs/about/ [3]

bryanlarsen 7y ago
I'm a fan of jkcfg[1]. No crappy text templating, no having to learn a new language. 1: https://jkcfg.github.io/#/ https://jkcfg.github.io/#/
oalessandr 7y ago
There is also https://dhall-lang.org/ https://dhall-lang.org/
mfer 7y ago
Two things to consider... First, there has been a lot of talk about how much work it is to onboard to Kubernetes. There have even been recent discussions in Kubernetes SIGs. In UX terms, Kubernetes has a high barrier to entry. I realize that many people who use and know Kubernetes like it. We folks are a group that has gotten past the high barrier to entry. Most have not. Second, Helm is a package manager. It's like apt, yum, homebrew, and others. It lets someone who knows how to run an application (ops business logic) on a platform (like debian or k8s) and package it up so someone else who doesn't know this can easily run an application. This is why so many use brew and apt to install something rather than install everything by hand. Something like jsonnet doesn't solve for these situations. I know some folks don't like text-templating. I'm not a big fan of go templating. But, templating is something that people are used to and generally comfortable with. This means it has a fairly low barrier to entry. Notice, a lot of the reasons for why Helm does things is to reduce friction for a large audience. To reduce the barrier to entry.
- atomicity 7y ago
  Yeah, the core reason to use Helm is for the ecosystem. The value that comes from being able to quickly set up and compose packages on https://github.com/helm/charts/ https://github.com/helm/charts/ is larger than the (very) annoying YAML templating and chart reliability issues. Lots of Helm charts have bugs, but lots of PyPI/NPM/Maven packages have bugs as well. Jsonnet, Cue, and operators are nice, but unless someone can create an ecosystem that is as valuable as the Helm ecosystem, I don't see Helm disappearing.
- bshipp 7y ago
  This resonated with my recent experience. I have been using Docker and docker-compose quite comfortably for a few years now. I thought it'd try and dip toes into Kubernetes to extend my containerization skill-set and learn something new. I ended up shelving it until I have time to take a proper course and dedicate the time to it. Docker compose was very easy to learn for me, with a few easy early "wins" and reasonably comprehensive and helpful documentation when I wanted to expand into new areas. For some reason, I really struggled grokking how to structure a k8 chart. Maybe I'll try again after a good night's sleep.
  - curryst 7y ago
    Have you tried plain manifests? Unless you need to support a lot of permutations (because of many environments, or you're distributing it and need to support many deployment options), Charts are generally a suboptimal way to get things running in k8s. The major "wins" for using charts are usually around being able to use the same set of manifests with different values interpolated in. Which in turn, is really only valuable if you're supporting a large number of permutations that need to be kept in sync.
    - bshipp 7y ago
      Thanks very much for the suggestion! I really didn't know where to start and the first few sites seemed to dive right into charts but I'll check out manifests this time. Thanks again.
dcolkitt 7y ago
I think all of this is indicative of a deeper problem, which is YAML is a great config but doesn't work when you need code. When you're doing simpler deployments then config files are great. It keeps the learning curve flat for new users. The problem is that as users get more experienced, they ramp up the complexity, then start trying to shoehorn more abstraction into a format where it doesn't fit. The most visible example is the need for code reuse. Any sufficiently complex artifact is going to have a lot of similar sub-components. What you want is a macro or subroutine that translates simple parameters into common patterns. What you end up with is a lot of brittle copy-pasting. Text templating is a kludge that usually works fine until it doesn't. At the end of the day everything ends up messily converging to Greenspun's Tenth Law.
AaronFriel 7y ago
I've been using Pulumi to actually do the templating and deployment together, which I think works quite well. It supports Kubernetes, but also quite a lot more!
SrslyJosh 7y ago
> The biggest flaw in Helm is the decision to use text-templating for semantic data. This leads to ridiculous situations like having to specify the number of spaces to indent a block, so the output YAML can be parsed properly. My mind was blown when I saw this the first time. Let's hope that the developers are planning on supporting alternate templating schemes.
- ivan4th 7y ago
  The complaints about Lisp parentheses sound especially funny in the context of that kind of mainstream DevOps tooling https://github.com/helm/charts/blob/eceb82f6308578b824d58f80dd8c28cd18d31c60/stable/influxdb/templates/deployment.yaml#L23 https://github.com/helm/charts/blob/eceb82f6308578b824d58f80... (EDIT: yep, I'm a Lisp fan and a Kubernetes guy at the same time, and this kind of templating is like worst kind of "blasphemy" for me :)
  - p_l 7y ago
    Same here, and with Helm 2 it wouldn't even return an error, just silently fail with a timeout... Spent a week debugging that...
Cedricgc 7y ago
I agree, a package manager is designed for installing a graph of versioned (effectively immutable) artifacts like a binary or code package. Kubernetes is built to run services, which are not static but continuously running and react to inputs. Services need monitoring and self-healing to ensure they are healthy and operational. In the future I predict we are moving towards a model where we have ways expressively define configuration (CUE) locally which we push to Kubernetes operators [0] which do the heavy lifting to deploy, monitor, and manage/upgrade a defined service that the operator is responsible for. [0] https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...
joseph 7y ago
The worst aspect of Helm is that is is not idempotent. If you "upgrade" a release, it always rewrites the config map that the release is stored in, even if there are no changes. The config maps also get corrupted if there is an interruption during a deploy. Ansible does an admirable job of being a better Helm. It's too bad there isn't a community for Ansible roles for use on Kubernetes instead of Helm charts IMO. It's more powerful and provides the same functionality with better behavior.
- bacongobbler 7y ago
  I highly suggest giving Helm 3 a try, as much of the release ledger system was overhauled. If you can provide links to the issues you're seeing, that's always helpful too.
  - joseph 7y ago
    Oh, I probably will give it a try. Helm is pretty much inescapable since the community has rallied around using it for distributing manifests. It's just sad to need yet another bespoke tool when the general purpose ones do a fine job.
ithkuil 7y ago
I also don't like Helm for these and more reasons, and thus I often have the wrong personal incentives to point out that the yaml indentation issue can be solved in a trivial way: yaml 1.2 is (for all practical purposes) a superset of JSON, thus all you have to do when you emit a bit of yaml from a template is to pipe it through the "to json" function and the resulting string is valid YAML yet independent of the indentation of the insertion point.
- joseph 7y ago
  Exactly right. I do the same when dealing with YAML in general.
pacala 7y ago
Cue is intriguing. The flexible typing mechanism is very powerful. Alas, the documentation is lacking. There is quite a bit of musing around the type/value lattice semantic and how the core operation is "<set of nodes>: <constraints>". Then the first realistic example [0] introduces a whole bunch of new features that are neither explained, nor motivated. * Templates: deployment <Name>: .... * List comprehensions: [ "-\(k)=\(v)" for k, v in arg ]. * Conditionals: targetPort: Port if port != Port * Special templates: expose port <N>: .... What does 'expose' mean? Doesn't look like folded struct. * Some form of comprehension/conditional composition: for k, spec in deployment if len(spec.expose.port) > 0. Can we nest two comprehensions? Can we zip? Can we reduce? * Side note: What are the debug capabilities? One major usability issue with GCL circa 2010 was that debugging required sifting through MB [or was it GB?] of intermediate struct expansions. * Side note: I wish they used Python f-string syntax: "-{k}={v}" instead of "-\(k)=\(v)". [0] https://github.com/cuelang/cue/blob/e5d8d09b3ba2e4f48c84a3b5da2acbf0716379bf/doc/tutorial/kubernetes/manual/services/cloud.cue https://github.com/cuelang/cue/blob/e5d8d09b3ba2e4f48c84a3b5...

bryanlarsen 7y ago

I'm a fan of jkcfg[1]. No crappy text templating, no having to learn a new language. 1: https://jkcfg.github.io/#/ https://jkcfg.github.io/#/

oalessandr 7y ago

There is also https://dhall-lang.org/ https://dhall-lang.org/

mfer 7y ago

Two things to consider... First, there has been a lot of talk about how much work it is to onboard to Kubernetes. There have even been recent discussions in Kubernetes SIGs. In UX terms, Kubernetes has a high barrier to entry. I realize that many people who use and know Kubernetes like it. We folks are a group that has gotten past the high barrier to entry. Most have not. Second, Helm is a package manager. It's like apt, yum, homebrew, and others. It lets someone who knows how to run an application (ops business logic) on a platform (like debian or k8s) and package it up so someone else who doesn't know this can easily run an application. This is why so many use brew and apt to install something rather than install everything by hand. Something like jsonnet doesn't solve for these situations. I know some folks don't like text-templating. I'm not a big fan of go templating. But, templating is something that people are used to and generally comfortable with. This means it has a fairly low barrier to entry. Notice, a lot of the reasons for why Helm does things is to reduce friction for a large audience. To reduce the barrier to entry.

atomicity 7y ago
Yeah, the core reason to use Helm is for the ecosystem. The value that comes from being able to quickly set up and compose packages on https://github.com/helm/charts/ https://github.com/helm/charts/ is larger than the (very) annoying YAML templating and chart reliability issues. Lots of Helm charts have bugs, but lots of PyPI/NPM/Maven packages have bugs as well. Jsonnet, Cue, and operators are nice, but unless someone can create an ecosystem that is as valuable as the Helm ecosystem, I don't see Helm disappearing.
bshipp 7y ago
This resonated with my recent experience. I have been using Docker and docker-compose quite comfortably for a few years now. I thought it'd try and dip toes into Kubernetes to extend my containerization skill-set and learn something new. I ended up shelving it until I have time to take a proper course and dedicate the time to it. Docker compose was very easy to learn for me, with a few easy early "wins" and reasonably comprehensive and helpful documentation when I wanted to expand into new areas. For some reason, I really struggled grokking how to structure a k8 chart. Maybe I'll try again after a good night's sleep.
- curryst 7y ago
  Have you tried plain manifests? Unless you need to support a lot of permutations (because of many environments, or you're distributing it and need to support many deployment options), Charts are generally a suboptimal way to get things running in k8s. The major "wins" for using charts are usually around being able to use the same set of manifests with different values interpolated in. Which in turn, is really only valuable if you're supporting a large number of permutations that need to be kept in sync.
  - bshipp 7y ago
    Thanks very much for the suggestion! I really didn't know where to start and the first few sites seemed to dive right into charts but I'll check out manifests this time. Thanks again.

dcolkitt 7y ago

I think all of this is indicative of a deeper problem, which is YAML is a great config but doesn't work when you need code. When you're doing simpler deployments then config files are great. It keeps the learning curve flat for new users. The problem is that as users get more experienced, they ramp up the complexity, then start trying to shoehorn more abstraction into a format where it doesn't fit. The most visible example is the need for code reuse. Any sufficiently complex artifact is going to have a lot of similar sub-components. What you want is a macro or subroutine that translates simple parameters into common patterns. What you end up with is a lot of brittle copy-pasting. Text templating is a kludge that usually works fine until it doesn't. At the end of the day everything ends up messily converging to Greenspun's Tenth Law.

AaronFriel 7y ago

I've been using Pulumi to actually do the templating and deployment together, which I think works quite well. It supports Kubernetes, but also quite a lot more!

SrslyJosh 7y ago

> The biggest flaw in Helm is the decision to use text-templating for semantic data. This leads to ridiculous situations like having to specify the number of spaces to indent a block, so the output YAML can be parsed properly. My mind was blown when I saw this the first time. Let's hope that the developers are planning on supporting alternate templating schemes.

ivan4th 7y ago
The complaints about Lisp parentheses sound especially funny in the context of that kind of mainstream DevOps tooling https://github.com/helm/charts/blob/eceb82f6308578b824d58f80dd8c28cd18d31c60/stable/influxdb/templates/deployment.yaml#L23 https://github.com/helm/charts/blob/eceb82f6308578b824d58f80... (EDIT: yep, I'm a Lisp fan and a Kubernetes guy at the same time, and this kind of templating is like worst kind of "blasphemy" for me :)
- p_l 7y ago
  Same here, and with Helm 2 it wouldn't even return an error, just silently fail with a timeout... Spent a week debugging that...

Cedricgc 7y ago

I agree, a package manager is designed for installing a graph of versioned (effectively immutable) artifacts like a binary or code package. Kubernetes is built to run services, which are not static but continuously running and react to inputs. Services need monitoring and self-healing to ensure they are healthy and operational. In the future I predict we are moving towards a model where we have ways expressively define configuration (CUE) locally which we push to Kubernetes operators [0] which do the heavy lifting to deploy, monitor, and manage/upgrade a defined service that the operator is responsible for. [0] https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ https://kubernetes.io/docs/concepts/extend-kubernetes/operat...

joseph 7y ago

The worst aspect of Helm is that is is not idempotent. If you "upgrade" a release, it always rewrites the config map that the release is stored in, even if there are no changes. The config maps also get corrupted if there is an interruption during a deploy. Ansible does an admirable job of being a better Helm. It's too bad there isn't a community for Ansible roles for use on Kubernetes instead of Helm charts IMO. It's more powerful and provides the same functionality with better behavior.

bacongobbler 7y ago
I highly suggest giving Helm 3 a try, as much of the release ledger system was overhauled. If you can provide links to the issues you're seeing, that's always helpful too.
- joseph 7y ago
  Oh, I probably will give it a try. Helm is pretty much inescapable since the community has rallied around using it for distributing manifests. It's just sad to need yet another bespoke tool when the general purpose ones do a fine job.

ithkuil 7y ago

I also don't like Helm for these and more reasons, and thus I often have the wrong personal incentives to point out that the yaml indentation issue can be solved in a trivial way: yaml 1.2 is (for all practical purposes) a superset of JSON, thus all you have to do when you emit a bit of yaml from a template is to pipe it through the "to json" function and the resulting string is valid YAML yet independent of the indentation of the insertion point.

joseph 7y ago
Exactly right. I do the same when dealing with YAML in general.

pacala 7y ago

Cue is intriguing. The flexible typing mechanism is very powerful. Alas, the documentation is lacking. There is quite a bit of musing around the type/value lattice semantic and how the core operation is "<set of nodes>: <constraints>". Then the first realistic example [0] introduces a whole bunch of new features that are neither explained, nor motivated. * Templates: deployment <Name>: .... * List comprehensions: [ "-\(k)=\(v)" for k, v in arg ]. * Conditionals: targetPort: Port if port != Port * Special templates: expose port <N>: .... What does 'expose' mean? Doesn't look like folded struct. * Some form of comprehension/conditional composition: for k, spec in deployment if len(spec.expose.port) > 0. Can we nest two comprehensions? Can we zip? Can we reduce? * Side note: What are the debug capabilities? One major usability issue with GCL circa 2010 was that debugging required sifting through MB [or was it GB?] of intermediate struct expansions. * Side note: I wish they used Python f-string syntax: "-{k}={v}" instead of "-\(k)=\(v)". [0] https://github.com/cuelang/cue/blob/e5d8d09b3ba2e4f48c84a3b5da2acbf0716379bf/doc/tutorial/kubernetes/manual/services/cloud.cue https://github.com/cuelang/cue/blob/e5d8d09b3ba2e4f48c84a3b5...