4 ms·
In our experience running production workloads on k8s for over three years templating (helm) and structured editing approaches both have their place, and both a
by markbnj 7y ago
In our experience running production workloads on k8s for over three years templating (helm) and structured editing approaches both have their place, and both are valuable. We don't feel the need to replace declarative approaches with another imperative language, or to use complicated helm charts for straightforward service deployments.
There are many ways to classify workloads, but one big distinction that we find valuable is between stable infrastructure components and our own rapidly deployed services. The former have complicated configuration surfaces but change relatively rarely, while the latter are usually much simpler (because microservices) and change daily in many cases.
We find helm works very well for the infrastructure pieces. Yes it's complicated and easy to get wrong, but so are most other package management systems. Charts for complicated things can be quite dense and hard to comprehend. See the stable prometheus chart for an excellent example. But once the work is done and as long as there is commitment to maintain it the results can be pretty awesome. We use helm charts to install and upgrade prometheus, fluentd, elasticsearch, nginx, etcd and a ton of other tools. Yes we've had to fork some charts because the configuration surface wasn't fully exposed, but they are a minority.
For our own services charts are overkill. They're hard to read, and crufted up with control structures and substitution vars. Essentially all of our microservices are deployments with simple ingress requirements. We currently use kustomize to process base manifests that live in the service repos and combine them with environment-specific patches from config repos. Both are just straight yaml and very easy for backend developers to read and understand, and different groups (i.e. devops, sre, backend dev) can contribute patches to the config repos to control how the services are deployed.
Bottom line: if you're going all-in on kubernetes, which you really need to do to get the most benefit from it, then you're going to need more than one approach to deploying workloads.
- Aqua 7y agoTotally agree. Helm is difficult enough to get up to speed with for engineers without prior experience with Kubernetes. Anything more complicated (and ksonnet was significantly more complicated, which is one of the reasons why it died out) is just not scalable. Helm is good enough and brings a decent balance between usability and complexity.
- sandstrom 7y agoFor anyone curious, this is what the prometheus chart look like (I think): https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml https://github.com/helm/charts/blob/master/stable/prometheus...
- markbnj 7y agoThat's the values file. The actual manifest templates are just as horrifyingly impressive :). https://github.com/helm/charts/tree/master/stable/prometheus/templates https://github.com/helm/charts/tree/master/stable/prometheus...
- ithkuil 7y agoOne of the main problems with helm IMO is that it effectively turns the values.yaml file in a structure that almost mirrors the manifests themselves, and the biggest problem of that is hidden in the word "almost": there are many subtle ways this mapping can be done, including subtle or not so subtle field naming/spelling differences.
- markbnj 7y agoYeah absolutely, the multiple layers of indirection you can end up with are frustrating for back end engineers trying to understand how a thing got deployed the way it did. For our stuff just patching the environment in as a yaml block is way more comprehensible. But then when you want to let people one-line a complete elasticsearch cluster install helm has that power if you're willing to commit to it.
- ithkuil 7y agoI found my personal sweet spot in kubecfg, which allows me to override the fields I need (without having to know which one advance) and when needed it can do more advanced stuff such as variables, expressions, loops, etc which allow one-line complete elastic search cluster-like scenarios