7 ms·

Vanilla k8s is pretty good. But once the 8 trillion vendors have you 'curl | helm' ing you end up with a knot of a system. Keep it simple, use GitOps (ArgoCD i

Vanilla k8s is pretty good. But once the 8 trillion vendors have you 'curl | helm' ing you end up with a knot of a system. Keep it simple, use GitOps (ArgoCD is great), let k8s do what it's good at, managing workloads, not as a delivery mechanism for a vendor. As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template.

p_l 4y ago
I judiciously delete any and all helm I can find, and fight text-based templating where possible :/ Especially since k8s doesn't even use "text" for the format, it's just happens that JSON is the serialization, use datastructures and serialize them dammit :/
bushbaba 4y ago
I'm fairly convinced that helm in production is an anti-pattern. With you instead having all K8s manifests checked into your Git repository with CI/CD to handle any changes. Helm just has too much auto-magic and remove's one of k8s best features, the git-blame / git-diff for Infrastructure.
- blamarvt 4y ago
  I don't understand why this isn't the prevailing philosophy. I'm over here Terraforming helm deployments and having no idea what is actually happening under the covers.
  - cortesoft 4y ago
    We have our CI run a helm —-dry-run —-debug before the deploy step, so we can see the exact kubernetes resources that are going to be applied. You can even save that as an artifact so you can diff it later.
    - a012 4y ago
      There's Helm plugin (https://github.com/databus23/helm-diff https://github.com/databus23/helm-diff) that show diff results for you, for example helm diff upgrade <release> <repo> --namespace <namespace>
- the_duke 4y ago
  Helm still saves you an incredible amount of work for setting up all the third party services. If your team is big enough you can just write your own configs, but that takes a lot of time and often quite a bit of knowledge about the relevant service. Rollbacks and resource cleanup is not to be underestimated either if you are not getting that from other tools like Argo. Note: You can still use the Git-centric approach by generating configuration via `helm template`.
  - 0x457 4y ago
    I'm convinced that the only reason Helm saves you time with 3rd party services is because 3rd party services only provide Helm chart or don't provide means of deployment to k8s at all. I've been using kustomize to deploy some things with ArgoCD and it's so much easier. Now, I'm trying to never use helm for 3rd party. However, for your internal things, Helm is hard to replace. It's easy to start a chart that is capable of deploying most of your internal services, but maintaining it is a nightmare. Actually, using helm, as in `helm` the binary, directly sounds bonkers to me and I wouldn't wish this upon anyone.
- FridgeSeal 4y ago
  I use Helm only for installing “supporting applications” -elastic searchoperator, jupyterHub, etc. Our normal deployments are standard K8s configs. These apps use Helm because a lot of the settings are complicated, Co dependent, etc. Absolutely would not write helm charts from scratch for normal deployments, and if I got these apps in a better format than helm, I’d probably drop it immediately.
  - bottled_poe 4y ago
    It’s the fact that they are complicated which warrants manual configuration. If the org can’t write the configuration, how will they support it when something goes wrong? It’s a problem waiting to happen.
  - FridgeSeal 4y ago
    I can't edit this anymore - but to anyone reading, I ought to specify that I do _not_ use Helm directly, only via Flux, our CD tool. It ablates a lot of the issues of dealing with helm charts.
halfmatthalfcat 4y ago
I don't understand this comment. How else are you going to deploy pieces of k8s infra into k8s if not with Helm and Helm Charts? Sure, you can use Argo to deploy and sync Helm charts into k8s but...you're still going to be using Helm (if not indirectly via Argo) and you will inevitably need to template things that need to be dynamically configured at render-time.
- justinsaccount 4y ago
  See https://xeiaso.net/blog/dhall-kubernetes-2020-01-25 https://xeiaso.net/blog/dhall-kubernetes-2020-01-25
  - halfmatthalfcat 4y ago
    So instead of Go Templates, I'm going to use Dhall? Why? I'd be losing interop with the entire Helm ecosystem too, so there go dependencies and such to maintain my IaC. That blog post doesn't alleviate any issues one might have using the traditional Go Templates + Helm.
- qbasic_forever 4y ago
  Just kubectl apply the manifests. You can even use kubectl -k for the Kustomize configuration engine that can more or less replace most of what helm does today.
  - halfmatthalfcat 4y ago
    So what, I'm going to have a big Makefile or something with a bunch of kubectl applies? For each environment too? What if one of my dependencies (cert-manager for example) doesn't support directly applying via kubectl but have to be rendered with Helm? How do I manage versions of these dependencies too? For better or for worse Helm is the defacto standard for deploying into k8s. Kustomizing toy apps or simple things may work but I have yet to see a large production stack use anything but Helm.
    - 4y ago
n3t 4y ago
> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. This. My first thought when I saw the indentation hack was "it can't be serious, production-ready software". My take on this is as follows. If you have a simple use case, write your K8s manifests directly. If you have a complex use case, Helm is often more pain than it's worth. Use alternatives, for example Jsonnet[0] with kubecfg[1]. Or emit manifests from your language of choice. Just don't use Helm. [0]: https://jsonnet.org/ https://jsonnet.org/ [1]: https://github.com/kubecfg/kubecfg https://github.com/kubecfg/kubecfg
- 015a 4y ago
  For a complex use case: cdk8s is cool https://cdk8s.io/ https://cdk8s.io/
- vlunkr 4y ago
  It is shocking that such a clearly bad design choice has stuck, and that Helm has become so popular in spite of it. I've had my eye on jsonnet, I'll have to try that next time I do something in k8s.
  - runevault 4y ago
    We are using helm at work but I've never touched it so I cannot comment on it being bad. However cargo cult habits have made quite a few technologies over the years take off when they never should have just because someone with either the right sort of intelligence and/or charisma made people believe it was good, and then the ripple effect made it take off.
    - nfhshy68 4y ago
      I had someone on here assume we were a small dev shop because we don't use microservices the other day. I don't think they were expecting 2k devs to be what I responded with. Apparently monoliths a scale a lot bigger than you'd think.
  - 0xbadcafebee 4y ago
topspin 4y ago
> not as a delivery mechanism for a vendor Amen. I got turned off from k8s following a tutorial that used Helm. I ran it and watched mysteries and line noise scroll past and walked away for a year. I thought "no, I will never put this squirming mass of tapeworms in front of anyone." Then I took up with k3s and got underway.
jrockway 4y ago
> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. Yeah, I think helm will be the death of Kubernetes. Some other workload management tool will come out, it will have a computer-science-inspired templating system (think Lisp macros, not C #defines) that is also easy to use, and the increased maintainability will be a breath of fresh air to people in helm hell and they'll switch over, even if the workload management fundamentals are worse. It is a shame that ksonnet was abandoned. jsonnet is a very pleasant programming language for problems of similar scope. I think that people have to see some adoption or they give up; so Helm stays motivated to continue existing despite the fact that the design is fundamentally flawed, while alternatives give up when nobody uses them. If you're looking for a lesson in "worse is better", look no farther from helm. Easy things are hard, but everything is possible. That turns out to be important. I also liked kustomize a lot, but it definitely sacrificed features for security and a clean overall design posture. I don't really know what it's missing, but it must be missing something. (I use kustomize for everything, but it seems rare in the industry. And I don't use it for complicated things like "make a namespace and app deployment for every PR"; I think to manage things like that it's missing features people want. I say just run your development branch on your workstation and commit it when it works. Running the app you write shouldn't be rocket science, and so a computer program that configures it shouldn't be necessary. The industry definitely disagrees with me there, though.) One of the biggest problems I have with helm is that it's not easy to make local modifications to charts. That needs to be a first class feature, and something everyone knows how to use. As a vendor who ships a helm chart, I feel like almost all of my work these days is hearing from users "our k8s security team requires that every manifest set field X", and then I have to go add that for them and release it. If kustomize were the default, they'd just add that themselves instead of asking. But hey, opportunity to provide good customer service, I guess. (Luckily, most of the requests are compatible with each other. At one point we ran as root because that's the default, a customer required running as a user, and now everyone can run a locked down version; nobody NEEDED to run as root. So these requests generally improve the product, but it's a little tedious to make them ask.)
- moondev 4y ago
  kustomize supports using a helm chart as a resource https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/ https://kubectl.docs.kubernetes.io/references/kustomize/kust...
whalesalad 4y ago
I really think this is why most people dislike and misunderstand the value of kube. If you raw dog it, it’s pretty amazing what you can build. It’s not very hard to roll your own Heroku (tailored to your workflows and workloads) with kube, if you shy away from all the noise of helm and other vendors as you say.

p_l 4y ago

I judiciously delete any and all helm I can find, and fight text-based templating where possible :/ Especially since k8s doesn't even use "text" for the format, it's just happens that JSON is the serialization, use datastructures and serialize them dammit :/

bushbaba 4y ago

I'm fairly convinced that helm in production is an anti-pattern. With you instead having all K8s manifests checked into your Git repository with CI/CD to handle any changes. Helm just has too much auto-magic and remove's one of k8s best features, the git-blame / git-diff for Infrastructure.

blamarvt 4y ago
I don't understand why this isn't the prevailing philosophy. I'm over here Terraforming helm deployments and having no idea what is actually happening under the covers.
- cortesoft 4y ago
  We have our CI run a helm —-dry-run —-debug before the deploy step, so we can see the exact kubernetes resources that are going to be applied. You can even save that as an artifact so you can diff it later.
  - a012 4y ago
    There's Helm plugin (https://github.com/databus23/helm-diff https://github.com/databus23/helm-diff) that show diff results for you, for example helm diff upgrade <release> <repo> --namespace <namespace>
the_duke 4y ago
Helm still saves you an incredible amount of work for setting up all the third party services. If your team is big enough you can just write your own configs, but that takes a lot of time and often quite a bit of knowledge about the relevant service. Rollbacks and resource cleanup is not to be underestimated either if you are not getting that from other tools like Argo. Note: You can still use the Git-centric approach by generating configuration via `helm template`.
- 0x457 4y ago
  I'm convinced that the only reason Helm saves you time with 3rd party services is because 3rd party services only provide Helm chart or don't provide means of deployment to k8s at all. I've been using kustomize to deploy some things with ArgoCD and it's so much easier. Now, I'm trying to never use helm for 3rd party. However, for your internal things, Helm is hard to replace. It's easy to start a chart that is capable of deploying most of your internal services, but maintaining it is a nightmare. Actually, using helm, as in `helm` the binary, directly sounds bonkers to me and I wouldn't wish this upon anyone.
FridgeSeal 4y ago
I use Helm only for installing “supporting applications” -elastic searchoperator, jupyterHub, etc. Our normal deployments are standard K8s configs. These apps use Helm because a lot of the settings are complicated, Co dependent, etc. Absolutely would not write helm charts from scratch for normal deployments, and if I got these apps in a better format than helm, I’d probably drop it immediately.
- bottled_poe 4y ago
  It’s the fact that they are complicated which warrants manual configuration. If the org can’t write the configuration, how will they support it when something goes wrong? It’s a problem waiting to happen.
- FridgeSeal 4y ago
  I can't edit this anymore - but to anyone reading, I ought to specify that I do _not_ use Helm directly, only via Flux, our CD tool. It ablates a lot of the issues of dealing with helm charts.

halfmatthalfcat 4y ago

I don't understand this comment. How else are you going to deploy pieces of k8s infra into k8s if not with Helm and Helm Charts? Sure, you can use Argo to deploy and sync Helm charts into k8s but...you're still going to be using Helm (if not indirectly via Argo) and you will inevitably need to template things that need to be dynamically configured at render-time.

justinsaccount 4y ago
See https://xeiaso.net/blog/dhall-kubernetes-2020-01-25 https://xeiaso.net/blog/dhall-kubernetes-2020-01-25
- halfmatthalfcat 4y ago
  So instead of Go Templates, I'm going to use Dhall? Why? I'd be losing interop with the entire Helm ecosystem too, so there go dependencies and such to maintain my IaC. That blog post doesn't alleviate any issues one might have using the traditional Go Templates + Helm.
qbasic_forever 4y ago
Just kubectl apply the manifests. You can even use kubectl -k for the Kustomize configuration engine that can more or less replace most of what helm does today.
- halfmatthalfcat 4y ago
  So what, I'm going to have a big Makefile or something with a bunch of kubectl applies? For each environment too? What if one of my dependencies (cert-manager for example) doesn't support directly applying via kubectl but have to be rendered with Helm? How do I manage versions of these dependencies too? For better or for worse Helm is the defacto standard for deploying into k8s. Kustomizing toy apps or simple things may work but I have yet to see a large production stack use anything but Helm.
  - 4y ago

n3t 4y ago

> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. This. My first thought when I saw the indentation hack was "it can't be serious, production-ready software". My take on this is as follows. If you have a simple use case, write your K8s manifests directly. If you have a complex use case, Helm is often more pain than it's worth. Use alternatives, for example Jsonnet[0] with kubecfg[1]. Or emit manifests from your language of choice. Just don't use Helm. [0]: https://jsonnet.org/ https://jsonnet.org/ [1]: https://github.com/kubecfg/kubecfg https://github.com/kubecfg/kubecfg

015a 4y ago
For a complex use case: cdk8s is cool https://cdk8s.io/ https://cdk8s.io/
vlunkr 4y ago
It is shocking that such a clearly bad design choice has stuck, and that Helm has become so popular in spite of it. I've had my eye on jsonnet, I'll have to try that next time I do something in k8s.
- runevault 4y ago
  We are using helm at work but I've never touched it so I cannot comment on it being bad. However cargo cult habits have made quite a few technologies over the years take off when they never should have just because someone with either the right sort of intelligence and/or charisma made people believe it was good, and then the ripple effect made it take off.
  - nfhshy68 4y ago
    I had someone on here assume we were a small dev shop because we don't use microservices the other day. I don't think they were expecting 2k devs to be what I responded with. Apparently monoliths a scale a lot bigger than you'd think.
- 0xbadcafebee 4y ago

topspin 4y ago

> not as a delivery mechanism for a vendor Amen. I got turned off from k8s following a tutorial that used Helm. I ran it and watched mysteries and line noise scroll past and walked away for a year. I thought "no, I will never put this squirming mass of tapeworms in front of anyone." Then I took up with k3s and got underway.

jrockway 4y ago

> As an aside, the existence of the '{{ | indent 4 }}' function in helm should disqualify it from any serious use. Render, don't template. Yeah, I think helm will be the death of Kubernetes. Some other workload management tool will come out, it will have a computer-science-inspired templating system (think Lisp macros, not C #defines) that is also easy to use, and the increased maintainability will be a breath of fresh air to people in helm hell and they'll switch over, even if the workload management fundamentals are worse. It is a shame that ksonnet was abandoned. jsonnet is a very pleasant programming language for problems of similar scope. I think that people have to see some adoption or they give up; so Helm stays motivated to continue existing despite the fact that the design is fundamentally flawed, while alternatives give up when nobody uses them. If you're looking for a lesson in "worse is better", look no farther from helm. Easy things are hard, but everything is possible. That turns out to be important. I also liked kustomize a lot, but it definitely sacrificed features for security and a clean overall design posture. I don't really know what it's missing, but it must be missing something. (I use kustomize for everything, but it seems rare in the industry. And I don't use it for complicated things like "make a namespace and app deployment for every PR"; I think to manage things like that it's missing features people want. I say just run your development branch on your workstation and commit it when it works. Running the app you write shouldn't be rocket science, and so a computer program that configures it shouldn't be necessary. The industry definitely disagrees with me there, though.) One of the biggest problems I have with helm is that it's not easy to make local modifications to charts. That needs to be a first class feature, and something everyone knows how to use. As a vendor who ships a helm chart, I feel like almost all of my work these days is hearing from users "our k8s security team requires that every manifest set field X", and then I have to go add that for them and release it. If kustomize were the default, they'd just add that themselves instead of asking. But hey, opportunity to provide good customer service, I guess. (Luckily, most of the requests are compatible with each other. At one point we ran as root because that's the default, a customer required running as a user, and now everyone can run a locked down version; nobody NEEDED to run as root. So these requests generally improve the product, but it's a little tedious to make them ask.)

moondev 4y ago
kustomize supports using a helm chart as a resource https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/ https://kubectl.docs.kubernetes.io/references/kustomize/kust...

whalesalad 4y ago

I really think this is why most people dislike and misunderstand the value of kube. If you raw dog it, it’s pretty amazing what you can build. It’s not very hard to roll your own Heroku (tailored to your workflows and workloads) with kube, if you shy away from all the noise of helm and other vendors as you say.