7 ms·

I think helm is at it's best when you need to _publicly distribute_ a complex application to a large number of people in a way that's configurable through param

I think helm is at it's best when you need to _publicly distribute_ a complex application to a large number of people in a way that's configurable through parameters. For internal applications, it's in an awkward place of being both too complex and too simple, and in a lot of cases what you really want to do is just write your own operator for the complex cases and use kustomize for the simple cases. Most of the problems with updating and installing helm charts go away if you manage it with something like argocd to automatically keep everything up to date.

numbsafari 3y ago
Personally much prefer kustomize for the “ship an app” business. Probably even better is to ship a controller and a CRD for the config. Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.
- mountainriver 3y ago
  Controller + CRD is the way to go and seems more in line with how k8s was intended to be used. The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
  - arccy 3y ago
    operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
    - mountainriver 3y ago
      Fair enough, the UX is just so much better that I'd gamble it in most use cases
- cortesoft 3y ago
  We switched from kustomize to helm and I really can't understand why anyone would prefer kustomize. Having the weird syntax for replacing things, having to look at a bunch of different files to see what is going on... I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it. People complain about it being a template language, but that is exactly what you need!
  - Hamuko 3y ago
    >Having the weird syntax for replacing things Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard? >having to look at a bunch of different files to see what is going on I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.
    - cortesoft 3y ago
      By "weird syntax" I mean stuff like "patchesJson6902" or "configMapGenerator" or "patchesStrategicMerge" where you have to know what each field means and how they work. A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.
      - ahoka 3y ago
        Do you have any really complex Helm template? It either becomes an unreadable mess or you end up with basically manifest snippets in your values file.
      - miiiiiike 3y ago
- morelisp 3y ago
  > Probably even better is to ship a controller and a CRD for the config. Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)
  - jen20 3y ago
    If you run a multi-tenant Kubernetes cluster at scale, operators with poor discipline spamming the API servers and taking etcd down is a leading cause of sadness.
    - morelisp 3y ago
      This is the common view among our ops team, sure, but for a vocation so prima facie obsessed with postmortems/five-whys/root-causes/etc it's depressingly shallow.
  - cassianoleal 3y ago
    Generally speaking, operators and CRDs are more in the domain of your platform rather than your products. They should provide common interfaces to implement the business requirements around things like uptime, HA, healthchecking, observability, etc. If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach. As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.
- jpdb 3y ago
  > Probably even better is to ship a controller and a CRD for the config. But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.
  - numbsafari 3y ago
    Just use kustomize. It’s baked into kubectl. No need for a separate tool.
    - cassianoleal 3y ago
      Agree. I'd recommend to start with static YAML though. Use kustomize for the very few customisations required for, say, different environments. Keep them to a minimum - there's no reason for a controller's deployment to vary too much - they're usually deployed once per cluster.
evancordell 3y ago
This is interesting, I have the opposite opinion. I dislike helm for public distribution, because everyone wants _their_ thing templated, so you end up making every field of your chart templated and it becomes a mess to maintain. Internal applications don't have this problem, so you can easily keep your chart interface simple and scoped to the different ways you need to deploy your own stack. With Kustomize, you just publish the base manifests and users can override whatever they want. Not that Kustomize doesn't have its own set of problems.
- moondev 3y ago
  Kustomize also supports helm charts as a "resource" which makes it handy to do last mile modifications of values and 'non value exposed" items without touching or forking the upstream chart.
- preisschild 3y ago
  There are good common libraries which expose every property by default, so you dont need to make everything template-able yourself https://github.com/bjw-s/helm-charts/tree/main/charts/library/common https://github.com/bjw-s/helm-charts/tree/main/charts/librar...
- mieubrisse 3y ago
  How would you feel if you could use Starlark (if you're familiar with it) to parameterize a la Helm, and then can add more Starlark commande later to update previously-defined infra a la Kustomize? Full disclosure: our startup is trying to build a tool where you don't have to pick, so trying to test the hypothesis
jen20 3y ago
The need to use something like Helm to distribute a complex application is a good indication you've built something which is a mess, and probably should be rethought from first principles. Most of the problems associated with Helm go away if you stop using Kubernetes.
- imglorp 3y ago
  Vendors shipping things for customers to run in their clouds and prems have a very limited set of common denominators. When you add in requirements like workload scaling, availability, and durability, that set is very small. So yeah we do this. Our product runs in 3 public clouds (working on 5), single VM, etc. and our customers install it themselves. We're helm plus Replicated. AMA.
  - dog321 3y ago
    When deploying into different clouds, do you require any cloud provider resources that require management with terraform etc. or is it relatively self contained? Also curious what issues you've seen replicated prevent.
    - imglorp 3y ago
      For public cloud k8s, no we don't provision or TF anything, we just shove in a manifest and k8s creates the workloads and it provisions persistent volumes and load balancers on your behalf. That's either Helm or Replicated (Kots) on top of Helm. Yes, it's basically self-contained and manages to abstract most of the cloud differences. We do have a custom storage class for each cloud but probably don't need it. The network load balancers need a little cloud specific annotation. Replicated saved work by handling a configuration gui for the end user, licensing/ entitlements, support bundle collection, private image proxy, things like that we didn't want to deal with.
  - Too 3y ago
    Once you add in workload scaling, availability and durability, there is surely a dedicated ops team that want to control every aspect of how it’s deployed, including the security around it. They are not just going to blindly apply a chart without at least having reviewed it in great detail first. What I found is that when doing such review, you realize 99 of the template variables are not relevant for you and the one place you need to template is missing a value. Just extracting the rendered manifests and modify them by hand from there becomes more maintainable. Like you say, there is a very limited set of common denominators. For smaller orgs, just running a single container and increasing the Node size takes you a very long way. That doesn’t need helm.
hinkley 3y ago
I barely have a horse in this race, but I think what I'd like to see is more apps that behave like 'npm config' or 'git config' where you can imperatively change one configuration value. I take your image as the FROM for my own Dockerfile, tweak a few settings, maybe alter the CMD, and then run my image instead of trying to do some sort of ad absurdum variation on a Twelve Factor App.
zzyzxd 3y ago
> helm is at it's best when you need to _publicly distribute_ a complex application I would say, helm is at it's best when you need to _publicly distribute_ a complex application, AND the majority of the users on the receiving end don't care about the complexity. You don't need Helm if your manifests are simple. But when your manifests become complex, helm will make it even more complex by turning each field in the plain k8s manifests into a toggle in a values.yaml file.
- pas 3y ago
  but usually even the structure is wrong, and there's no option to disable a template in Helm, so I need to install and then manually edit/replace things, or just script Helm with bash or something, which is terrible .. so the whole things is just a big ball of WTF. yes, it's nice to do env-var-substitution, and --set is not that dumb.

numbsafari 3y ago

Personally much prefer kustomize for the “ship an app” business. Probably even better is to ship a controller and a CRD for the config. Doing it that means you ship a schema for the parameters of the config, and that you have code that can handle complexities of upgrades/migrations that tools like kustomize and helm struggle or fail at altogether.

mountainriver 3y ago
Controller + CRD is the way to go and seems more in line with how k8s was intended to be used. The challenge has historically been that controllers are a lot harder to write, but I think that story has improved over the years
- arccy 3y ago
  operators are great when you control it. less so when it's some third party one that doesn't support that field you need on a resource it creates and all the customizations just end up being yaml merges from a configmap string or CRD if you're lucky
  - mountainriver 3y ago
    Fair enough, the UX is just so much better that I'd gamble it in most use cases
cortesoft 3y ago
We switched from kustomize to helm and I really can't understand why anyone would prefer kustomize. Having the weird syntax for replacing things, having to look at a bunch of different files to see what is going on... I love how in Helm I can just look at the templates and figure out what values I need to change to get what I want, and I love each environment only needing a single values file to see all the customizations for it. People complain about it being a template language, but that is exactly what you need!
- Hamuko 3y ago
  >Having the weird syntax for replacing things Isn't the "weird syntax" just either Yaml files or just JSON Patches, which is a pretty easy standard? >having to look at a bunch of different files to see what is going on I consider that a feature, not a bug. prod/larger-memory-request.yaml makes it much easier for me to see what goes into deploying the prod environment instead of for example the test environment.
  - cortesoft 3y ago
    By "weird syntax" I mean stuff like "patchesJson6902" or "configMapGenerator" or "patchesStrategicMerge" where you have to know what each field means and how they work. A template is much easier to read. I had zero experience with go templating, but was able to figure out what it all meant just by looking at the templates... they still looked like kubernetes resources As for looking at a bunch of different files, if you like having a "larger-memory-request" file, you can still do that with helm... you can use as many values files as you want, just include them in precedence order. You can have your "larger-memory-request" values file.
    - ahoka 3y ago
      Do you have any really complex Helm template? It either becomes an unreadable mess or you end up with basically manifest snippets in your values file.
    - miiiiiike 3y ago
morelisp 3y ago
> Probably even better is to ship a controller and a CRD for the config. Maybe it's just us, but our operations team puts pretty hard restrictions on how we're allowed to talk to the K8s API directly. We can turn a regular Deployment around as fast as we can write it, but if we needed a controller and CRD update it'd take us like three days minimum. (Which, I even sort of understand because I see the absolute garbage code in some of the operators the other teams are asking them to deploy...)
- jen20 3y ago
  If you run a multi-tenant Kubernetes cluster at scale, operators with poor discipline spamming the API servers and taking etcd down is a leading cause of sadness.
  - morelisp 3y ago
    This is the common view among our ops team, sure, but for a vocation so prima facie obsessed with postmortems/five-whys/root-causes/etc it's depressingly shallow.
- cassianoleal 3y ago
  Generally speaking, operators and CRDs are more in the domain of your platform rather than your products. They should provide common interfaces to implement the business requirements around things like uptime, HA, healthchecking, observability, etc. If a product team sees itself needing to deploy an operator, it's likely the platform is subpar and should be improved, or the product team is overengineering something and could do with rethinking their approach. As in most cases, a conversation with your platform/ops/devops/sre/infra team should help clarify things.
jpdb 3y ago
> Probably even better is to ship a controller and a CRD for the config. But how do you package the controller + CRD? The two leading choices are `kubectl apply -f` on a url or Helm and as soon as you need any customization to the controller itself you end up needing a tool like helm.
- numbsafari 3y ago
  Just use kustomize. It’s baked into kubectl. No need for a separate tool.
  - cassianoleal 3y ago
    Agree. I'd recommend to start with static YAML though. Use kustomize for the very few customisations required for, say, different environments. Keep them to a minimum - there's no reason for a controller's deployment to vary too much - they're usually deployed once per cluster.

evancordell 3y ago

This is interesting, I have the opposite opinion. I dislike helm for public distribution, because everyone wants _their_ thing templated, so you end up making every field of your chart templated and it becomes a mess to maintain. Internal applications don't have this problem, so you can easily keep your chart interface simple and scoped to the different ways you need to deploy your own stack. With Kustomize, you just publish the base manifests and users can override whatever they want. Not that Kustomize doesn't have its own set of problems.

moondev 3y ago
Kustomize also supports helm charts as a "resource" which makes it handy to do last mile modifications of values and 'non value exposed" items without touching or forking the upstream chart.
preisschild 3y ago
There are good common libraries which expose every property by default, so you dont need to make everything template-able yourself https://github.com/bjw-s/helm-charts/tree/main/charts/library/common https://github.com/bjw-s/helm-charts/tree/main/charts/librar...
mieubrisse 3y ago
How would you feel if you could use Starlark (if you're familiar with it) to parameterize a la Helm, and then can add more Starlark commande later to update previously-defined infra a la Kustomize? Full disclosure: our startup is trying to build a tool where you don't have to pick, so trying to test the hypothesis

jen20 3y ago

The need to use something like Helm to distribute a complex application is a good indication you've built something which is a mess, and probably should be rethought from first principles. Most of the problems associated with Helm go away if you stop using Kubernetes.

imglorp 3y ago
Vendors shipping things for customers to run in their clouds and prems have a very limited set of common denominators. When you add in requirements like workload scaling, availability, and durability, that set is very small. So yeah we do this. Our product runs in 3 public clouds (working on 5), single VM, etc. and our customers install it themselves. We're helm plus Replicated. AMA.
- dog321 3y ago
  When deploying into different clouds, do you require any cloud provider resources that require management with terraform etc. or is it relatively self contained? Also curious what issues you've seen replicated prevent.
  - imglorp 3y ago
    For public cloud k8s, no we don't provision or TF anything, we just shove in a manifest and k8s creates the workloads and it provisions persistent volumes and load balancers on your behalf. That's either Helm or Replicated (Kots) on top of Helm. Yes, it's basically self-contained and manages to abstract most of the cloud differences. We do have a custom storage class for each cloud but probably don't need it. The network load balancers need a little cloud specific annotation. Replicated saved work by handling a configuration gui for the end user, licensing/ entitlements, support bundle collection, private image proxy, things like that we didn't want to deal with.
- Too 3y ago
  Once you add in workload scaling, availability and durability, there is surely a dedicated ops team that want to control every aspect of how it’s deployed, including the security around it. They are not just going to blindly apply a chart without at least having reviewed it in great detail first. What I found is that when doing such review, you realize 99 of the template variables are not relevant for you and the one place you need to template is missing a value. Just extracting the rendered manifests and modify them by hand from there becomes more maintainable. Like you say, there is a very limited set of common denominators. For smaller orgs, just running a single container and increasing the Node size takes you a very long way. That doesn’t need helm.

hinkley 3y ago

I barely have a horse in this race, but I think what I'd like to see is more apps that behave like 'npm config' or 'git config' where you can imperatively change one configuration value. I take your image as the FROM for my own Dockerfile, tweak a few settings, maybe alter the CMD, and then run my image instead of trying to do some sort of ad absurdum variation on a Twelve Factor App.

zzyzxd 3y ago

> helm is at it's best when you need to _publicly distribute_ a complex application I would say, helm is at it's best when you need to _publicly distribute_ a complex application, AND the majority of the users on the receiving end don't care about the complexity. You don't need Helm if your manifests are simple. But when your manifests become complex, helm will make it even more complex by turning each field in the plain k8s manifests into a toggle in a values.yaml file.

pas 3y ago
but usually even the structure is wrong, and there's no option to disable a template in Helm, so I need to install and then manually edit/replace things, or just script Helm with bash or something, which is terrible .. so the whole things is just a big ball of WTF. yes, it's nice to do env-var-substitution, and --set is not that dumb.