3 ms·

In which situations would you consider using Kustomize over helm? Helm seems like a decent option for defining your app, especially if you’re using other helm d

In which situations would you consider using Kustomize over helm? Helm seems like a decent option for defining your app, especially if you’re using other helm dependencies. I’m yet to deploy anything to a production k8s cluster, so I’m looking for opinions of people that know what they’re talking about.

chupasaurus 8y ago
> Helm seems like a decent option If you want to: have PITA with RBAC, forget about manual changes because there are no 3-way-merge in Helm2 (that's what kubectl apply does, only hope it will be in Helm 3) and after this change helm cannot apply desired state, some of the biggest pains lasting for years had been resolved only in 2.13 with --atomic parameter. Aside from dependency resolution, everything that Helm does could be made by any template engine + API caller.
littlekosh 8y ago
Helm is great right up until you run into a chart that doesn’t support what you need, i.e. if you want to run on tainted nodes, it needs to support tolerations. Want to add a sidecar to a pod? No. It also lies about the actual status of the installation. A deployment that applies successfully but has issues with the new pods coming up can still show successful. The larger charts are a nightmare which isn’t helm’s fault but it doesn’t make things appreciably easier either. Kustomize has a lot of warts too. If the type definition in k8s source doesn’t have a merge key defined, it’s going to overwrite the entire section of the resource. You can use JsonPatches to get around that problem but that is super gross too. The entire ecosystem is trying to solve the same problem and no one has come up with the definitive way of doing it (because it’s a HARD problem) It seems like Helm templating and then passing to kustomize to add in the one-off changes is the direction a lot of people are heading but it’s more of a least-bad solution than a good one. I will say, if you’re using helm templates for your own stuff, you very likely going to be happy with it.
- jaxxstorm 8y ago
  This is a problem we got sick and tired of at $work, so we made kr8: https://github.com/apptio/kr8 https://github.com/apptio/kr8 You can define a component which you install into many clusters and then slightly differentiate them based on cluster parameters, kind of like Puppet or Chef (without the application stage) Alongside this, you can actually patch helm charts. An example component can be found here https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master/components/external_dns https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master... The patches.jsonnet allows you to add a commandline flag that wasn't in the helm chart at one time.
  - littlekosh 8y ago
    Oh no, jsonnet! Seriously though, gonna look closer at this tomorrow at the office. Thanks for the heads up!
- grantlmiller 8y ago
  The pattern of Helm template + Kustomize is operationalized with Ship: https://github.com/replicatedhq/ship https://github.com/replicatedhq/ship plus some other functionality to automate pull requests from upstream updates.
  - captn3m0 8y ago
    Isn't it a native usecase as well? This is linked from the kustomize website as a valid usecase: https://testingclouds.wordpress.com/2018/07/20/844/ https://testingclouds.wordpress.com/2018/07/20/844/ (Title = Customizing Upstream Helm Charts with Kustomize)
- jjeaff 8y ago
  You're just running helm charts directly from the public repo? I assume so since you say customization is a nightmare. I use helm fetch and pull the charts with dependencies into my repo. Running direct from the public repo is a little bit too much like piping the contents of a url to bash without checking first. You can then customize however you like. When I want to update the charts, helm fetch it again and merge the changes.
  - alexellisuk 8y ago
    So you are forking and maintaining the helm templates.. :-/
    - jjeaff 8y ago
      Not really. I'm essentially just pinning the version and then checking the updates before upgrading.
halbritt 8y ago
My team uses Helm extensively and will probably stick with it for the time being as it seems to have the most traction at the moment. As an example, MSSQL isn't a part of my core stack, but occasionally data engineers get databases from folks that they need to extract data from. Helm allows me to get one running in approximately 2 minutes or less. For many of the core components of our infrastructure we can deploy similarly quickly. We do our best to track stable for those charts. This is the tremendous advantage of helm. Unfortunately, many of those stable charts for core infrastructure are barely POC-worthy and take a great deal of effort to get into production state. For example, there is no helm chart for Kafka with TLS support. Maybe not necessary for every production implementation, but dammit, this is 2019. That said, helm is not safe. A helm "release" is not an artifact. It's a deployment of a helm chart of a certain version. That helm chart version may or may not correlate to the underlying version of the software you're running as that artifact is presumably defined with an "image" in values. For example, I've run the same aforementioned MSSQL helm chart with 2017-CU7-ubuntu, 2017-CU8-ubuntu, and 2017-CU9-ubuntu tags. A quick "helm list" might show you the same chart version across three different environments but they're still very different. Helm is not idempotent. Helm upgrade is a nightmare. One has to perform a helm upgrade to set values. Helm upgrades often fail. Even though they report failure, a look under the hood reveals that it actually succeeded. Occasionally a helm upgrade blows away the persistent volume attached to a stateful thing. That's always the best. tl;dr helm is the most popular, but has some massive shortcomings so we're all looking for something to replace it and kustomize is exciting because it's now for better or worse part of the core.

chupasaurus 8y ago

> Helm seems like a decent option If you want to: have PITA with RBAC, forget about manual changes because there are no 3-way-merge in Helm2 (that's what kubectl apply does, only hope it will be in Helm 3) and after this change helm cannot apply desired state, some of the biggest pains lasting for years had been resolved only in 2.13 with --atomic parameter. Aside from dependency resolution, everything that Helm does could be made by any template engine + API caller.

littlekosh 8y ago

Helm is great right up until you run into a chart that doesn’t support what you need, i.e. if you want to run on tainted nodes, it needs to support tolerations. Want to add a sidecar to a pod? No. It also lies about the actual status of the installation. A deployment that applies successfully but has issues with the new pods coming up can still show successful. The larger charts are a nightmare which isn’t helm’s fault but it doesn’t make things appreciably easier either. Kustomize has a lot of warts too. If the type definition in k8s source doesn’t have a merge key defined, it’s going to overwrite the entire section of the resource. You can use JsonPatches to get around that problem but that is super gross too. The entire ecosystem is trying to solve the same problem and no one has come up with the definitive way of doing it (because it’s a HARD problem) It seems like Helm templating and then passing to kustomize to add in the one-off changes is the direction a lot of people are heading but it’s more of a least-bad solution than a good one. I will say, if you’re using helm templates for your own stuff, you very likely going to be happy with it.

jaxxstorm 8y ago
This is a problem we got sick and tired of at $work, so we made kr8: https://github.com/apptio/kr8 https://github.com/apptio/kr8 You can define a component which you install into many clusters and then slightly differentiate them based on cluster parameters, kind of like Puppet or Chef (without the application stage) Alongside this, you can actually patch helm charts. An example component can be found here https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master/components/external_dns https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master... The patches.jsonnet allows you to add a commandline flag that wasn't in the helm chart at one time.
- littlekosh 8y ago
  Oh no, jsonnet! Seriously though, gonna look closer at this tomorrow at the office. Thanks for the heads up!
grantlmiller 8y ago
The pattern of Helm template + Kustomize is operationalized with Ship: https://github.com/replicatedhq/ship https://github.com/replicatedhq/ship plus some other functionality to automate pull requests from upstream updates.
- captn3m0 8y ago
  Isn't it a native usecase as well? This is linked from the kustomize website as a valid usecase: https://testingclouds.wordpress.com/2018/07/20/844/ https://testingclouds.wordpress.com/2018/07/20/844/ (Title = Customizing Upstream Helm Charts with Kustomize)
jjeaff 8y ago
You're just running helm charts directly from the public repo? I assume so since you say customization is a nightmare. I use helm fetch and pull the charts with dependencies into my repo. Running direct from the public repo is a little bit too much like piping the contents of a url to bash without checking first. You can then customize however you like. When I want to update the charts, helm fetch it again and merge the changes.
- alexellisuk 8y ago
  So you are forking and maintaining the helm templates.. :-/
  - jjeaff 8y ago
    Not really. I'm essentially just pinning the version and then checking the updates before upgrading.

halbritt 8y ago

My team uses Helm extensively and will probably stick with it for the time being as it seems to have the most traction at the moment. As an example, MSSQL isn't a part of my core stack, but occasionally data engineers get databases from folks that they need to extract data from. Helm allows me to get one running in approximately 2 minutes or less. For many of the core components of our infrastructure we can deploy similarly quickly. We do our best to track stable for those charts. This is the tremendous advantage of helm. Unfortunately, many of those stable charts for core infrastructure are barely POC-worthy and take a great deal of effort to get into production state. For example, there is no helm chart for Kafka with TLS support. Maybe not necessary for every production implementation, but dammit, this is 2019. That said, helm is not safe. A helm "release" is not an artifact. It's a deployment of a helm chart of a certain version. That helm chart version may or may not correlate to the underlying version of the software you're running as that artifact is presumably defined with an "image" in values. For example, I've run the same aforementioned MSSQL helm chart with 2017-CU7-ubuntu, 2017-CU8-ubuntu, and 2017-CU9-ubuntu tags. A quick "helm list" might show you the same chart version across three different environments but they're still very different. Helm is not idempotent. Helm upgrade is a nightmare. One has to perform a helm upgrade to set values. Helm upgrades often fail. Even though they report failure, a look under the hood reveals that it actually succeeded. Occasionally a helm upgrade blows away the persistent volume attached to a stateful thing. That's always the best. tl;dr helm is the most popular, but has some massive shortcomings so we're all looking for something to replace it and kustomize is exciting because it's now for better or worse part of the core.