8 ms·

We've been running in production on GKE for a little over two years and it's been a solid platform since day one. It's nice to read articles like this and see o

We've been running in production on GKE for a little over two years and it's been a solid platform since day one. It's nice to read articles like this and see others coming to the same conclusions we've come to. If your practices and workflow are oriented to containers and you outgrow your PAAS then k8s is the logical place to land. With respect to the choice of helm: we started out rolling our own pipeline with sed and awk and the usual suspects. When that became too complex helm was just taking off and we moved to that. We still use it to install various charts from stable that implement infrastructure things. For our own applications we found that there was just too much cognitive dissonance between the helm charts and the resulting resources. Essentially the charts and the values we plugged into them became a second API to kubernetes, obfuscating the actual API below. The conventions around the "package manager" role that the tool has taken for itself also contribute to lessen readability due to scads of boilerplate and name mangling. We recently started deploying things in a new pipeline based on kustomize. We keep base yaml resources in the application repo and apply patches from a config repo to finalize them for a given environment. So far it's working out quite well and the applications engineers like it much better. Now with kubectl 1.14 kustomize's features have been pulled in to that tool, something I have mixed feelings about, but at least the more declarative approach does seem to be the way the wind is blowing.

zapita 8y ago
Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14?
- markbnj 8y ago
  > Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14? It's nothing too surprising :). Simply a preference for simple, compose-able tools. I personally feel like patching yaml client-side is outside the scope of a "ctl" tool.
physicles 8y ago
Perhaps I don’t understand the design decisions behind Helm, but it’s always struck me as having a severe impedance mismatch with k8s itself. It defines another entirely different schema, and relies on an agent running in your cluster that’s also trying to reconcile desired state with actual state (which k8s itself is also doing). I’m skeptical that you could use it extensively without also understanding the k8s stuff underneath it. kustomize came along just as it’s become untenable for us to copy/paste config to multiple environments. I like that it’s pretty much the simplest possible way to customize yaml, and plan to dive in soon.
- techntoke 8y ago
  Once you start to see both Kustomize and Helm as a templating language, then you'll realize that Kustomize doesn't cover many use cases and is intentionally limited in scope. There is a reason almost every major project in the Kubernetes ecosystem has a Helm chart, but not a Kustomize configuration file. That doesn't mean that Helm doesn't have it's issues, because their implementation of Go templating is constrained, and Go templating is challenging in itself; however, it does a lot more than Kustomize can offer and covers a significantly larger amount of use cases than Kustomize patch files. Kustomize is great for getting a job done quickly if you don't mind some duplication of code and effort throughout your projects, but Helm is ideal for managing dependencies and templating across a large amount of projects where you may want to re-utilize other charts. I think it is ideal to not just discover these tools by reading about them, but actually start using them and spend a few hours trying them out and testing what they can and can't do.
  - markbnj 8y ago
    You're right about these things in my view. There's a lot you can do in a helm chart that you can't do with kustomize and patches. The helm template functions are powerful in their own right; you have conditionals, loops over ranges, you can define your own helpers, and then you have the whole sprig library at your fingertips. You can't do any of those things with kustomize. And that's really the (admittedly very opinionated) point. I think helm is perfect for the role the project assumed for itself, as a kubernetes package manager. It works well in that role. If you follow the conventions (as you must if you contribute to stable) then your thing has full config coverage with sensible defaults, you can install it multiple times and everything is namespaced. Like any package manager it's somewhat difficult to write good packages that have all these attributes. And I think you have to admit that a well-written helm chart that covers all the bases is a lot less readable (albeit much more powerful, stipulated) than a simple yaml description of the object you want. It really does constitute a separate API between the developer and the system. For our own deployable services we don't need all those package manager-like attributes. What we need is for the on-disk representation of our resources to be readable, and we want to be able to make simple changes to single files and not chase value substitution around through layers of indirection. We'll probably continue using helm for stable infrastructure things, but for our own services that are under continuous development and deployment it's come to feel like a mismatch.
SomaticPirate 8y ago
Second this, helm is good for day one operations of k8s. If you just want to have a redis deployment running now I think it’s fine for that. But then managing helm charts becomes its own task. Kustomize seems to be a happy medium. Also Helm v3 seems to be doubling down on that “second api” level with addition of embedding Lua in their templates.
- techntoke 8y ago
  Managing Kustomize configuration files is even worse in most regards, and you are likely to have a significantly larger amount of Kustomize files than Helm files, especially if you create a common chart and use dependencies correctly within Helm.
shosti 8y ago
Author here; yeah, helm is the part of our stack we’re least happy with tbh (it’s turned into a huge pile of templated yaml files for each project that seems like it might not be maintainable long-term). I’m curious about looking into kustomize, but the package management/rollback capabilities of helm are quite nice; is there a good “pre-baked” solution for that that doesn’t involve helm?
- markbnj 8y ago
  Great article, thanks for writing it. > I’m curious about looking into kustomize, but the package management/rollback capabilities of helm are quite nice; is there a good “pre-baked” solution for that that doesn’t involve helm? Not that I'm aware of. I'm only really familiar with helm, kustomize and ksonnet at this point, and of the three only helm took the approach of running a server-side stateful thing that could take responsibility for pseudo-resources called releases. I haven't followed work on the next version closely, but it will be interesting to see what changes as they ditch the client-server architecture. I assume it will be more like terraform with some sort of state file. Rollbacks for us basically mean go back and redeploy the last good ref. The pipeline is completely deterministic from the commit. The underlying build system that produced the container might not be, but we don't have to rebuild that since it's still sitting in the registry.
- tjungblut 8y ago
  I had some really good experience with ArgoCD so far, it is agnostic to what you're using as a configuration tool (helm, pure yamls, etc) and it just works and has a nice UI.
- mirchiseth 8y ago
  Excellent write up, thank you. I was also looking to read on how the dev experience changed moving to k8s. Were they using heroku cli? Did you create an equivalent experience in k8s or adopted a new one?
xnxn 8y ago
One intriguing new development is the possibility of leveraging Helm's chart ecosystem and Kustomize's patching mechanism—see Replicated Ship or the Kustomize Generators and Transformers KEP.
atombender 8y ago
We use Helm, but we really only use it for two things: Templating and atomic deploys/deletes. Helm templating is pretty terrible. Whoever thought generating YAML as text was a good idea deserves a solid wedgie. But it gets us where we need to be. During our prototyping of our GKE environment, we had lots of individual YAML files, which was not tenable. Atomic deploys/rollbacks is essential. What Helm brings to the table is a high-level way of tying multiple resources together into a group, allowing you to both identify everything that belongs together, and to then atomically apply the next version (which will delete anything that's not supposed to be there anymore). Labels would be sufficient to track that, in principle, but you still need a tool to ensure that the label schema is enforced. We don't use any of the other features of Helm -- they're just in the way. We don't use the package repo; we keep the chart for every app in the app's Git repo, so that it's versioned along with the code. We've written a nice wrapper around Helm so people just do "tool app deploy myapp -e staging", and it knows where to look for the chart, the values by environment etc. and invoke the right commands. (It also does nice things like check the CI status, lint the Kubernetes resources for errors, show a diff of what commits this will deploy, etc.) I've looked at Kustomize, and I don't think it's sufficient. For one, as far as I can see, it's not atomic. I'm hoping a clear winner will emerge soon, but nothing stands out. My favourite so far is Kubecfg, which is similar to the unnecessarily complex Ksonnet project, which has apparently been abandoned. Kubecfg is a very simple wrapper that only does Jsonnet templating for you. I'd be interested in how Google does these things with Borg. My suspicion is that they're using BCL (which Jsonnet is based on, last I checked) to describe their resources.
- markbnj 8y ago
  > I've looked at Kustomize, and I don't think it's sufficient. For one, as far as I can see, it's not atomic. Kustomize just applies structured edits to yaml. We run it to apply all the patches and output a single manifest file with all the resources, then send that to the master with kubectl apply. I suspect its as atomic as anything helm does, but I could be wrong.
  - atombender 8y ago
    The "atomicity" (a misleading term, I agree, but I couldn't think of a better one as I was writing the comment) I was referring to was its ability to do a destructive diff/patch. In other words, if you apply state (A+B+C), then (A+B), it will remove C. With plain "kubectl apply", there's the "--prune" flag, which is supposed to be able to track upstream resources via annotations. But it's still considered experimental alpha functionality, as least according to the "kubectl --help" for Kubernetes 1.11.9.
brown9-2 8y ago
I think it’s important to separate the use cases of a) deploying off the shelf software like Mysql and Redis from b) deploying your own custom built software. Helm seems a weird choice for B.

zapita 8y ago

Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14?

markbnj 8y ago
> Thank you for sharing this! I’m curious to learn more about your “mixed feelings” regarding built-in kustomize support in 1.14? It's nothing too surprising :). Simply a preference for simple, compose-able tools. I personally feel like patching yaml client-side is outside the scope of a "ctl" tool.

physicles 8y ago

Perhaps I don’t understand the design decisions behind Helm, but it’s always struck me as having a severe impedance mismatch with k8s itself. It defines another entirely different schema, and relies on an agent running in your cluster that’s also trying to reconcile desired state with actual state (which k8s itself is also doing). I’m skeptical that you could use it extensively without also understanding the k8s stuff underneath it. kustomize came along just as it’s become untenable for us to copy/paste config to multiple environments. I like that it’s pretty much the simplest possible way to customize yaml, and plan to dive in soon.

techntoke 8y ago
Once you start to see both Kustomize and Helm as a templating language, then you'll realize that Kustomize doesn't cover many use cases and is intentionally limited in scope. There is a reason almost every major project in the Kubernetes ecosystem has a Helm chart, but not a Kustomize configuration file. That doesn't mean that Helm doesn't have it's issues, because their implementation of Go templating is constrained, and Go templating is challenging in itself; however, it does a lot more than Kustomize can offer and covers a significantly larger amount of use cases than Kustomize patch files. Kustomize is great for getting a job done quickly if you don't mind some duplication of code and effort throughout your projects, but Helm is ideal for managing dependencies and templating across a large amount of projects where you may want to re-utilize other charts. I think it is ideal to not just discover these tools by reading about them, but actually start using them and spend a few hours trying them out and testing what they can and can't do.
- markbnj 8y ago
  You're right about these things in my view. There's a lot you can do in a helm chart that you can't do with kustomize and patches. The helm template functions are powerful in their own right; you have conditionals, loops over ranges, you can define your own helpers, and then you have the whole sprig library at your fingertips. You can't do any of those things with kustomize. And that's really the (admittedly very opinionated) point. I think helm is perfect for the role the project assumed for itself, as a kubernetes package manager. It works well in that role. If you follow the conventions (as you must if you contribute to stable) then your thing has full config coverage with sensible defaults, you can install it multiple times and everything is namespaced. Like any package manager it's somewhat difficult to write good packages that have all these attributes. And I think you have to admit that a well-written helm chart that covers all the bases is a lot less readable (albeit much more powerful, stipulated) than a simple yaml description of the object you want. It really does constitute a separate API between the developer and the system. For our own deployable services we don't need all those package manager-like attributes. What we need is for the on-disk representation of our resources to be readable, and we want to be able to make simple changes to single files and not chase value substitution around through layers of indirection. We'll probably continue using helm for stable infrastructure things, but for our own services that are under continuous development and deployment it's come to feel like a mismatch.

SomaticPirate 8y ago

Second this, helm is good for day one operations of k8s. If you just want to have a redis deployment running now I think it’s fine for that. But then managing helm charts becomes its own task. Kustomize seems to be a happy medium. Also Helm v3 seems to be doubling down on that “second api” level with addition of embedding Lua in their templates.

techntoke 8y ago
Managing Kustomize configuration files is even worse in most regards, and you are likely to have a significantly larger amount of Kustomize files than Helm files, especially if you create a common chart and use dependencies correctly within Helm.

shosti 8y ago

Author here; yeah, helm is the part of our stack we’re least happy with tbh (it’s turned into a huge pile of templated yaml files for each project that seems like it might not be maintainable long-term). I’m curious about looking into kustomize, but the package management/rollback capabilities of helm are quite nice; is there a good “pre-baked” solution for that that doesn’t involve helm?

markbnj 8y ago
Great article, thanks for writing it. > I’m curious about looking into kustomize, but the package management/rollback capabilities of helm are quite nice; is there a good “pre-baked” solution for that that doesn’t involve helm? Not that I'm aware of. I'm only really familiar with helm, kustomize and ksonnet at this point, and of the three only helm took the approach of running a server-side stateful thing that could take responsibility for pseudo-resources called releases. I haven't followed work on the next version closely, but it will be interesting to see what changes as they ditch the client-server architecture. I assume it will be more like terraform with some sort of state file. Rollbacks for us basically mean go back and redeploy the last good ref. The pipeline is completely deterministic from the commit. The underlying build system that produced the container might not be, but we don't have to rebuild that since it's still sitting in the registry.
tjungblut 8y ago
I had some really good experience with ArgoCD so far, it is agnostic to what you're using as a configuration tool (helm, pure yamls, etc) and it just works and has a nice UI.
mirchiseth 8y ago
Excellent write up, thank you. I was also looking to read on how the dev experience changed moving to k8s. Were they using heroku cli? Did you create an equivalent experience in k8s or adopted a new one?

xnxn 8y ago

One intriguing new development is the possibility of leveraging Helm's chart ecosystem and Kustomize's patching mechanism—see Replicated Ship or the Kustomize Generators and Transformers KEP.

atombender 8y ago

We use Helm, but we really only use it for two things: Templating and atomic deploys/deletes. Helm templating is pretty terrible. Whoever thought generating YAML as text was a good idea deserves a solid wedgie. But it gets us where we need to be. During our prototyping of our GKE environment, we had lots of individual YAML files, which was not tenable. Atomic deploys/rollbacks is essential. What Helm brings to the table is a high-level way of tying multiple resources together into a group, allowing you to both identify everything that belongs together, and to then atomically apply the next version (which will delete anything that's not supposed to be there anymore). Labels would be sufficient to track that, in principle, but you still need a tool to ensure that the label schema is enforced. We don't use any of the other features of Helm -- they're just in the way. We don't use the package repo; we keep the chart for every app in the app's Git repo, so that it's versioned along with the code. We've written a nice wrapper around Helm so people just do "tool app deploy myapp -e staging", and it knows where to look for the chart, the values by environment etc. and invoke the right commands. (It also does nice things like check the CI status, lint the Kubernetes resources for errors, show a diff of what commits this will deploy, etc.) I've looked at Kustomize, and I don't think it's sufficient. For one, as far as I can see, it's not atomic. I'm hoping a clear winner will emerge soon, but nothing stands out. My favourite so far is Kubecfg, which is similar to the unnecessarily complex Ksonnet project, which has apparently been abandoned. Kubecfg is a very simple wrapper that only does Jsonnet templating for you. I'd be interested in how Google does these things with Borg. My suspicion is that they're using BCL (which Jsonnet is based on, last I checked) to describe their resources.

markbnj 8y ago
> I've looked at Kustomize, and I don't think it's sufficient. For one, as far as I can see, it's not atomic. Kustomize just applies structured edits to yaml. We run it to apply all the patches and output a single manifest file with all the resources, then send that to the master with kubectl apply. I suspect its as atomic as anything helm does, but I could be wrong.
- atombender 8y ago
  The "atomicity" (a misleading term, I agree, but I couldn't think of a better one as I was writing the comment) I was referring to was its ability to do a destructive diff/patch. In other words, if you apply state (A+B+C), then (A+B), it will remove C. With plain "kubectl apply", there's the "--prune" flag, which is supposed to be able to track upstream resources via annotations. But it's still considered experimental alpha functionality, as least according to the "kubectl --help" for Kubernetes 1.11.9.

brown9-2 8y ago

I think it’s important to separate the use cases of a) deploying off the shelf software like Mysql and Redis from b) deploying your own custom built software. Helm seems a weird choice for B.