5 ms·

Since I was at the core of what the article wrote about I feel the need to jump in. Some of the quotes are from me, after all. kustomize vs Helm is a bit if an

by mfer 8y ago

Since I was at the core of what the article wrote about I feel the need to jump in. Some of the quotes are from me, after all. kustomize vs Helm is a bit if an improper comparison. Kustomize and the ways to use kubectl are often about configuration management and Helm is about package management. This is an important difference. For example, take Ansible doing configuration management and using Helm to install a package in much the same way you might use an apt package elsewhere. The key question that has to do with configuration management is should kubectl provide configuration management of things outside of the Kubernetes API, such as when you manage the config in git? This is both about scope (is this a Kubernetes concern) and in how to build things (a monolithic CLI?). If kubectl has a configuration management opinion outside of the Kubernetes API what does that mean for configuration management projects like ansible, armada, chef, and the oh so many others? And, who working on Kubernetes gets to choose the opinion on configuration management it has outside the API? At this point imperative vs declarative as a conversation point comes up. In all cases the configuration is declared to the Kubernetes API and stored by Kubernetes in a declarative manner. It becomes a question of how far outside the Kubernetes API do things need to remain declarative. After all, me typing into yaml into a keyboard is imperative. As for the KEP process as a whole and the graduation criteria that was mentioned here, we know it needs work. There are efforts going to get it into better shape with better example guidance. There is actually a PR being worked on for this right now. Projects with a healthy ecosystem and people of strong opinions have to deal with issues of scope. What do you hold onto and where should it go? How do you curate a healthy ecosystem? It's not always easy. Having a place to do some experiments outside of a project like Kubernetes is part of the reason the CNCF added sandbox projects. Without being forceful people can work together on something and see how it fairs in the market. Experiments in the ecosystem can happen. In any case, if people have questions please feel free to ask.

marcc 8y ago
Helm is a good package management tool for application level configuration items. The possible configuration options that a cluster operator might want to configure is fixed. The application packager knows what they may be and can template them into environment variables or write them to configmaps. It doesn't matter how they are delivered, the values.yaml is a great place to collect this data, and Helm manages it nicely. However, when it comes to cluster operators, or "runtime configuration", the application packager cannot possibly know all possible configuration changes that a cluster operator might want to make. Some are obvious, and Helm ecourages putting them into values.yaml. For example, most charts include a way to change resource limits or imagepullpolicy. But what if a cluster operator wants to change a config map to be deployed as a bitnami SealedSecret, because it's more secure and more compatible with the infrastructure? Or what if a cluster operator wants to deploy the Cloudflare Argo Tunnel Ingress controller which requires more: not only an annotation, but also additional fields. This cluster operator often ends up forking the Helm chart or deploying these parts out-of-band. Differentiating between runtime config and application config is why both Kustomize and Helm are both useful, together. I like Helm as a package manager. But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included? I like the sandbox projects and how the CNCF is managing them. I'm actively working on a different project (Open Policy Agent), and it's much younger than Helm or Kubectl. I agree with your concerns about the KEP process, and how it should be more well defined and followed at all times.
- geerlingguy 8y ago
  This is exactly why I’m not using Helm for most applications; it would be more maintenance to try to maintain all the more specific forked helm charts and a helm chart repository than to maintain the direct and lightly templated manifests which allow me to have 100% immutable definitions of all my infrastructure (like I prefer with or without Kubernetes), but without also maintaining a huge set of more complicated Helm charts.
- mfer 8y ago
  Ya know what's great about debating tools? Because we have a separation, different things can innovate at different rates and people can do different things. You can move from deploying to AWS to Kubernetes all while your CI pipeline and config management stay the same. Maybe at another time you change your CI system. In all of this we can even have tool fights. Jenkins vs CircleCI. vim vs emacs. And new players can come along like vscode. Different departments in the same companies, sometimes billion dollar companies, can even do this. This is one of the reasons I personally like a separation of concerns with projects. There is one thing about Helm... > But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included? I don't think so. Without being long winded, this is about user experience and encapsulations. Kubernetes is hard to learn. Then do learn the business logic for installing all the things you use. To use the old wordpress example, should someone installing wordpress into a cluster need to know the business needs of MySQL to create the k8s config for it? Most app devs don't want to learn all the k8s object configs... it's a lot to learn. As for CRDs, that's about dependency management. But, if we make Kubernetes really hard, with lots of CRDs we have to start to deal with what CRDs are in what clusters? And, what experience does that provide for end users? How do we not make the UX barrier to entry to high for the people who need to use it? Are users interested in Kubernetes or are they interested in their apps and core business logic?
  - marcc 8y ago
    > people can do different things That's one of my favorite things about that composability of the Kubernetes tools. I don't believe there's a reason for a "tool fight" in the Kubernetes ecosystem. There might be disagreements about best practices, for example how to manage code in a cluster (gitops vs kubectl apply vs helm install vs ...), but that's not a tool fight as much as a methodology difference. And there's room for more than one pattern to emerge. I think that any tool that is able to deploy an open source project to my cluster, while minimizing the amount of operational overhead I need to assume, is the tool that I want. I don't care if that is Kustomize, Helm, Ksonnet, or anything else, as long as it mets the requirements of a) it has to work with my environment and b) it shouldn't introduce unecessary operational overhead. You also mention that Kubernetes is hard to learn, which is absolutely a problem. Adoption is growing, but it's getting harder to learn as more features get merged in. And you are right, nobody should have to learn Kubernetes YAML to deploy a standard, off-the-shelf Wordpress installation. But what about more complicated software that needs "last mile" customization done to work in a specific environment? This blog post (https://testingclouds.wordpress.com/2018/07/20/844/ https://testingclouds.wordpress.com/2018/07/20/844/) shows a great way to combine the power of the Helm community and chart format with the last-mile tooling that Kustomize can provide to help keep charts simple but still flexible. That feels much better than forking the chart and maintaining a separate copy of it just to make a few changes that are specific to a single use case. I have mixed feelings about Kustomize getting merged into kubectl. I don't like the idea of Google "crowning" a winner, and I hope the sig-architecture group and Google teams remain diligent to prevent that from happening. Kustomize is not a replacement for Helm, it's a very good tool to handle specific use cases that often involve Helm charts at the source.
aberoham 8y ago
Hi mfer! Writer here, wonderful to virtually meet you. I applaud your unique gusto and framing of questions. (btw, pretty much every question/quote was from you) From afar it looks like `kubectl -k` is due to land despite the expressed concerns. What was your response to sig-arch and sig-cli leads who mentioned that the kustomize features were always intended to be there? I may have missed that, if it was public. Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed? And what are you setting your sights on next? May I write about it? :)
- mfer 8y ago
  > What was your response to sig-arch and sig-cli leads who mentioned that the kustomize features were always intended to be there? I may have missed that, if it was public. The document describing what kustomize should do and that looked at some other tools was started in mid-2017. Do you have tracability to the idea that overlays were meant to be there from the beginning? I ask because I'm honestly interested in reading that. As far as people go, 1 of the 3 SIG Arch chairs has been around a long time. In addition to him I would want to poke at the co-founders on this one. Joe Beda and Craig McLuckie (2/3 of the K8s cofounders) started Heptio and they produced ksonnet (kubernetes + jsonnet which both came out of Google). This is different than the direction for kustomize. Kubernetes has been declarative from the beginning. It's an important concept in all of this. Isn't declaring it to the API and then storing/creating objects outside the API separate concerns? Back in 2017 Helm was a core part of Kubernetes as were projects like Komponse, that lets you convert Docker Compose files into Kubernetes manifests. Helm v2, which was part of Kubernetes, was the product of merging deployment manager, a Kubernetes project out of Google, and Helm v1, by Deis. Helm v2 began work before the CNCF existed. It was a part of Kubernetes back when it was governed by Google. A history of tooling outside of the API had long been there. To not have opinions outside the API to allow more diversity there, in 2017 we started treating Helm and Kompose like other ecosystem projects. We talked about not adding opinionated projects like this within Kubernetes anymore to keep the door open for the ecosystem and diversity there. > Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed? I think it depends where the usage data comes from. Can we get real end user data? Or, will the feedback come from the Kubernetes insiders as has happened too often in the surveys? (I ran past surveys so I feel this pain) What will the data look like from large companies with bespoke processes and strong opinions on their toolchains? I hope real useful data is collected? I would love data on more of the problem their facing.
guslees 8y ago
I feel like the KEP was never examined "adversarially". And that isn't the right word, because nobody wants to create conflict where there is none. Perhaps "skeptically" is a better word, but even that implies some level of external objection. To pick on some almost unfairly cherry-picked examples: - the KEP called out "don't want to run scripts from github" as a design goal _for_ kustomize, when kustomize itself clearly ran arbitrary scripts in well advertised situations. Nobody during the (lengthy and open) KEP review stage raised this. (Since fixed by removing this kustomize feature) - the various "to plugin or not to plugin" discussions are highly subjective (and always will be) and dance around the elephant in the room: "We think the plugin mechanism is great" obviously doesn't marry with "nobody wants to use it". - the scope and approach of kustomize vs other "less stylistically opinionated" alternatives was never (openly?) considered afaics. For example, perhaps we could _just_ have a stack of json/s-m-p patches that get applied, or a generic mechanism for applying arbitrary "post-process" transformations. The kustomize KEP didn't really consider "not using kustomize" as an alternative, which is understandable from a kustomize pov (we must presume the authors like their proposal!), but not from a sig-cli and overall process pov. Afaics, the kustomize folks did everything right through this, as did the rest of sig-sli, and the above is only apparent in hindsight. The reality is that the sig contributor group is small, and necessarily can only consider proposals that actually exist. The KEP process is entirely human-based, with all the biases and fallibility that comes from that. I don't have any useful suggestions for how to institutionalise scepticism that won't also bring far worse chilling effects. I note the KEP template already includes "drawbacks" and "alternatives" sections, but it relies on cultural expectations to ensure these are used effectively. TL;DR: I actually think this "kerfuffle" is an example of the larger process working _well_. A decision graduated to a larger audience, was questioned, and rolled back to be reexamined in greater detail. We must _expect_ to have to iterate on some decisions sometimes, and that's ok, normal and healthy.

by mfer 8y ago

marcc 8y ago

Helm is a good package management tool for application level configuration items. The possible configuration options that a cluster operator might want to configure is fixed. The application packager knows what they may be and can template them into environment variables or write them to configmaps. It doesn't matter how they are delivered, the values.yaml is a great place to collect this data, and Helm manages it nicely. However, when it comes to cluster operators, or "runtime configuration", the application packager cannot possibly know all possible configuration changes that a cluster operator might want to make. Some are obvious, and Helm ecourages putting them into values.yaml. For example, most charts include a way to change resource limits or imagepullpolicy. But what if a cluster operator wants to change a config map to be deployed as a bitnami SealedSecret, because it's more secure and more compatible with the infrastructure? Or what if a cluster operator wants to deploy the Cloudflare Argo Tunnel Ingress controller which requires more: not only an annotation, but also additional fields. This cluster operator often ends up forking the Helm chart or deploying these parts out-of-band. Differentiating between runtime config and application config is why both Kustomize and Helm are both useful, together. I like Helm as a package manager. But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included? I like the sandbox projects and how the CNCF is managing them. I'm actively working on a different project (Open Policy Agent), and it's much younger than Helm or Kubectl. I agree with your concerns about the KEP process, and how it should be more well defined and followed at all times.

geerlingguy 8y ago
This is exactly why I’m not using Helm for most applications; it would be more maintenance to try to maintain all the more specific forked helm charts and a helm chart repository than to maintain the direct and lightly templated manifests which allow me to have 100% immutable definitions of all my infrastructure (like I prefer with or without Kubernetes), but without also maintaining a huge set of more complicated Helm charts.
mfer 8y ago
Ya know what's great about debating tools? Because we have a separation, different things can innovate at different rates and people can do different things. You can move from deploying to AWS to Kubernetes all while your CI pipeline and config management stay the same. Maybe at another time you change your CI system. In all of this we can even have tool fights. Jenkins vs CircleCI. vim vs emacs. And new players can come along like vscode. Different departments in the same companies, sometimes billion dollar companies, can even do this. This is one of the reasons I personally like a separation of concerns with projects. There is one thing about Helm... > But to keep everyone using the upstream Helm chart, wouldn't all Helm charts eventually have to look the same, and just be a templated version of the entire Kubernetes API with all CRDs included? I don't think so. Without being long winded, this is about user experience and encapsulations. Kubernetes is hard to learn. Then do learn the business logic for installing all the things you use. To use the old wordpress example, should someone installing wordpress into a cluster need to know the business needs of MySQL to create the k8s config for it? Most app devs don't want to learn all the k8s object configs... it's a lot to learn. As for CRDs, that's about dependency management. But, if we make Kubernetes really hard, with lots of CRDs we have to start to deal with what CRDs are in what clusters? And, what experience does that provide for end users? How do we not make the UX barrier to entry to high for the people who need to use it? Are users interested in Kubernetes or are they interested in their apps and core business logic?
- marcc 8y ago
  > people can do different things That's one of my favorite things about that composability of the Kubernetes tools. I don't believe there's a reason for a "tool fight" in the Kubernetes ecosystem. There might be disagreements about best practices, for example how to manage code in a cluster (gitops vs kubectl apply vs helm install vs ...), but that's not a tool fight as much as a methodology difference. And there's room for more than one pattern to emerge. I think that any tool that is able to deploy an open source project to my cluster, while minimizing the amount of operational overhead I need to assume, is the tool that I want. I don't care if that is Kustomize, Helm, Ksonnet, or anything else, as long as it mets the requirements of a) it has to work with my environment and b) it shouldn't introduce unecessary operational overhead. You also mention that Kubernetes is hard to learn, which is absolutely a problem. Adoption is growing, but it's getting harder to learn as more features get merged in. And you are right, nobody should have to learn Kubernetes YAML to deploy a standard, off-the-shelf Wordpress installation. But what about more complicated software that needs "last mile" customization done to work in a specific environment? This blog post (https://testingclouds.wordpress.com/2018/07/20/844/ https://testingclouds.wordpress.com/2018/07/20/844/) shows a great way to combine the power of the Helm community and chart format with the last-mile tooling that Kustomize can provide to help keep charts simple but still flexible. That feels much better than forking the chart and maintaining a separate copy of it just to make a few changes that are specific to a single use case. I have mixed feelings about Kustomize getting merged into kubectl. I don't like the idea of Google "crowning" a winner, and I hope the sig-architecture group and Google teams remain diligent to prevent that from happening. Kustomize is not a replacement for Helm, it's a very good tool to handle specific use cases that often involve Helm charts at the source.

aberoham 8y ago

Hi mfer! Writer here, wonderful to virtually meet you. I applaud your unique gusto and framing of questions. (btw, pretty much every question/quote was from you) From afar it looks like `kubectl -k` is due to land despite the expressed concerns. What was your response to sig-arch and sig-cli leads who mentioned that the kustomize features were always intended to be there? I may have missed that, if it was public. Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed? And what are you setting your sights on next? May I write about it? :)

mfer 8y ago
> What was your response to sig-arch and sig-cli leads who mentioned that the kustomize features were always intended to be there? I may have missed that, if it was public. The document describing what kustomize should do and that looked at some other tools was started in mid-2017. Do you have tracability to the idea that overlays were meant to be there from the beginning? I ask because I'm honestly interested in reading that. As far as people go, 1 of the 3 SIG Arch chairs has been around a long time. In addition to him I would want to poke at the co-founders on this one. Joe Beda and Craig McLuckie (2/3 of the K8s cofounders) started Heptio and they produced ksonnet (kubernetes + jsonnet which both came out of Google). This is different than the direction for kustomize. Kubernetes has been declarative from the beginning. It's an important concept in all of this. Isn't declaring it to the API and then storing/creating objects outside the API separate concerns? Back in 2017 Helm was a core part of Kubernetes as were projects like Komponse, that lets you convert Docker Compose files into Kubernetes manifests. Helm v2, which was part of Kubernetes, was the product of merging deployment manager, a Kubernetes project out of Google, and Helm v1, by Deis. Helm v2 began work before the CNCF existed. It was a part of Kubernetes back when it was governed by Google. A history of tooling outside of the API had long been there. To not have opinions outside the API to allow more diversity there, in 2017 we started treating Helm and Kompose like other ecosystem projects. We talked about not adding opinionated projects like this within Kubernetes anymore to keep the door open for the ecosystem and diversity there. > Do you expect that the march toward kustomize inclusion will change after the kubectl usage data has been collected and analyzed? I think it depends where the usage data comes from. Can we get real end user data? Or, will the feedback come from the Kubernetes insiders as has happened too often in the surveys? (I ran past surveys so I feel this pain) What will the data look like from large companies with bespoke processes and strong opinions on their toolchains? I hope real useful data is collected? I would love data on more of the problem their facing.

guslees 8y ago

I feel like the KEP was never examined "adversarially". And that isn't the right word, because nobody wants to create conflict where there is none. Perhaps "skeptically" is a better word, but even that implies some level of external objection. To pick on some almost unfairly cherry-picked examples: - the KEP called out "don't want to run scripts from github" as a design goal _for_ kustomize, when kustomize itself clearly ran arbitrary scripts in well advertised situations. Nobody during the (lengthy and open) KEP review stage raised this. (Since fixed by removing this kustomize feature) - the various "to plugin or not to plugin" discussions are highly subjective (and always will be) and dance around the elephant in the room: "We think the plugin mechanism is great" obviously doesn't marry with "nobody wants to use it". - the scope and approach of kustomize vs other "less stylistically opinionated" alternatives was never (openly?) considered afaics. For example, perhaps we could _just_ have a stack of json/s-m-p patches that get applied, or a generic mechanism for applying arbitrary "post-process" transformations. The kustomize KEP didn't really consider "not using kustomize" as an alternative, which is understandable from a kustomize pov (we must presume the authors like their proposal!), but not from a sig-cli and overall process pov. Afaics, the kustomize folks did everything right through this, as did the rest of sig-sli, and the above is only apparent in hindsight. The reality is that the sig contributor group is small, and necessarily can only consider proposals that actually exist. The KEP process is entirely human-based, with all the biases and fallibility that comes from that. I don't have any useful suggestions for how to institutionalise scepticism that won't also bring far worse chilling effects. I note the KEP template already includes "drawbacks" and "alternatives" sections, but it relies on cultural expectations to ensure these are used effectively. TL;DR: I actually think this "kerfuffle" is an example of the larger process working _well_. A decision graduated to a larger audience, was questioned, and rolled back to be reexamined in greater detail. We must _expect_ to have to iterate on some decisions sometimes, and that's ok, normal and healthy.