8 ms·

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someon

We are using k8s as well for a project that we also provide on-prem install too. We decided to use kustomize for now vs doing helm for now. Curious, have someone had experience with both to compare. What are the benefits of using helm?

dpc_pw 6y ago
`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.
- justicezyx 6y ago
  Insider joke: K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem. One of them is the config files hell. Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm) In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google. I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade). Well, it seems I am still on the track to be on the correct camp...
  - esprehn 6y ago
    Relevant thread (with you) from the nearby past: https://news.ycombinator.com/item?id=20849454 https://news.ycombinator.com/item?id=20849454 The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.
  - sandGorgon 6y ago
    What do you think about Cue ? There's a very interesting issue about why Cue improves upon Borg. https://github.com/cuelang/cue/issues/33#issuecomment-483615374 https://github.com/cuelang/cue/issues/33#issuecomment-483615...
    - justicezyx 6y ago
      Sorry, I have little knowledge on Cuelang. @mpvl is one of my most admired Googler. I promised to contribute to the Cue project, but haven't found time (typical startup employee situation, but it seems there is no trouble to find time surfing hacker news...).
- AaronFriel 6y ago
  Are you familiar with Pulumi? It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch. It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as safely propagate configuration outputs and secrets.
  - gscho 6y ago
    ...and you need to use their SaaS to make it work, no?
    - cottonseed 6y ago
      I don't think so, but you need to manage/sync the infrastructure state yourself.
    - Aeolun 6y ago
      Nope, you can sync state using just an S3 bucket (or local filesystem). It’s a bit hidden in their docs though.
    - AaronFriel 6y ago
      No, just like Terraform works with different storage backends, theirs does too.
- richm31415 6y ago
  Or eschew config files altogether and write operators in golang . . . https://sdk.operatorframework.io/ https://sdk.operatorframework.io/
  - SomaticPirate 6y ago
    Still have to write a CRD if you want to do anything useful (arguably still requires a config).
- tootie 6y ago
  If you're doing more than storing config values in YAML, you're playing with fire. I've really grown to dislike the trend in "developing by descriptor". It's great when it works and impossible to debug when anything goes wrong. And also, not easily testable.
- creztoe 6y ago
  Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](https://github.com/kubernetes-client/python https://github.com/kubernetes-client/python) without much trouble. Personally I prefer editing them as JSON instead of python objects, but it isn't too bad.
  - Nullabillity 6y ago
    > Could you explain why helm is garbage? Not the OP, but.. 1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating. 2. "Values" aren't typechecked or cleaned. 3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by hand. 4. No good way to preview what a deploy will change. 5. Weird interactions when resources are edited manually (especially back in Helm 2, but still a thing). 6. No good way to migrate objects into a Helm chart without deleting and recreating them. 7. Tons of repetitive boilerplate in each chart to customize basic settings (like replica counts). It's a typical Go solution, in all the wrong ways.
    - alephu5 6y ago
      It's not going to solve all your problems but dhall can fix your first few gripes. I've been using it for several months and it's an excellent way to write configuration imo.
      - Nullabillity 6y ago
        Yeah, I have used Nix to generate them in the past, which worked pretty great too. But Helm does, admittedly, solve a real problem: garbage collecting old resources when they're deleted from the repo. I just wish we could have something much simpler that only did that...
- dilyevsky 6y ago
  Sounds like you may enjoy https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod (some assembly may be required)
- gscho 6y ago
  You can't just say helm is garbage without saying why.
- bfrydl 6y ago
  I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC. I've actually been using Terraform to manage Kubernetes resources at my latest job, along with literally everything else. I don't even use alb-ingress-controller or external-dns or any of that. I just have Terraform make target groups from the service resources. It breaks a lot less.
  - OJFord 6y ago
    But Helm barely adds any boilerplate over that 'single deployment or similar + RBAC'? Just a Chart.yaml (minimal of which is very small) and adding the option to parameterise or templatise the resources. I don't know why you wouldn't, we don't say 'I don't know if .deb/.rpm/PKGINFO/etc. is worth it, most packages are just a binary'. For Helm to add LoC you'd have to all but not use it, and even then it'd be about 3LoC. Setup is no harder, with N kubectl applies becoming 1 helm install. I also started using terraform (directly instead of just triggering helm) - are you using kubernetes tf resources directly though, or the helm provider?
  - jjeaff 6y ago
    Most helm charts I have seen and used have a deployment yaml or statefulset or daemonset plus a service yaml, optional ingress yaml, sidecar containers for metrics, and several others I'm forgetting at the moment. And I would say most as in 90% of the charts on helm hub.
- x87678r 6y ago
  > Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me. Someone here talked about the joys of using the API with protobufs. Something I want to try out because it sounds clean.
- aeontech 6y ago
  Have you looked at Dhall or Tanka? They are both designed to solve exactly this problem (and I couldn’t agree with you more)
  - dpc_pw 6y ago
    Honestly, I am not a fan of "configuration languages". Why do we need another bunch of languages. Were the existing languages not good enough? There's already shitload of them. The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill. Not strong feelings about it though, maybe I'm wrong. All I want is a library for my favorite language, where I can easily output bunch of `yaml` files to feed to k8s. I guess pulumi is kind of like that, but I never had a chance to try it.
    - kqr 6y ago
      I don't know about the others, but at least Dhall provides features that are literally impossible to get from a general purpose programming language, like semantic hashing, totality, and such.
    - kdkeyser 6y ago
      My experience with using general purpose languages for things like configuration or build systems has not been great. You always seem to end up with building a bunch of abstractions on top of that language, which culminates in another in-house configuration/build DSL, for which no public info is available. It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects. Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity. On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language. I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)
- knocte 6y ago
  > .. with code Use Pulumi, it does exactly this.
- ecnahc515 6y ago
  I generally agree. I'm using kustomize right now, but I'm also using helm charts via a kustomize plugin called chartInflator. It allows us to re-use existing upstream boilerplate while still using kustomize. Though I've not been entirely happy with kustomize either, and I'm thinking I might begin to investigate jsonnet for some things.
- AlphaSite 6y ago
  Carvel/k14s has been very nice for the usecase I had. Use ytt to generate the yams, using a yaml aware variant of skylark and kapp to manage the actual atomic deployment, delete, etc. It’s very simple and worked quite well for me. Disclaimer: work at VMware but on a different team (it’s an OS tool built by some of our engineers, which we use internally)
- kkapelon 6y ago
  > What I'd like is to generate serialized yaml files ... you know ... with code Check out Pulumi (I am not affiliated with them in any way)
- cbushko 6y ago
  This is why I use Terraform for my kubernetes objects. Yes, it is a DSL and not perfect but it has many advantages over helm such as: - State. I cannot express how useful having the desired and actual state of a system is. It makes it so much easier to determined what has changed in between terraform applies which in turn makes tracking down problems easier. - Providers. Being able to chain resources together from other systems is very powerful. As an example, I can: 1) create a database with a random name 2) create a user/password for that database 3) save the data above as a kubernetes secret 4) pass that secret into my kubernetes container for the service to use. And you can easily change database to be anything such as a storage bucket, dns entry, twilio account, stripe account, etc. Now try and do the exact same thing with helm! - Providers Part 2: You can write your own providers to talk to your OWN services. I have done this and written up some go code to talk to our internal APIs when setting up new customers. It is very easy to create your own terraform resources to do this. There are more reasons but I have work to do :)
ypeter 6y ago
Two benefits: - I already use Helm to install open source software; there are already so many Charts available. So that makes it easy to also use Helm for my own apps (I use it just for templating though) - Helm templating language is more flexible, but that can also bite you since it can become complex. With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)
- snuxoll 6y ago
  > With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching) Config maps, enough said. I have a CD pipeline that writes environment variables to files, Kustomize eats them up and creates a configmap, environment variables in the deployment reference the config map, done. Kapp as a layer on top of Kustomize is showing some progress. Ultimately Kustomize is like every other resource bundled with k8s, it’s a building block for higher level tooling.
cpaika 6y ago
Helm is amazing. If you are migrating legacy applications that rely on filesystem config, you can publish that config in a helm chart that applications require as sub charts. Gives you the ability to do centralized, versioned config management for legacy workloads going to the cloud Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason the infra space has been using declarative syntax for years now
- magnawave 6y ago
  The declarative functionality comes from K8s not helm. (Of course K8s has imperative tooling too but that’s not the point here). The main reason folks dislike helm are 2 fold: 1) using string templating on a space formatted language is just messy and ugly for no good reason. If you don’t do anything terribly complicated it’s not too bad but quickly can be pretty fragile and unreadable. 2) golang(which I like generally) but the templating is messy and adds to the hard to read later problem Personally find helm’s opportunity for code reuse to be restrictive and folks just end up cut and pasting a lot of declarations all over which is ... so terrible. But the first bullet above is pretty a fundamental design flaw.
gravypod 6y ago
Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommend kubecfg [0] as it's a tool that "just works" and is built at the perfect level of abstraction: "just give me something and I'll find all the kubernetes objects inside of it". It's remarkably simple and forgiving and it makes it possible to describe even the most complex application configs + kube configs in a single language. You can also take it a step further and describe your CI stages (pre-merge, release, builds, etc), kube yaml, and infrastructure provisioning (terraform) all in one language. > What are the benefits of using helm? The main benefit is redistribution. Since everyone uses helm it's easy to give someone a chart because they know what to do with it. That's pretty much the only upside I've found. [0] - https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg
- SomaticPirate 6y ago
  Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough. If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.
  - gravypod 6y ago
    Yea, anything that allows you to think in a higher-level than raw kube objects will save you a lot of time and heartache. For instance, if your application needs a redis cluster in helm you have to mess with subcharts/other headaches. In jsonnet it becomes `redis = import "redis"; redis.Cluster("my-app-name") { nodes: 10 }` or something similar. DRY configs make me very happy.
  - iso8859-1 6y ago
    Given that the Dhall type system supports things that the other configuration languages do not, isn't it very hard to migrate to for a project like k8s? I don't see how they could support something in parallel to Dhall, and still enable all the features Dhall provides.
theptip 6y ago
Helm is good for importing other components (eg Redis). I find it pretty tedious for authoring my own Yaml specs; any time you need to do some sort of composition you have to write templates, and then mess with indentation operators, because significant white space. Templating yaml is a pretty bad idea in my experience. As of Helm3 the most egregious part (Tiller) is gone, so it’s more recommendable. Scaffold lets you write (roughly) native k8s yaml files and layer them together. IDEs know the schema for this yaml at this point, so you get code completion on all your input files. This breaks in Helm. Scaffold is good for simple apps, and falls down a bit if you need to dynamically configure your specs (eg render a parameterized manifest for each review app). The programming model for scaffold is a bit hard to grok, based on my experience of ramping up a team on it. Not rocket science just a bit unintuitive at first.
pnathan 6y ago
Helm is a wannabe package manager, designed for people who really like writing yaml instead of executable code. K8s doesn't really have a fully built out "apps" resource, so that starts the whole process down a gravelly slope of pain... with initial care and if your team need to groom their k8s configs, you can go a long way with helm before the pain starts. A particular problem with helm is that it isn't really a fully fledged programming language, but a glue laid over the golang templating system. I use helm as little as possible. Usually only enough to template out a single file with the different bits of k8s infra stuck together with `---`. If I was doing some relatively serious work on custom services with minimal needs to conform to best^H^H^H^Hpopular practices, my service's configuration would be a neatly packaged Java program (its libraries for K8s direct API access are good). There are different approaches to handling ops tasks; I favor the ones that are rooted in writing executable code over the ones with yaml. Some bright chappie will probably write a Common Lisp library derivation library and k8s will be obvious and simple with that interface, but I'm not that bright - all I can see is the potential.
jarv 6y ago
Blog author here - we are using Helm for our application because of how much we have already invested in our public Helm chart. We are not sold on using Helm however for K8s in general and are currently evaluating Tanka for some of our monitoring infrastructure. One of our engineers wrote up some notes on that here if you are curious https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11260 https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issu...
kkapelon 6y ago
Helm is a full package manager. Kustomize is just a templating mechanism. They are not comparable. For example, with Helm you can rollback applications to previous versions

dpc_pw 6y ago

`helm` is an absolute garbage. I'll reserve judgment about `kustomize` until I have more practical experience, but so far it looks to me that it's going to be another YAML disaster. The problem with k8s ecosystem is that developers conflated k8s using yaml for a semi-human-readable serialization of k8s resources, as reason to employ YAML to absolutely everything. What I'd like is to generate serialized yaml files ... you know ... with code. Using some typed programming language, to be able to build any abstractions required for the job, and get some checks, compiler errors, and even ability to have asserts, unit-tests and so on. Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me.

justicezyx 6y ago
Insider joke: K8s was born to solve the same problem Borg solved or simply design in such a way to not have that problem. One of them is the config files hell. Borgcfg files ranks #3 in terms of line count across all of Google. (Borgcfg language is similar to Jsonnet, helm) In terms of the amount of power wielded buy a single CLI, `borgcfg`, this little program that has less than 50k lines of nontesting code, stands at a peak dwarfs almost anything else inside Google. I declared in 2015 that k8s will eventually enjoy the same borgcfg problem. And I was advocating code driving application management approach as you suggested, not the configuration as code (behind the infrastructure as code facade). Well, it seems I am still on the track to be on the correct camp...
- esprehn 6y ago
  Relevant thread (with you) from the nearby past: https://news.ycombinator.com/item?id=20849454 https://news.ycombinator.com/item?id=20849454 The yaml monster in k8s is quite unpleasant. It's always been surprising to me how we somehow ended up with executable BUILD files but totally static deploy files.
- sandGorgon 6y ago
  What do you think about Cue ? There's a very interesting issue about why Cue improves upon Borg. https://github.com/cuelang/cue/issues/33#issuecomment-483615374 https://github.com/cuelang/cue/issues/33#issuecomment-483615...
  - justicezyx 6y ago
    Sorry, I have little knowledge on Cuelang. @mpvl is one of my most admired Googler. I promised to contribute to the Cue project, but haven't found time (typical startup employee situation, but it seems there is no trouble to find time surfing hacker news...).
AaronFriel 6y ago
Are you familiar with Pulumi? It does away with the notion that you need serialized yaml files, although in its current incarnation it can produce them, and instead runs programs which generates collections of named (and hierarchically defined) resources which can then be "applied" as a batch. It uses lazy values to generate a directed graph to determine the order for creating and tearing down resources as well as safely propagate configuration outputs and secrets.
- gscho 6y ago
  ...and you need to use their SaaS to make it work, no?
  - cottonseed 6y ago
    I don't think so, but you need to manage/sync the infrastructure state yourself.
  - Aeolun 6y ago
    Nope, you can sync state using just an S3 bucket (or local filesystem). It’s a bit hidden in their docs though.
  - AaronFriel 6y ago
    No, just like Terraform works with different storage backends, theirs does too.
richm31415 6y ago
Or eschew config files altogether and write operators in golang . . . https://sdk.operatorframework.io/ https://sdk.operatorframework.io/
- SomaticPirate 6y ago
  Still have to write a CRD if you want to do anything useful (arguably still requires a config).
tootie 6y ago
If you're doing more than storing config values in YAML, you're playing with fire. I've really grown to dislike the trend in "developing by descriptor". It's great when it works and impossible to debug when anything goes wrong. And also, not easily testable.
creztoe 6y ago
Could you explain why helm is garbage? I think it suits its purpose rather well without being too complex. You can essentially "plug-in" different types of resources rather easily. Especially in v3 now that you don't need to install Tiller and can avoid setting those cluster permission requirements. Have you tried some Kubernetes api libraries? You can generate and configure resources with [python kubernetes-client](https://github.com/kubernetes-client/python https://github.com/kubernetes-client/python) without much trouble. Personally I prefer editing them as JSON instead of python objects, but it isn't too bad.
- Nullabillity 6y ago
  > Could you explain why helm is garbage? Not the OP, but.. 1. YAML string templating makes it very easy to get indentation and/or quotation wrong, and the error messages can easily end up pretty far from the actual errors. Structured data should be generated with structured templating. 2. "Values" aren't typechecked or cleaned. 3. Easy to end up in a state where a failed deploy leaves you with a mess to clean up by hand. 4. No good way to preview what a deploy will change. 5. Weird interactions when resources are edited manually (especially back in Helm 2, but still a thing). 6. No good way to migrate objects into a Helm chart without deleting and recreating them. 7. Tons of repetitive boilerplate in each chart to customize basic settings (like replica counts). It's a typical Go solution, in all the wrong ways.
  - alephu5 6y ago
    It's not going to solve all your problems but dhall can fix your first few gripes. I've been using it for several months and it's an excellent way to write configuration imo.
    - Nullabillity 6y ago
      Yeah, I have used Nix to generate them in the past, which worked pretty great too. But Helm does, admittedly, solve a real problem: garbage collecting old resources when they're deleted from the repo. I just wish we could have something much simpler that only did that...
dilyevsky 6y ago
Sounds like you may enjoy https://github.com/cruise-automation/isopod https://github.com/cruise-automation/isopod (some assembly may be required)
gscho 6y ago
You can't just say helm is garbage without saying why.
bfrydl 6y ago
I don't know if I think Helm is garbage but I feel like it's almost always more trouble than its worth. Most charts can be boiled down to just a couple resources and instead of sifting through its weirdo templates you might as well just look at the actual resource configs. It's almost always just a single Deployment or similar + RBAC. I've actually been using Terraform to manage Kubernetes resources at my latest job, along with literally everything else. I don't even use alb-ingress-controller or external-dns or any of that. I just have Terraform make target groups from the service resources. It breaks a lot less.
- OJFord 6y ago
  But Helm barely adds any boilerplate over that 'single deployment or similar + RBAC'? Just a Chart.yaml (minimal of which is very small) and adding the option to parameterise or templatise the resources. I don't know why you wouldn't, we don't say 'I don't know if .deb/.rpm/PKGINFO/etc. is worth it, most packages are just a binary'. For Helm to add LoC you'd have to all but not use it, and even then it'd be about 3LoC. Setup is no harder, with N kubectl applies becoming 1 helm install. I also started using terraform (directly instead of just triggering helm) - are you using kubernetes tf resources directly though, or the helm provider?
- jjeaff 6y ago
  Most helm charts I have seen and used have a deployment yaml or statefulset or daemonset plus a service yaml, optional ingress yaml, sidecar containers for metrics, and several others I'm forgetting at the moment. And I would say most as in 90% of the charts on helm hub.
x87678r 6y ago
> Instead, I have to dig YAML with a pickaxe in the YAML-mine without any technology to assist me. Someone here talked about the joys of using the API with protobufs. Something I want to try out because it sounds clean.
aeontech 6y ago
Have you looked at Dhall or Tanka? They are both designed to solve exactly this problem (and I couldn’t agree with you more)
- dpc_pw 6y ago
  Honestly, I am not a fan of "configuration languages". Why do we need another bunch of languages. Were the existing languages not good enough? There's already shitload of them. The only place where I find it useful is when the config is provider by an untrusted party. But even then scheme, lua would probably fit the bill. Not strong feelings about it though, maybe I'm wrong. All I want is a library for my favorite language, where I can easily output bunch of `yaml` files to feed to k8s. I guess pulumi is kind of like that, but I never had a chance to try it.
  - kqr 6y ago
    I don't know about the others, but at least Dhall provides features that are literally impossible to get from a general purpose programming language, like semantic hashing, totality, and such.
  - kdkeyser 6y ago
    My experience with using general purpose languages for things like configuration or build systems has not been great. You always seem to end up with building a bunch of abstractions on top of that language, which culminates in another in-house configuration/build DSL, for which no public info is available. It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects. Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity. On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language. I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)
knocte 6y ago
> .. with code Use Pulumi, it does exactly this.
ecnahc515 6y ago
I generally agree. I'm using kustomize right now, but I'm also using helm charts via a kustomize plugin called chartInflator. It allows us to re-use existing upstream boilerplate while still using kustomize. Though I've not been entirely happy with kustomize either, and I'm thinking I might begin to investigate jsonnet for some things.
AlphaSite 6y ago
Carvel/k14s has been very nice for the usecase I had. Use ytt to generate the yams, using a yaml aware variant of skylark and kapp to manage the actual atomic deployment, delete, etc. It’s very simple and worked quite well for me. Disclaimer: work at VMware but on a different team (it’s an OS tool built by some of our engineers, which we use internally)
kkapelon 6y ago
> What I'd like is to generate serialized yaml files ... you know ... with code Check out Pulumi (I am not affiliated with them in any way)
cbushko 6y ago
This is why I use Terraform for my kubernetes objects. Yes, it is a DSL and not perfect but it has many advantages over helm such as: - State. I cannot express how useful having the desired and actual state of a system is. It makes it so much easier to determined what has changed in between terraform applies which in turn makes tracking down problems easier. - Providers. Being able to chain resources together from other systems is very powerful. As an example, I can: 1) create a database with a random name 2) create a user/password for that database 3) save the data above as a kubernetes secret 4) pass that secret into my kubernetes container for the service to use. And you can easily change database to be anything such as a storage bucket, dns entry, twilio account, stripe account, etc. Now try and do the exact same thing with helm! - Providers Part 2: You can write your own providers to talk to your OWN services. I have done this and written up some go code to talk to our internal APIs when setting up new customers. It is very easy to create your own terraform resources to do this. There are more reasons but I have work to do :)

ypeter 6y ago

Two benefits: - I already use Helm to install open source software; there are already so many Charts available. So that makes it easy to also use Helm for my own apps (I use it just for templating though) - Helm templating language is more flexible, but that can also bite you since it can become complex. With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching)

snuxoll 6y ago
> With Kustomize, changing a single environment variable value in a Deployment is quite some work compared to Helm (I don’t like JSON patching) Config maps, enough said. I have a CD pipeline that writes environment variables to files, Kustomize eats them up and creates a configmap, environment variables in the deployment reference the config map, done. Kapp as a layer on top of Kustomize is showing some progress. Ultimately Kustomize is like every other resource bundled with k8s, it’s a building block for higher level tooling.

cpaika 6y ago

Helm is amazing. If you are migrating legacy applications that rely on filesystem config, you can publish that config in a helm chart that applications require as sub charts. Gives you the ability to do centralized, versioned config management for legacy workloads going to the cloud Declarative paradigms get a bad rap but I really just believe its people who missing imperative/functional programmimg. Theres a reason the infra space has been using declarative syntax for years now

magnawave 6y ago
The declarative functionality comes from K8s not helm. (Of course K8s has imperative tooling too but that’s not the point here). The main reason folks dislike helm are 2 fold: 1) using string templating on a space formatted language is just messy and ugly for no good reason. If you don’t do anything terribly complicated it’s not too bad but quickly can be pretty fragile and unreadable. 2) golang(which I like generally) but the templating is messy and adds to the hard to read later problem Personally find helm’s opportunity for code reuse to be restrictive and folks just end up cut and pasting a lot of declarations all over which is ... so terrible. But the first bullet above is pretty a fundamental design flaw.

gravypod 6y ago

Helm is probably the worst influence I've experienced on the kubernetes ecosystem. It takes something that could have been really good (metaconfig) and makes it illegable, complex, and impossible to debug. Kustomize, too, doesn't get at the root of all the needs I've found myself having. A metaconfig language like Jsonnet, Dhall, or Cuelang seems to be where it's at. If you're using jsonnet I can also highly recommend kubecfg [0] as it's a tool that "just works" and is built at the perfect level of abstraction: "just give me something and I'll find all the kubernetes objects inside of it". It's remarkably simple and forgiving and it makes it possible to describe even the most complex application configs + kube configs in a single language. You can also take it a step further and describe your CI stages (pre-merge, release, builds, etc), kube yaml, and infrastructure provisioning (terraform) all in one language. > What are the benefits of using helm? The main benefit is redistribution. Since everyone uses helm it's easy to give someone a chart because they know what to do with it. That's pretty much the only upside I've found. [0] - https://github.com/bitnami/kubecfg https://github.com/bitnami/kubecfg

SomaticPirate 6y ago
Dhall continues to a be a glimmer of hope in this space. I agree that helm caused the entire space to distort. There is so little reusability in charts. Dhall seems to be the start of reversing this, but I hesitate to mention it since the ergonomics are still rough. If YAML is the assembly language of k8s, Dhall looks like what you would write the compiler in.
- gravypod 6y ago
  Yea, anything that allows you to think in a higher-level than raw kube objects will save you a lot of time and heartache. For instance, if your application needs a redis cluster in helm you have to mess with subcharts/other headaches. In jsonnet it becomes `redis = import "redis"; redis.Cluster("my-app-name") { nodes: 10 }` or something similar. DRY configs make me very happy.
- iso8859-1 6y ago
  Given that the Dhall type system supports things that the other configuration languages do not, isn't it very hard to migrate to for a project like k8s? I don't see how they could support something in parallel to Dhall, and still enable all the features Dhall provides.

theptip 6y ago

Helm is good for importing other components (eg Redis). I find it pretty tedious for authoring my own Yaml specs; any time you need to do some sort of composition you have to write templates, and then mess with indentation operators, because significant white space. Templating yaml is a pretty bad idea in my experience. As of Helm3 the most egregious part (Tiller) is gone, so it’s more recommendable. Scaffold lets you write (roughly) native k8s yaml files and layer them together. IDEs know the schema for this yaml at this point, so you get code completion on all your input files. This breaks in Helm. Scaffold is good for simple apps, and falls down a bit if you need to dynamically configure your specs (eg render a parameterized manifest for each review app). The programming model for scaffold is a bit hard to grok, based on my experience of ramping up a team on it. Not rocket science just a bit unintuitive at first.

pnathan 6y ago

Helm is a wannabe package manager, designed for people who really like writing yaml instead of executable code. K8s doesn't really have a fully built out "apps" resource, so that starts the whole process down a gravelly slope of pain... with initial care and if your team need to groom their k8s configs, you can go a long way with helm before the pain starts. A particular problem with helm is that it isn't really a fully fledged programming language, but a glue laid over the golang templating system. I use helm as little as possible. Usually only enough to template out a single file with the different bits of k8s infra stuck together with `---`. If I was doing some relatively serious work on custom services with minimal needs to conform to best^H^H^H^Hpopular practices, my service's configuration would be a neatly packaged Java program (its libraries for K8s direct API access are good). There are different approaches to handling ops tasks; I favor the ones that are rooted in writing executable code over the ones with yaml. Some bright chappie will probably write a Common Lisp library derivation library and k8s will be obvious and simple with that interface, but I'm not that bright - all I can see is the potential.

jarv 6y ago

Blog author here - we are using Helm for our application because of how much we have already invested in our public Helm chart. We are not sold on using Helm however for K8s in general and are currently evaluating Tanka for some of our monitoring infrastructure. One of our engineers wrote up some notes on that here if you are curious https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11260 https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issu...

kkapelon 6y ago

Helm is a full package manager. Kustomize is just a templating mechanism. They are not comparable. For example, with Helm you can rollback applications to previous versions