4 ms·

A migration with the goal of improving the infrastructure foundation is great. However, I was surprised to see that one of the motivations was to allow teams to

by wrs 2y ago

A migration with the goal of improving the infrastructure foundation is great. However, I was surprised to see that one of the motivations was to allow teams to use Helm charts rather than converting to Terraform. I haven’t seen in practice the consistent ability to actually use random Helm charts unmodified, so by encouraging its use you end up with teams forking and modifying the charts. And Helm is such a horrendous tool, you don’t really want to be maintaining your own bespoke Helm charts. IMO you’re actually better off rewriting in Terraform so at least your local version is maintainable. Happy to hear counterexamples, though — maybe the “indent 4” insanity and multi-level string templating in Helm is gone nowadays?

smellybigbelly 2y ago
Our team also suffered from the problems you described of public helm charts. There is always something you need to customise to make things work on your own environment. Our approach has been to use the public helm chart as-is and do any customisation with `kustomize —enable-helm`.
BobbyJo 2y ago
Helm is quite often the default supported way of launching containerized third-party products. I have works at two separate startups whose 'on prem' product was offered this way.
- freedomben 2y ago
  Indeed. I try hard to minimize the amount of Helm we use, but a significant amount of tools are only shipped as Helm charts. Fortunately I'm increasingly seeing people provide "raw k8s" yaml, but it's far from universal.
cwiggs 2y ago
Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not. I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places. Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.
- wrs 2y ago
  I agree, I wouldn’t generate k8s from Terraform either, that’s just the alternative I thought the OP was presenting. But I’d still rather convert charts from Helm to pretty much anything else than maintain them.
- stackskipton 2y ago
  Lack of Variable substitution in Kustomize is downright frustrating. We use Flux so we have the feature anyways, but I wish it was built into Kustomize.
  - no_circuit 2y ago
    I don't miss variable substitution at all. For my setup anything that needs to be variable or secret gets specified in a custom json/yaml file which is read by a plugin which in turn outputs the rendered manifest if I can't write it as a "patch". That way the CI/CD runner can access things like the resolved secrets for production without being accessible by developers without elevated access. It requires some digging but there are even annotations that can be used to control things like if Kustomize should add a hash suffix or not to ConfigMap or Secret manifests you generate with plugins.
  - hronecviktor 2y ago
    Trivial to work around using _envsubst_ utility (part of _gettext_). If you write something like: name: app-$KUSTOMIZE_NAMESPACE and run `kubectl kustomize | envsubst "$(printf '${%s} ' ${!KUSTOMIZE_} ${!CI_} ${!GITLAB_*})" | kubectl apply -f -` it will substitute the env vars that match the wildcard.
- tionate 2y ago
  Re your kustomizen complaint, just create a complete env-specific ingress for each env instead of patching. - it is not really any more lines - doesn’t break if dev upgrades to a different version of the resource (has happened before) - allows you to experiment with dev with other setups (eg additional ingresses, different paths etc) instead of changing a base config which will impact other envs TLDR patch things that are more or less the same in each env; create complete resources for things that change more. There is a bit of duplication but it is a lot more simple (see ‘simple made easy’ - rich hockey) than tracing through patches/templates.
gouggoug 2y ago
Talking about helm - I personally have come to profoundly loathe it. It was amazing when it came out and filled a much needed gap. However it is loaded with so many footguns that I spend my time redoing and debugging others engineers work. I’m hoping this new tool called « timoni » picks up steam. It fixes pretty every qualm I have with helm. So if like me you’re looking for a better solution, go check timoni.
- roryokane 2y ago
  Link: https://timoni.sh/ https://timoni.sh/
JohnMakin 2y ago
It's completely cursed, but I've started deploying helm via terraform lately. Many people, ironically me included, find that managing deployments via terraform is an anti pattern. I'm giving it a try and I don't despise it yet, but it feels gross - application configs are typically far more mutable and dynamic than cloud infrastructure configs, and IME, terraform does not likey super dynamic configs.
solatic 2y ago
My current employer (BigCo) has Terraform managing both infra + deployments in Terraform, at (ludicrous) scale. It's a nightmare. The problem with Terraform is that you must plan your workspaces such that you will not exceed the best-practice amount of resources per workspace (~100-200) or else plans will drastically slow down your time-to-deploy, checking stuff like databases and networking that you haven't touched and have no desire to touch. In practice this means creating a latticework of Terraform workspaces that trigger each other, and there are currently no good open-source tools that support it. Best practice as I can currently see it is to have Terraform set up what you need for continuous delivery (e.g. ArgoCD) as part of the infrastructure, then use the CD tool to handle day-to-day deployments. Most CD tooling then asks you to package your deployment in something like Helm.
- chucky_z 2y ago
  You can setup dependent stacks in CDKTF. It’s far from as clean as a standard TF DAG plan/apply but I’m having a lot of success with it right now. If I were actively using k8s at the moment I would probably setup dependent cluster resources using this method, e.g: ensure a clean, finished CSI daemon deployment before deploying a deployment using that CSI provider :)
  - solatic 2y ago
    You're right that CDKTF with dependent stacks is probably better than nothing, but (a) CDKTF's compatibility with OpenTofu depends on a lack of breaking changes in CDKTF, since the OpenTofu team didn't fork CDKTF, so this is a little hairy for building production infrastructure; (b) CDKTF stacks, even when they can run in parallel, still run on the same machine that invoked CDKTF. When you have (ludicrous scale) X number of "stacks", this isn't a good fit. It's something that should be doable in one of the managed Terraform services, but the pricing if you try to do (ludicrous scale) parallelism gets to be insane.
mnahkies 2y ago
Whilst I'll agree that writing helm charts isn't particularly delightful, consuming them can be. In our case we have a single application/service base helm chart that provides sane defaults and all our deployments extend from. The amount of helm values config required by the consumers is minimal, and there has been very little occasion for a consumer to include their own templates - the base chart exposes enough knobs to avoid this. When it comes to third-party charts, many we've been able to deploy as is (sometimes with some PRs upstream to add extra functionality), and occasionally we've needed to wrap/fork them. We've deployed far more third-party charts as-is than not though. One thing probably worth mentioning w.r.t to maintaining our custom charts is the use of helm unittest (https://github.com/helm-unittest/helm-unittest https://github.com/helm-unittest/helm-unittest) - it's been a big help to avoid regressions. We do manage a few kubernetes resources through terraform, including Argocd (via the helm provider which is rather slow when you have a lot of CRDs), but generally we've found helm chart deployed through Argocd to be much more manageable and productive.
brainzap 2y ago
Helm charts are a bit painful but they have a few critical features. Atomic deploy that rolls back onfailure. Ability to generate the full kubernetes definition with helm template. Ability to print out all configuration values with description. At our company we have all deployments wrapped into a flat helm chart with as little variables as possible. (I always have to fight for that because devs like to abstract helm 100 levels and end up with nothing)

by wrs 2y ago

smellybigbelly 2y ago

Our team also suffered from the problems you described of public helm charts. There is always something you need to customise to make things work on your own environment. Our approach has been to use the public helm chart as-is and do any customisation with `kustomize —enable-helm`.

BobbyJo 2y ago

Helm is quite often the default supported way of launching containerized third-party products. I have works at two separate startups whose 'on prem' product was offered this way.

freedomben 2y ago
Indeed. I try hard to minimize the amount of Helm we use, but a significant amount of tools are only shipped as Helm charts. Fortunately I'm increasingly seeing people provide "raw k8s" yaml, but it's far from universal.

cwiggs 2y ago

Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not. I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places. Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.

wrs 2y ago
I agree, I wouldn’t generate k8s from Terraform either, that’s just the alternative I thought the OP was presenting. But I’d still rather convert charts from Helm to pretty much anything else than maintain them.
stackskipton 2y ago
Lack of Variable substitution in Kustomize is downright frustrating. We use Flux so we have the feature anyways, but I wish it was built into Kustomize.
- no_circuit 2y ago
  I don't miss variable substitution at all. For my setup anything that needs to be variable or secret gets specified in a custom json/yaml file which is read by a plugin which in turn outputs the rendered manifest if I can't write it as a "patch". That way the CI/CD runner can access things like the resolved secrets for production without being accessible by developers without elevated access. It requires some digging but there are even annotations that can be used to control things like if Kustomize should add a hash suffix or not to ConfigMap or Secret manifests you generate with plugins.
- hronecviktor 2y ago
  Trivial to work around using _envsubst_ utility (part of _gettext_). If you write something like: name: app-$KUSTOMIZE_NAMESPACE and run `kubectl kustomize | envsubst "$(printf '${%s} ' ${!KUSTOMIZE_} ${!CI_} ${!GITLAB_*})" | kubectl apply -f -` it will substitute the env vars that match the wildcard.
tionate 2y ago
Re your kustomizen complaint, just create a complete env-specific ingress for each env instead of patching. - it is not really any more lines - doesn’t break if dev upgrades to a different version of the resource (has happened before) - allows you to experiment with dev with other setups (eg additional ingresses, different paths etc) instead of changing a base config which will impact other envs TLDR patch things that are more or less the same in each env; create complete resources for things that change more. There is a bit of duplication but it is a lot more simple (see ‘simple made easy’ - rich hockey) than tracing through patches/templates.

gouggoug 2y ago

Talking about helm - I personally have come to profoundly loathe it. It was amazing when it came out and filled a much needed gap. However it is loaded with so many footguns that I spend my time redoing and debugging others engineers work. I’m hoping this new tool called « timoni » picks up steam. It fixes pretty every qualm I have with helm. So if like me you’re looking for a better solution, go check timoni.

roryokane 2y ago
Link: https://timoni.sh/ https://timoni.sh/

JohnMakin 2y ago

It's completely cursed, but I've started deploying helm via terraform lately. Many people, ironically me included, find that managing deployments via terraform is an anti pattern. I'm giving it a try and I don't despise it yet, but it feels gross - application configs are typically far more mutable and dynamic than cloud infrastructure configs, and IME, terraform does not likey super dynamic configs.

solatic 2y ago

My current employer (BigCo) has Terraform managing both infra + deployments in Terraform, at (ludicrous) scale. It's a nightmare. The problem with Terraform is that you must plan your workspaces such that you will not exceed the best-practice amount of resources per workspace (~100-200) or else plans will drastically slow down your time-to-deploy, checking stuff like databases and networking that you haven't touched and have no desire to touch. In practice this means creating a latticework of Terraform workspaces that trigger each other, and there are currently no good open-source tools that support it. Best practice as I can currently see it is to have Terraform set up what you need for continuous delivery (e.g. ArgoCD) as part of the infrastructure, then use the CD tool to handle day-to-day deployments. Most CD tooling then asks you to package your deployment in something like Helm.

chucky_z 2y ago
You can setup dependent stacks in CDKTF. It’s far from as clean as a standard TF DAG plan/apply but I’m having a lot of success with it right now. If I were actively using k8s at the moment I would probably setup dependent cluster resources using this method, e.g: ensure a clean, finished CSI daemon deployment before deploying a deployment using that CSI provider :)
- solatic 2y ago
  You're right that CDKTF with dependent stacks is probably better than nothing, but (a) CDKTF's compatibility with OpenTofu depends on a lack of breaking changes in CDKTF, since the OpenTofu team didn't fork CDKTF, so this is a little hairy for building production infrastructure; (b) CDKTF stacks, even when they can run in parallel, still run on the same machine that invoked CDKTF. When you have (ludicrous scale) X number of "stacks", this isn't a good fit. It's something that should be doable in one of the managed Terraform services, but the pricing if you try to do (ludicrous scale) parallelism gets to be insane.

mnahkies 2y ago

Whilst I'll agree that writing helm charts isn't particularly delightful, consuming them can be. In our case we have a single application/service base helm chart that provides sane defaults and all our deployments extend from. The amount of helm values config required by the consumers is minimal, and there has been very little occasion for a consumer to include their own templates - the base chart exposes enough knobs to avoid this. When it comes to third-party charts, many we've been able to deploy as is (sometimes with some PRs upstream to add extra functionality), and occasionally we've needed to wrap/fork them. We've deployed far more third-party charts as-is than not though. One thing probably worth mentioning w.r.t to maintaining our custom charts is the use of helm unittest (https://github.com/helm-unittest/helm-unittest https://github.com/helm-unittest/helm-unittest) - it's been a big help to avoid regressions. We do manage a few kubernetes resources through terraform, including Argocd (via the helm provider which is rather slow when you have a lot of CRDs), but generally we've found helm chart deployed through Argocd to be much more manageable and productive.

brainzap 2y ago

Helm charts are a bit painful but they have a few critical features. Atomic deploy that rolls back onfailure. Ability to generate the full kubernetes definition with helm template. Ability to print out all configuration values with description. At our company we have all deployments wrapped into a flat helm chart with as little variables as possible. (I always have to fight for that because devs like to abstract helm 100 levels and end up with nothing)