5 ms·

Keel does a really good job of this, good to see people are starting to have options with their k8 deployments! My two cents: So far my deployments have been

by _5meq 9y ago

Keel does a really good job of this, good to see people are starting to have options with their k8 deployments! My two cents: So far my deployments have been pretty painless, so changing over to something like Gitkube / keel isn't my top priority currently. Configuration management / Creating my kubernetes resources ( deployments / services / ingress / etc. ) in a repeatable and elegant way is my biggest pain point by far. # Helm I've been very unhappy with Helm. Helm is insanely verbose. Go ahead and take a look at the kubernetes/charts repo and look at the various stable charts. Note the huge amount of copy-pasta code hanging out. Too much of my life has been spent wrangling YAML templates that are rendered using the under-featured gotmpl library. Also, helm's client library support is less than stellar. Also, schlepping my configurations around is a pain. In Helm, it's a crazy amount of code to implement a `ingress -> service -> deployment` pattern that is standard for 99% of kubernetes resources. You have to write the same things over and over, which means that making changes takes forever and is brittle. I believe this is due to gotmpl being a poor templating engine for this use case. It doesn't have enough features to allow one to develop decent abstractions over declaring k8 resources The helm tiller doesn't expose a restful API or proper tooling to allow other resources to interact with it effectively, making automated deployments a chore. My only option is to call out to the shell and try and be devensive. Finally, Helm doesn't really do that great a job of validating my resources are going to be valid. `helm install --dry-run` will tell you everything is great, and then break in the middle of an actual installation, leaving a half-configured mess in it's wake. # KSonnet KSonnet looks like an attractive option, it's certainly less verbose. But the documentation isn't there yet, and if you peek under the hood there is a MOUNTAIN of ksonnet code waiting to be read and understood. I belive it's going through some churn currently, so features and impovements have been slow to appear. Also KSonnet doesn't have enough example implementations to hit the ground running when trying it out on my clusters. Targeting individual contexts in Ksonnet provides some nice additional safety, and some very simple implementations using google KMS or something similar could be a really special way to safely store my secret configs at rest, similar to ansible-vault. # Terraform Terraform has this NAILED in the VM space. I declare my environment: `terraform plan` tells me what is going to happen, `terraform apply` checks to ensure I've not protected any of my resources with a flag, and applies the changes it listed if not. I'd love to write my k8 resources in terraform, but it doesn't currently support modern k8 resources, and I don't LOVE how terraform handles storing my tfstate files either way.

meddlepal 9y ago
Yea I think I would love if Terraform supported modern Kubernetes concepts. The plan-apply model is IMO the best. I don't really grok why people don't think Terraform should be used this way as I've seen in other comments here and outside of HN. At the end of the day I fail to see how managing Kubernetes resources with Terraform would be any different than AWS or GCP or whathaveyou resources.
- antoncohen 9y ago
  Terraform is pretty awful for Continuous Delivery. I inherited a deployment system were new app versions were deployed with Terraform, and I did quite a lot of automation to make it not totally suck from a workflow perspective, not it's still not the right tool. Devs should be able to merge a PR, and have it safely rollout to production (e.g., canary), possibly with a staging step and manual promotion if desired. They should have a simple ability to rollback to a previous version. Terraform typically would involve editing TF files to update a version, committing that TF change, opening and merge a PR for the change. The running plan, reading the output of plan, then applying the plan. If you have staging and stable environments, repeat the steps. If you want to rollback, repeat the steps. And it gets oh so much more complex. TF binary versions need to be globally consistent, because the version is stored in the state file. You basically need to only run TF from the CI/CD system, people can't be allowed to run apply locally because they might use a newer version of TF which would then break CI/CD which has the older version. So then you wrap TF commands with a Makefile that checks TF version, or you pin the TF version within the TF code. How many resources are you managing? Hundreds? Thousands? How many apps? Are they all deployed using the same state file? How long does it take to run a plan and apply? 10 minutes? 20 minutes? Same for a rollback... What if Bob want to deploy version 13 of app 'foo', but when he runs plan it says it will also update app 'bar' to version 27. But Bob doesn't know anything about bar, his team only works on foo. So you break up the TF files so every app has its own state file. Cool. But now you need to upgrade TF itself. And it is a major upgrade. Now you need to update the TF version pinned in the source files, and update the build servers to have the new TF, and run a TF command to upgrade on every state file. Then you start adding symlinks into the TF source tree, because TF doesn't have any sane way to reuse code. But there are slight changes that need to be made for each app, and TF doesn't allow variables in some parts of the code, and modules are OK be still too verbose, and you still end up with places you can't use variables. So no you start using templates to write you TF code, so you can reuse code, because 99% is the same for each app. Ugh. Terraform is great for defining infrastructure that rarely changes, and is managed by an infrastructure team. When you are trying to use it for app deployments, which happen multiple times a day, and are handles by app devs, it is a royal PITA.

by _5meq 9y ago

meddlepal 9y ago

Yea I think I would love if Terraform supported modern Kubernetes concepts. The plan-apply model is IMO the best. I don't really grok why people don't think Terraform should be used this way as I've seen in other comments here and outside of HN. At the end of the day I fail to see how managing Kubernetes resources with Terraform would be any different than AWS or GCP or whathaveyou resources.

antoncohen 9y ago
Terraform is pretty awful for Continuous Delivery. I inherited a deployment system were new app versions were deployed with Terraform, and I did quite a lot of automation to make it not totally suck from a workflow perspective, not it's still not the right tool. Devs should be able to merge a PR, and have it safely rollout to production (e.g., canary), possibly with a staging step and manual promotion if desired. They should have a simple ability to rollback to a previous version. Terraform typically would involve editing TF files to update a version, committing that TF change, opening and merge a PR for the change. The running plan, reading the output of plan, then applying the plan. If you have staging and stable environments, repeat the steps. If you want to rollback, repeat the steps. And it gets oh so much more complex. TF binary versions need to be globally consistent, because the version is stored in the state file. You basically need to only run TF from the CI/CD system, people can't be allowed to run apply locally because they might use a newer version of TF which would then break CI/CD which has the older version. So then you wrap TF commands with a Makefile that checks TF version, or you pin the TF version within the TF code. How many resources are you managing? Hundreds? Thousands? How many apps? Are they all deployed using the same state file? How long does it take to run a plan and apply? 10 minutes? 20 minutes? Same for a rollback... What if Bob want to deploy version 13 of app 'foo', but when he runs plan it says it will also update app 'bar' to version 27. But Bob doesn't know anything about bar, his team only works on foo. So you break up the TF files so every app has its own state file. Cool. But now you need to upgrade TF itself. And it is a major upgrade. Now you need to update the TF version pinned in the source files, and update the build servers to have the new TF, and run a TF command to upgrade on every state file. Then you start adding symlinks into the TF source tree, because TF doesn't have any sane way to reuse code. But there are slight changes that need to be made for each app, and TF doesn't allow variables in some parts of the code, and modules are OK be still too verbose, and you still end up with places you can't use variables. So no you start using templates to write you TF code, so you can reuse code, because 99% is the same for each app. Ugh. Terraform is great for defining infrastructure that rarely changes, and is managed by an infrastructure team. When you are trying to use it for app deployments, which happen multiple times a day, and are handles by app devs, it is a royal PITA.