5 ms·

Genuinely interested -- why would I choose this over running Terraform as part of CI/CD?

by kaidon 5y ago

dilyevsky 5y ago
We use it to almost fully replace terraform (still use it for some basic stuff) for couple reasons: * simplified tooling - you can use all the same tools you already use to manage your k8s configs * more standardized setup for multi-cloud/onprem * it’s based on vanilla kubeadm and cloud-init/systemd images - debugging problems is easier bc it’s all pretty standard, battle-tested tech
iwwr 5y ago
If you don't need to dive into the rabbit hole that is Hashicorp and Terraform and can manage with just clustetctl and kubectl. Or if you prefer yaml with no templating :) Terraform isn't a wrong decision, just that CNCF now has a more stable API/CLIs you can work with, with which Terraform also needs to integrate.
- ForHackernews 5y ago
  I guess it depends where you're starting from: I find the entire Hashicorp ecosystem much better documented, supported, more stable and generally less of a "rabbit hole" than the continuously shifting and expanding universe of k8s/"CNCF" projects. Unsurprising because all the Hashicorp stuff is from a single vendor, and there is the risk of lock-in.
redis_mlc 5y ago
k8s is the whole enchilada: IaaS, routing, container mgmt., etc. So terraform plus AWS ECS is roughly equivalent to k8s (or AWS ASGs and AMIs plus Docker.) A crude analogy is that k8s is the distributed systemd, and is the most portable. Saying "I know k8s." today generally also means you know the k8s ecosystem, including Istio, AWS Calico, etc. (I tell startups just to use EC2 and "yum update" as long as possible.) https://docs.aws.amazon.com/eks/latest/userguide/calico.html https://docs.aws.amazon.com/eks/latest/userguide/calico.html https://istio.io/ https://istio.io/
felixhuttmann 5y ago
If terraform crashes during apply, it leaves behind an inconsistent state by design: The lock is still set, and some resources which were created already are not yet in the statefile. Trying to re-run terraform after a crash during apply will generally lead to an error: Even if the lock is removed, resources may still conflict if they already exist. In contrast, when a kubernetes controller or operator crashes, it can be expected to continue seamlessly where it left off. It is easier to write kubernetes controllers that are able to continue seamlessly then to write terraform providers that do so, because of the granularity of the persistence of the state machine. Terraform locks the remote state, then applies all resources in the current root module, then unlocks the remote state again. In contrast, kubernetes operators can granularly update individual objects after each API call that is performed.
- klohto 5y ago
  I have never seen Terraform crash in the years I’ve been using it. Optimizing based on a lottery event seems counter-productive EDIT: Alright guys, good points all around. Mainly what I meant is the ratio of useful of Terraform vs its shortcomings. Not hating on the Cluster API pattern, but imho better to stick to standardized approach.
  - unbanned 5y ago
    "innovation"
  - vulkoingim 5y ago
    That's a very shortsighted comment. Just because you've never seen it happen, it doesn't mean that it doesn't. While terraform itself may not crash (maybe not in the manner that I assume you're referring), you are at the mercy of the implementation of the specific providers for which I've (and many others as well, I'm sure) seen plenty of issues. Even in "mature" ones as aws/gcp. Besides that, the control-loop pattern, which the parent comment describes is a very sound design, which is employed not only in kubernetes and has nothing to do with "Optimizing based on a lottery event".
    - 5y ago
znpy 5y ago
my understanding is that the machinery underlying it are a bit more complex than what terraform would do. if anything that would be because the kubernetes operators implementing the Cluster API for your specific cloud provider would be implemented in a continuously running control loop, thus picking up changes... whereas with terraform you would have to run terraform from time to time. edit: plus of course all the other features coming from kuernetes like mutating admission, validation, rbac (eg: only allow some people to create clusters or allow people to create clusters without granting access to the underlying cloud provider) etc etc...
spicyusername 5y ago
Many Kubernetes operators are sufficiently mature to use them to manage external infrastructure. When it works, it's nice to be able to deal with one templating language and one system to manage all of the elements of your deployments. Deploying a single helm chart is a nicer developer experience than managing both a helm chart and a terraform plan.
NexRebular 5y ago
or over running Joyent's Triton?

Genuinely interested -- why would I choose this over running Terraform as part of CI/CD?

by kaidon 5y ago

Genuinely interested -- why would I choose this over running Terraform as part of CI/CD?

We use it to almost fully replace terraform (still use it for some basic stuff) for couple reasons: * simplified tooling - you can use all the same tools you already use to manage your k8s configs * more standardized setup for multi-cloud/onprem * it’s based on vanilla kubeadm and cloud-init/systemd images - debugging problems is easier bc it’s all pretty standard, battle-tested tech

iwwr 5y ago

If you don't need to dive into the rabbit hole that is Hashicorp and Terraform and can manage with just clustetctl and kubectl. Or if you prefer yaml with no templating :) Terraform isn't a wrong decision, just that CNCF now has a more stable API/CLIs you can work with, with which Terraform also needs to integrate.

ForHackernews 5y ago
I guess it depends where you're starting from: I find the entire Hashicorp ecosystem much better documented, supported, more stable and generally less of a "rabbit hole" than the continuously shifting and expanding universe of k8s/"CNCF" projects. Unsurprising because all the Hashicorp stuff is from a single vendor, and there is the risk of lock-in.

redis_mlc 5y ago

k8s is the whole enchilada: IaaS, routing, container mgmt., etc. So terraform plus AWS ECS is roughly equivalent to k8s (or AWS ASGs and AMIs plus Docker.) A crude analogy is that k8s is the distributed systemd, and is the most portable. Saying "I know k8s." today generally also means you know the k8s ecosystem, including Istio, AWS Calico, etc. (I tell startups just to use EC2 and "yum update" as long as possible.) https://docs.aws.amazon.com/eks/latest/userguide/calico.html https://docs.aws.amazon.com/eks/latest/userguide/calico.html https://istio.io/ https://istio.io/

felixhuttmann 5y ago

If terraform crashes during apply, it leaves behind an inconsistent state by design: The lock is still set, and some resources which were created already are not yet in the statefile. Trying to re-run terraform after a crash during apply will generally lead to an error: Even if the lock is removed, resources may still conflict if they already exist. In contrast, when a kubernetes controller or operator crashes, it can be expected to continue seamlessly where it left off. It is easier to write kubernetes controllers that are able to continue seamlessly then to write terraform providers that do so, because of the granularity of the persistence of the state machine. Terraform locks the remote state, then applies all resources in the current root module, then unlocks the remote state again. In contrast, kubernetes operators can granularly update individual objects after each API call that is performed.

klohto 5y ago
I have never seen Terraform crash in the years I’ve been using it. Optimizing based on a lottery event seems counter-productive EDIT: Alright guys, good points all around. Mainly what I meant is the ratio of useful of Terraform vs its shortcomings. Not hating on the Cluster API pattern, but imho better to stick to standardized approach.
- unbanned 5y ago
  "innovation"
- vulkoingim 5y ago
  That's a very shortsighted comment. Just because you've never seen it happen, it doesn't mean that it doesn't. While terraform itself may not crash (maybe not in the manner that I assume you're referring), you are at the mercy of the implementation of the specific providers for which I've (and many others as well, I'm sure) seen plenty of issues. Even in "mature" ones as aws/gcp. Besides that, the control-loop pattern, which the parent comment describes is a very sound design, which is employed not only in kubernetes and has nothing to do with "Optimizing based on a lottery event".
  - 5y ago

znpy 5y ago

my understanding is that the machinery underlying it are a bit more complex than what terraform would do. if anything that would be because the kubernetes operators implementing the Cluster API for your specific cloud provider would be implemented in a continuously running control loop, thus picking up changes... whereas with terraform you would have to run terraform from time to time. edit: plus of course all the other features coming from kuernetes like mutating admission, validation, rbac (eg: only allow some people to create clusters or allow people to create clusters without granting access to the underlying cloud provider) etc etc...

spicyusername 5y ago

Many Kubernetes operators are sufficiently mature to use them to manage external infrastructure. When it works, it's nice to be able to deal with one templating language and one system to manage all of the elements of your deployments. Deploying a single helm chart is a nicer developer experience than managing both a helm chart and a terraform plan.

NexRebular 5y ago

or over running Joyent's Triton?