5 ms·

The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing a

The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool. Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.

redwood 6y ago
This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon
- t3rabytes 6y ago
  They used to have a managed Vault offering! But then it disappeared one day never to return.
Legogris 6y ago
This issue is to a large extent addressed by Terraform Cloud.
- solatic 6y ago
  Terraform Cloud runs the executable for you on the basis of VCS triggers e.g. open pull requests, merging to master. Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.
multani 6y ago
I think this was the promise of Docker's InfraKit [1] (now DeployKit), to have your infrastructure constantly monitored and adjusted depending on the configuration you pushed into it. Sadly it didn't go anywhere and it's now archived. [1] : https://github.com/docker/deploykit https://github.com/docker/deploykit
013a 6y ago
At least with many Kubernetes controllers that have been built in the past (I haven't tried ACK), they don't respond to or attempt to "live-correct" changes in configuration which are made outside of Kubernetes (e.g. via the AWS UI). They act on events that are fired in the cluster when a resource they monitor changes in the cluster. So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never. In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc. In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted. (Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.
- kragniz 6y ago
  I haven't checked the ACK code to see what they do, but this is what the resync period is for in controllers. Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files https://github.com/kubernetes/kubernetes/pull/75423/files
  - 6y ago

redwood 6y ago

This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon

t3rabytes 6y ago
They used to have a managed Vault offering! But then it disappeared one day never to return.

Legogris 6y ago

This issue is to a large extent addressed by Terraform Cloud.

solatic 6y ago
Terraform Cloud runs the executable for you on the basis of VCS triggers e.g. open pull requests, merging to master. Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.

multani 6y ago

I think this was the promise of Docker's InfraKit [1] (now DeployKit), to have your infrastructure constantly monitored and adjusted depending on the configuration you pushed into it. Sadly it didn't go anywhere and it's now archived. [1] : https://github.com/docker/deploykit https://github.com/docker/deploykit

013a 6y ago

At least with many Kubernetes controllers that have been built in the past (I haven't tried ACK), they don't respond to or attempt to "live-correct" changes in configuration which are made outside of Kubernetes (e.g. via the AWS UI). They act on events that are fired in the cluster when a resource they monitor changes in the cluster. So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never. In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc. In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted. (Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.

kragniz 6y ago
I haven't checked the ACK code to see what they do, but this is what the resync period is for in controllers. Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files https://github.com/kubernetes/kubernetes/pull/75423/files
- 6y ago