4 ms·

Am I the only one who's pessimistic about this? One of the big upsides of Kubernetes is having portable workloads and provisioning cloud-provider specific resou

Am I the only one who's pessimistic about this? One of the big upsides of Kubernetes is having portable workloads and provisioning cloud-provider specific resources (whose lifecycles very likely outlive clusters!) in Kubernetes just seems wrong to me. Kubernetes is great for managing, orchestrating and directing traffic for containerized workloads but it really shouldn't be The One Tool For Everything. Coupling everything together like this just seems to make things less manageable. IMO infrastructure including managed services are better provisioned through tools like Terraform and Pulumi.

solatic 6y ago
The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool. Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.
- redwood 6y ago
  This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon
  - t3rabytes 6y ago
    They used to have a managed Vault offering! But then it disappeared one day never to return.
- Legogris 6y ago
  This issue is to a large extent addressed by Terraform Cloud.
  - solatic 6y ago
    Terraform Cloud runs the executable for you on the basis of VCS triggers e.g. open pull requests, merging to master. Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.
- multani 6y ago
  I think this was the promise of Docker's InfraKit [1] (now DeployKit), to have your infrastructure constantly monitored and adjusted depending on the configuration you pushed into it. Sadly it didn't go anywhere and it's now archived. [1] : https://github.com/docker/deploykit https://github.com/docker/deploykit
- 013a 6y ago
  At least with many Kubernetes controllers that have been built in the past (I haven't tried ACK), they don't respond to or attempt to "live-correct" changes in configuration which are made outside of Kubernetes (e.g. via the AWS UI). They act on events that are fired in the cluster when a resource they monitor changes in the cluster. So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never. In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc. In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted. (Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.
  - kragniz 6y ago
    I haven't checked the ACK code to see what they do, but this is what the resync period is for in controllers. Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files https://github.com/kubernetes/kubernetes/pull/75423/files
    - 6y ago
vergessenmir 6y ago
You raise a good point on the chicken and egg scenario of having the K8S cluster created in AWS creating other AWS resources. Rebuilding everything from scratch will take a lot longer since you can't build concurrently. The advantages of pulumi and terraform over this also come from HCL and the Pulumi DSL _and_ their respective state files. This AWS controller only has the state through the API objects. Managing that is a pain because the organisation of those objects is left as an exercise for the user or other tools. Pulumi and Terraform constrain that to well thought-out DSLs. If there was a sane unifying DSL for managing kubernetes objects it would certainly simplify things. It would not, though, solve the speed of rebuilding the infrastructure.
nhoughto 6y ago
Yeah always been my thinking with aws or similar operators. Lifecycle of resources has to be stacked like a pyramid.. I don’t really want the definition of a bucket or a database(!) to live as a crd in a cluster. I’m much more likely to destructively update a cluster than a bucket. Coupling then doesn’t seem to make sense to me. Some upsertable stateless resources would be nice tho, like having a crd to create an iam role for a deployment would be neat, so I can deploy a workload and define a role and policy attachments using the same mechanism, works where the crd defines the entire scope of the resources, not just create a bucket that can be filled with stuff. Begs the question of the threat modeling tho, the aws controller has god privileges in the account? If I get to use my iam crd to provision iam resources for me (super helpful) then aws controller needs high IAM privs. So now kube cluster compromise could be escalated to aws account compromise, not really least privilege. Convenient though... But then also iam isn’t supported so ️
streetcat1 6y ago
So the vision behind operators is not what is managed but who is managing. The idea is to move operations and provisioning from humans to machines. What should exist, is a set of standards that defines generic cloud resources, and than each cloud provider would implement a translation operator from the standard to its own CRD. Of course, the leading cloud provider has no incentive to do that since this would reduce lock in. For example, there should be a VirtualBucket CRD which would map to AWS Bucket, Azure Container etc. Terraform was first into this area, but terraform is still manual in nature. I think that Pulumi uses that same code generation as Arc.
theptip 6y ago
Regarding lifecycle, most larger shops are going to have an infra team that owns “bedrock infrastructure” terraform config for things like VPCs, IAM, maybe project structure. Then each team will have their own terraform scripts for k8s cluster, DBaaS, queues, etc - the application infrastructure. This tool is great for the latter case; give me a k8s cluster and I can stand up all of my services in a one-shot command, with a single tool chain. TBD whether you’ll see infra teams moving their terraform scripts into an infra k8s cluster. I’m inclined to believe it’s less of a good fit there, as your bedrock should not change often, so the marginal value over terraform is less.

solatic 6y ago

The issue (or benefit, depending on your perspective) with Terraform is that it's a one-shot CLI binary. If you're not running the binary, then it's not doing anything. If you want a long-running daemon that responds to non-human-initiated events, then Terraform isn't a good tool. Any time you try to declaratively define state, if you don't have a daemon enforcing the declarative state, then you will suffer from drift. One approach is the one Terraform has - assume that drift is possible, so ask the user to run a separate plan stage, and manually reconcile drift if needed. Another approach is the controller approach, where the controller tries to do the right thing and errors/alerts if it doesn't know what to do.

redwood 6y ago
This is why Hashicorp needs to accelerate their cloud offering. Frankly I get the sense they got a little bit too addicted to central ops driven on-prem style deals for Vault but in the public cloud they need to be front and center with SaaS which is a long road. They have a rudimentary Terraform SaaS I believe but none for Vault as far as I'm aware. I see a lot of folks going straight to cloud provider services because of this. You sum it up well... In these times you don't want to run a daemon
- t3rabytes 6y ago
  They used to have a managed Vault offering! But then it disappeared one day never to return.
Legogris 6y ago
This issue is to a large extent addressed by Terraform Cloud.
- solatic 6y ago
  Terraform Cloud runs the executable for you on the basis of VCS triggers e.g. open pull requests, merging to master. Terraform Cloud does not know to spontaneously run a plan and apply when a bucket described by your infrastructure code is surreptitiously deleted by somebody in a team on the other side of the ocean.
multani 6y ago
I think this was the promise of Docker's InfraKit [1] (now DeployKit), to have your infrastructure constantly monitored and adjusted depending on the configuration you pushed into it. Sadly it didn't go anywhere and it's now archived. [1] : https://github.com/docker/deploykit https://github.com/docker/deploykit
013a 6y ago
At least with many Kubernetes controllers that have been built in the past (I haven't tried ACK), they don't respond to or attempt to "live-correct" changes in configuration which are made outside of Kubernetes (e.g. via the AWS UI). They act on events that are fired in the cluster when a resource they monitor changes in the cluster. So, again I haven't tried ACK, but they don't help to correct the drift that you outline as a problem. They will take action to correct the drift if the internal resource changes, but how often do you update the desired configuration of an S3 bucket? For me, almost never. In essence, the controller acts less like a live daemon always keeping things in sync, and more like Terraform in a CI pipeline. And, given you've probably got all your Kubernetes YAMLs inside a git repo anyway, all you've accomplished by deploying one of these things is trading a predictable, easy to debug step in a CI pipeline for a live running service inside your cluster that is in perpetual beta, could stop running, have bugs, could get evicted, uses cluster resources, doesn't talk to Github automatically, etc. In fact, you'll often times even get drift upon internal resource updates, in situations where the controller is too spooked to make a change that could be destructive. We've seen this with the ALB Ingress Controller, where it never deletes ALBs, even if the entire underlying service, ingress, etc are deleted. (Edit): To be clear: I think the direction of "specify everything in kubeyaml" could end up being a win for the infra world. If we could throw out a terraform+kubeyaml system for just "everything in kubeyaml", that feels like a simplification. But, I'm not convinced that the best way to get that kubeyaml into AWS is via live-running code inside clusters, especially since the complexity of AWS means its literally never going to work right (they can't even get CloudFormation to work right, and that's a managed service). A live-running controller is necessary for some things, like ALB Ingress, due to how quickly those changes need to be made (updating the ALB with new IP/ports of containers). But, for other things like S3/SQS/etc, I'm less sure.
- kragniz 6y ago
  I haven't checked the ACK code to see what they do, but this is what the resync period is for in controllers. Here's a comment with some discussion: https://github.com/kubernetes/kubernetes/pull/75423/files https://github.com/kubernetes/kubernetes/pull/75423/files
  - 6y ago

vergessenmir 6y ago

You raise a good point on the chicken and egg scenario of having the K8S cluster created in AWS creating other AWS resources. Rebuilding everything from scratch will take a lot longer since you can't build concurrently. The advantages of pulumi and terraform over this also come from HCL and the Pulumi DSL _and_ their respective state files. This AWS controller only has the state through the API objects. Managing that is a pain because the organisation of those objects is left as an exercise for the user or other tools. Pulumi and Terraform constrain that to well thought-out DSLs. If there was a sane unifying DSL for managing kubernetes objects it would certainly simplify things. It would not, though, solve the speed of rebuilding the infrastructure.

nhoughto 6y ago

Yeah always been my thinking with aws or similar operators. Lifecycle of resources has to be stacked like a pyramid.. I don’t really want the definition of a bucket or a database(!) to live as a crd in a cluster. I’m much more likely to destructively update a cluster than a bucket. Coupling then doesn’t seem to make sense to me. Some upsertable stateless resources would be nice tho, like having a crd to create an iam role for a deployment would be neat, so I can deploy a workload and define a role and policy attachments using the same mechanism, works where the crd defines the entire scope of the resources, not just create a bucket that can be filled with stuff. Begs the question of the threat modeling tho, the aws controller has god privileges in the account? If I get to use my iam crd to provision iam resources for me (super helpful) then aws controller needs high IAM privs. So now kube cluster compromise could be escalated to aws account compromise, not really least privilege. Convenient though... But then also iam isn’t supported so ️

streetcat1 6y ago

So the vision behind operators is not what is managed but who is managing. The idea is to move operations and provisioning from humans to machines. What should exist, is a set of standards that defines generic cloud resources, and than each cloud provider would implement a translation operator from the standard to its own CRD. Of course, the leading cloud provider has no incentive to do that since this would reduce lock in. For example, there should be a VirtualBucket CRD which would map to AWS Bucket, Azure Container etc. Terraform was first into this area, but terraform is still manual in nature. I think that Pulumi uses that same code generation as Arc.

theptip 6y ago

Regarding lifecycle, most larger shops are going to have an infra team that owns “bedrock infrastructure” terraform config for things like VPCs, IAM, maybe project structure. Then each team will have their own terraform scripts for k8s cluster, DBaaS, queues, etc - the application infrastructure. This tool is great for the latter case; give me a k8s cluster and I can stand up all of my services in a one-shot command, with a single tool chain. TBD whether you’ll see infra teams moving their terraform scripts into an infra k8s cluster. I’m inclined to believe it’s less of a good fit there, as your bedrock should not change often, so the marginal value over terraform is less.