7 ms·

AWS Controllers for Kubernetes

111 points by yonasb 6y ago

buzer 6y ago
Some prior discussion: https://news.ycombinator.com/item?id=24219448 https://news.ycombinator.com/item?id=24219448
sytse 6y ago
Kelsey Hightower had a good take in https://twitter.com/kelseyhightower/status/1296311951530704896 https://twitter.com/kelseyhightower/status/12963119515307048... "AWS Controllers for Kubernetes is pretty dope. You can leverage Kubernetes to manage AWS resources such as API gateways and S3 buckets. Think Terraform but backed by Kubernetes style APIs and "realtime" control loops." An in the thread he mentions Crossplane as the cross-cloud way to do this https://twitter.com/kelseyhightower/status/1296321377134231552 https://twitter.com/kelseyhightower/status/12963213771342315...
- rektide 6y ago
  Every Kubernetes shop I've seen uses Helm to deploy their workloads, & now you can set up all your S3 Buckets, SQS, SNS stuff in that same Helm chart as your app. This is hella ideal. I also just generally love the thought of being able to manage any cloud resources at all via standard open-protocols & systems. Compounding the investment, rather than having to invest in a bunch of specific not-interconnected areas is going to lead to great things. It'll be interesting to see what if any architectural flourishes or innovations went in to ACK's control loops.
  - etxm 6y ago
    It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?
    - rektide 6y ago
      metadata: annotations: helm.sh/resource-policy: keep works for any & all resources!
    - jrockway 6y ago
      Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RDS instances would have similar protection. (You might think that the k8s world is very mutable, but there are some immutable things hanging around. For cases exactly like this.) I don't use AWS so I didn't look into this thoroughly, but the word "finalizer" does show up in the code frequently: // MarkManaged places the supplied resource under the management of ACK. // What this typically means is that the resource manager will decorate the // underlying custom resource (CR) with a finalizer that indicates ACK is // managing the resource and the underlying CR may not be deleted until ACK // is finished cleaning up any backend AWS service resources associated // with the CR. I dunno how good it is, but clearly they've thought about it. Test it before you invest billions of dollars into it, though.
      - DanielDent 6y ago
        What this describes is a mechanism for k8s to delete underlying resources prior to removing them from k8s. E.g. before completely removing the CRD representing an S3 bucket, this provides a mechanism for that S3 bucket to be deleted from AWS systems. Which I think is the opposite of what you were hoping.
      - dilyevsky 6y ago
        Finalizers dont do much for safety. They are simply there to ensure controller (in this case ACK) won’t miss the deletion event and leave the resource dangling. To actually prevent object from being deleted you need a validation webhook
- k__ 6y ago
  Is there something like CloudFormation console for EKS/ACK? last time I deleted a cluster it failed because of a NLB still being around but not accounted for as CFN resource, even though I provisioned the cluster with CFN.
- takeda 6y ago
  I thought whole point of running K8S in AWS was to have infrastructure not tied to their offerings. If you use something like that, why use K8S and not just use AWS services natively?
  - scarface74 6y ago
    The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale. I have a feeling that anyone who thinks it is easy or even usually worth it has never been part of planning a large scale migration. At the same time, you usually end up spending more money and having worse results when you don’t go all in. As far as why use EKS vs ECS - the “native service”? They seem to have feature parity, ECS is easier to use for the unitiated. But, there are so many people who know k8s and your knowledge is portable. Which brings up my second point. Most software engineers don’t care about cloud mobility as much as they claim. They care about career mobility. There is a much better chance that you will leave a company and move to a company on a different provider than your company will. I’m not saying it’s a bad thing to focus on technologies that give you as an individual the most optionality.
    - dilyevsky 6y ago
      > The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale Pretty categorical statement and I don’t find this to be true at all if you avoid using anything managed except block devices and vms themselves. That is unless “any type or scale” is tens of thousands of VMs and petabytes of storage in which case it is indeed a moot endeavor since someone who is still on public cloud at that point apparently likes to give all their money to cloud vendors anyway.
      - scarface74 6y ago
        If all you’re doing is using a cloud provider to host a bunch of VMs, you’ve already lost the plot. Once you use any cloud provider as a glorified colo without changing any of your processes you’re already spending more than a colo and not getting any of the benefits of managed services. Have you ever budgeted a project plan involving a large migration.
cagenut 6y ago
On the one hand this could be such a cool and powerful concept. On the other hand my brain segfaults on the recursive loop of how the layer-inversion gets modeled as IaC with a CI/CD pipeline. I guess if you were very strict about having your provider-infra layer (cloudformation/terraform) do only the bare minimum to get your kube environment up, and then within that kube environment you used something like ACK to provision any cloud-provider resources that your kube-managed apps/pipelines needed. Yet another case where I'm like "I don't know if kube should be the answer to everything, but I sure as shit won't miss <x>".
- rektide 6y ago
  Yes, the goal is very much to bring provisioning & operations of all resources on to the core platform we're using, Kubernetes. I mentioned in another reply how excited I am to be able to manage things like SQS queues now with my app, via Helm Charts, rather than need separate machinery to manage/operate my app & the various resources it needs. And now there is a control loop. So if Ben in support accidentally deletes my queue, it'll get recreated. Breaking away from the proprietary platform underlay is going to be great. Managing things more consistently is going to be really great.
mfer 6y ago
One consequence is the accidental deletion of AWS things... If a CRD is deleted the CRs described it are also deleted. So, deleting a CRD (even accidentally) could end up deleting resources in AWS (e.g., backups). So, be careful. Some things being managed by Kubernetes would be really cool. Other things being managed by k8s could break things if something goes wrong. I would plan accordingly.
- outworlder 6y ago
  Exactly. Planning accordingly is the right answer. Assume anyone can destroy your infrastructure at any time (by mistake or otherwise). This could be done with cloudformation, terraform, API calls, and essentially any automation(with different levels of safeguards). Be prepared for that. Be careful with your data. Not so careful with individual servers - they should be cattle, not pets. EDIT: If this is a production system, one could take away any 'delete' permissions until they are needed again.
  - ForHackernews 6y ago
    "cattle, not pets" is a nice slogan, but I think most ranchers would be angry if a junior ranch hand accidentally poisoned the entire herd with a typo.
- mey 6y ago
  There are certain things that freak me out in IAC, like loosing a primary DB or secrets. I want IAC to create them, wire things together and index them, but the idea of an accidental mis-configuration deleting a production database gives me heartburn.
  - uHuge 6y ago
    That's what you should have QA pipeline ready to check before you'll be able to apply the config, right? Secrets should be encrypted in git to my best awareness.
- NathanKP 6y ago
  Hi I'm a developer advocate in the AWS Container organization. This question of how to handle resource destruction is an active issue on the project, and we'd welcome your comments (or those of anyone else) on this Github issue: https://github.com/aws/aws-controllers-k8s/issues/82 https://github.com/aws/aws-controllers-k8s/issues/82 Our goal is to make this project have "no surprises" and therefore no unexpected destruction of resources. The specifics of how we mark resource as safe to delete instead of retaining by default are under discussion on that Github issue.
compsciphd 6y ago
I started building something along this line a few years ago. the ability to control AWS VM and treat them as pods (i.e. can backend a service, access other services) to have a hybrid (VM / Container) infrastucture that is all managed in the kubernetes way. Future work would have been to try and manage other resources similarly. Sadly startup interest changed and then went under (but the freedom I was given to explore there was the best experience I have ever had) https://github.com/apporbit/infranetes https://github.com/apporbit/infranetes
brian_herman__ 6y ago
This reminds me of this classic comic. https://www.catmuseumsf.org/images/print/comix/bill.jpg https://www.catmuseumsf.org/images/print/comix/bill.jpg
- francesca 6y ago
  Reminds me of Nantucket's airport
- femto113 6y ago
  Or almost any Cathy strip https://mirabiledictu.org/cathy-ack-2/ https://mirabiledictu.org/cathy-ack-2/
just-juan-post 6y ago
You can create a S3 bucket but you can't set permissions on it. Pass, I'll check back in a year.
MuffinFlavored 6y ago
How is this different than Terraform?
- harpratap 6y ago
  Traditional Infrastructure-as-Code solutions were based on edge-triggers. Like create ingress, delete ingress. But what about when ingress misbehaves or is in an unrecoverable state? Kubernetes introduced edge-triggered level-driven with resync reconciliation based "controllers". User defines a state and the controller does it's best to keep the infra in this desired state all the times. (Although Terraform has also moved to this same design in recent times) This establishes a consistent experience. Everyone knows you just need to do kubectl get my-resource to check your desired state. All the issues will be logged in status and controller. You can combine multiple controllers to achieve your desired application design. For example, Knative has their own kind called "Service" which has some custom components, some inherited from istio and things like replicasets from default kubernetes controllers.
HereBeBeasties 6y ago
See also KubeForm - https://kubeform.com/ https://kubeform.com/ which will do a lot of this already, via Terraform.
nikolay 6y ago
I Terraform Kubernetes Operator sounds like a better idea.

111 points by yonasb 6y ago

Some prior discussion: https://news.ycombinator.com/item?id=24219448 https://news.ycombinator.com/item?id=24219448

Kelsey Hightower had a good take in https://twitter.com/kelseyhightower/status/1296311951530704896 https://twitter.com/kelseyhightower/status/12963119515307048... "AWS Controllers for Kubernetes is pretty dope. You can leverage Kubernetes to manage AWS resources such as API gateways and S3 buckets. Think Terraform but backed by Kubernetes style APIs and "realtime" control loops." An in the thread he mentions Crossplane as the cross-cloud way to do this https://twitter.com/kelseyhightower/status/1296321377134231552 https://twitter.com/kelseyhightower/status/12963213771342315...

rektide 6y ago
Every Kubernetes shop I've seen uses Helm to deploy their workloads, & now you can set up all your S3 Buckets, SQS, SNS stuff in that same Helm chart as your app. This is hella ideal. I also just generally love the thought of being able to manage any cloud resources at all via standard open-protocols & systems. Compounding the investment, rather than having to invest in a bunch of specific not-interconnected areas is going to lead to great things. It'll be interesting to see what if any architectural flourishes or innovations went in to ACK's control loops.
- etxm 6y ago
  It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?
  - rektide 6y ago
    metadata: annotations: helm.sh/resource-policy: keep works for any & all resources!
  - jrockway 6y ago
    Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RDS instances would have similar protection. (You might think that the k8s world is very mutable, but there are some immutable things hanging around. For cases exactly like this.) I don't use AWS so I didn't look into this thoroughly, but the word "finalizer" does show up in the code frequently: // MarkManaged places the supplied resource under the management of ACK. // What this typically means is that the resource manager will decorate the // underlying custom resource (CR) with a finalizer that indicates ACK is // managing the resource and the underlying CR may not be deleted until ACK // is finished cleaning up any backend AWS service resources associated // with the CR. I dunno how good it is, but clearly they've thought about it. Test it before you invest billions of dollars into it, though.
    - DanielDent 6y ago
      What this describes is a mechanism for k8s to delete underlying resources prior to removing them from k8s. E.g. before completely removing the CRD representing an S3 bucket, this provides a mechanism for that S3 bucket to be deleted from AWS systems. Which I think is the opposite of what you were hoping.
    - dilyevsky 6y ago
      Finalizers dont do much for safety. They are simply there to ensure controller (in this case ACK) won’t miss the deletion event and leave the resource dangling. To actually prevent object from being deleted you need a validation webhook
k__ 6y ago
Is there something like CloudFormation console for EKS/ACK? last time I deleted a cluster it failed because of a NLB still being around but not accounted for as CFN resource, even though I provisioned the cluster with CFN.
takeda 6y ago
I thought whole point of running K8S in AWS was to have infrastructure not tied to their offerings. If you use something like that, why use K8S and not just use AWS services natively?
- scarface74 6y ago
  The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale. I have a feeling that anyone who thinks it is easy or even usually worth it has never been part of planning a large scale migration. At the same time, you usually end up spending more money and having worse results when you don’t go all in. As far as why use EKS vs ECS - the “native service”? They seem to have feature parity, ECS is easier to use for the unitiated. But, there are so many people who know k8s and your knowledge is portable. Which brings up my second point. Most software engineers don’t care about cloud mobility as much as they claim. They care about career mobility. There is a much better chance that you will leave a company and move to a company on a different provider than your company will. I’m not saying it’s a bad thing to focus on technologies that give you as an individual the most optionality.
  - dilyevsky 6y ago
    > The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale Pretty categorical statement and I don’t find this to be true at all if you avoid using anything managed except block devices and vms themselves. That is unless “any type or scale” is tens of thousands of VMs and petabytes of storage in which case it is indeed a moot endeavor since someone who is still on public cloud at that point apparently likes to give all their money to cloud vendors anyway.
    - scarface74 6y ago
      If all you’re doing is using a cloud provider to host a bunch of VMs, you’ve already lost the plot. Once you use any cloud provider as a glorified colo without changing any of your processes you’re already spending more than a colo and not getting any of the benefits of managed services. Have you ever budgeted a project plan involving a large migration.

cagenut 6y ago

On the one hand this could be such a cool and powerful concept. On the other hand my brain segfaults on the recursive loop of how the layer-inversion gets modeled as IaC with a CI/CD pipeline. I guess if you were very strict about having your provider-infra layer (cloudformation/terraform) do only the bare minimum to get your kube environment up, and then within that kube environment you used something like ACK to provision any cloud-provider resources that your kube-managed apps/pipelines needed. Yet another case where I'm like "I don't know if kube should be the answer to everything, but I sure as shit won't miss <x>".

rektide 6y ago
Yes, the goal is very much to bring provisioning & operations of all resources on to the core platform we're using, Kubernetes. I mentioned in another reply how excited I am to be able to manage things like SQS queues now with my app, via Helm Charts, rather than need separate machinery to manage/operate my app & the various resources it needs. And now there is a control loop. So if Ben in support accidentally deletes my queue, it'll get recreated. Breaking away from the proprietary platform underlay is going to be great. Managing things more consistently is going to be really great.

mfer 6y ago

One consequence is the accidental deletion of AWS things... If a CRD is deleted the CRs described it are also deleted. So, deleting a CRD (even accidentally) could end up deleting resources in AWS (e.g., backups). So, be careful. Some things being managed by Kubernetes would be really cool. Other things being managed by k8s could break things if something goes wrong. I would plan accordingly.

outworlder 6y ago
Exactly. Planning accordingly is the right answer. Assume anyone can destroy your infrastructure at any time (by mistake or otherwise). This could be done with cloudformation, terraform, API calls, and essentially any automation(with different levels of safeguards). Be prepared for that. Be careful with your data. Not so careful with individual servers - they should be cattle, not pets. EDIT: If this is a production system, one could take away any 'delete' permissions until they are needed again.
- ForHackernews 6y ago
  "cattle, not pets" is a nice slogan, but I think most ranchers would be angry if a junior ranch hand accidentally poisoned the entire herd with a typo.
mey 6y ago
There are certain things that freak me out in IAC, like loosing a primary DB or secrets. I want IAC to create them, wire things together and index them, but the idea of an accidental mis-configuration deleting a production database gives me heartburn.
- uHuge 6y ago
  That's what you should have QA pipeline ready to check before you'll be able to apply the config, right? Secrets should be encrypted in git to my best awareness.
NathanKP 6y ago
Hi I'm a developer advocate in the AWS Container organization. This question of how to handle resource destruction is an active issue on the project, and we'd welcome your comments (or those of anyone else) on this Github issue: https://github.com/aws/aws-controllers-k8s/issues/82 https://github.com/aws/aws-controllers-k8s/issues/82 Our goal is to make this project have "no surprises" and therefore no unexpected destruction of resources. The specifics of how we mark resource as safe to delete instead of retaining by default are under discussion on that Github issue.

compsciphd 6y ago

I started building something along this line a few years ago. the ability to control AWS VM and treat them as pods (i.e. can backend a service, access other services) to have a hybrid (VM / Container) infrastucture that is all managed in the kubernetes way. Future work would have been to try and manage other resources similarly. Sadly startup interest changed and then went under (but the freedom I was given to explore there was the best experience I have ever had) https://github.com/apporbit/infranetes https://github.com/apporbit/infranetes

brian_herman__ 6y ago

This reminds me of this classic comic. https://www.catmuseumsf.org/images/print/comix/bill.jpg https://www.catmuseumsf.org/images/print/comix/bill.jpg

francesca 6y ago
Reminds me of Nantucket's airport
femto113 6y ago
Or almost any Cathy strip https://mirabiledictu.org/cathy-ack-2/ https://mirabiledictu.org/cathy-ack-2/

just-juan-post 6y ago

You can create a S3 bucket but you can't set permissions on it. Pass, I'll check back in a year.

MuffinFlavored 6y ago

How is this different than Terraform?

harpratap 6y ago
Traditional Infrastructure-as-Code solutions were based on edge-triggers. Like create ingress, delete ingress. But what about when ingress misbehaves or is in an unrecoverable state? Kubernetes introduced edge-triggered level-driven with resync reconciliation based "controllers". User defines a state and the controller does it's best to keep the infra in this desired state all the times. (Although Terraform has also moved to this same design in recent times) This establishes a consistent experience. Everyone knows you just need to do kubectl get my-resource to check your desired state. All the issues will be logged in status and controller. You can combine multiple controllers to achieve your desired application design. For example, Knative has their own kind called "Service" which has some custom components, some inherited from istio and things like replicasets from default kubernetes controllers.

HereBeBeasties 6y ago

See also KubeForm - https://kubeform.com/ https://kubeform.com/ which will do a lot of this already, via Terraform.

nikolay 6y ago

I Terraform Kubernetes Operator sounds like a better idea.