6 ms·

Kelsey Hightower had a good take in https://twitter.com/kelseyhightower/status/1296311951530704896 https://twitter.com/kelseyhightower/status/12963119515307048.

by sytse 6y ago

Kelsey Hightower had a good take in https://twitter.com/kelseyhightower/status/1296311951530704896 https://twitter.com/kelseyhightower/status/12963119515307048... "AWS Controllers for Kubernetes is pretty dope. You can leverage Kubernetes to manage AWS resources such as API gateways and S3 buckets. Think Terraform but backed by Kubernetes style APIs and "realtime" control loops." An in the thread he mentions Crossplane as the cross-cloud way to do this https://twitter.com/kelseyhightower/status/1296321377134231552 https://twitter.com/kelseyhightower/status/12963213771342315...

rektide 6y ago
Every Kubernetes shop I've seen uses Helm to deploy their workloads, & now you can set up all your S3 Buckets, SQS, SNS stuff in that same Helm chart as your app. This is hella ideal. I also just generally love the thought of being able to manage any cloud resources at all via standard open-protocols & systems. Compounding the investment, rather than having to invest in a bunch of specific not-interconnected areas is going to lead to great things. It'll be interesting to see what if any architectural flourishes or innovations went in to ACK's control loops.
- etxm 6y ago
  It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?
  - rektide 6y ago
    metadata: annotations: helm.sh/resource-policy: keep works for any & all resources!
  - jrockway 6y ago
    Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RDS instances would have similar protection. (You might think that the k8s world is very mutable, but there are some immutable things hanging around. For cases exactly like this.) I don't use AWS so I didn't look into this thoroughly, but the word "finalizer" does show up in the code frequently: // MarkManaged places the supplied resource under the management of ACK. // What this typically means is that the resource manager will decorate the // underlying custom resource (CR) with a finalizer that indicates ACK is // managing the resource and the underlying CR may not be deleted until ACK // is finished cleaning up any backend AWS service resources associated // with the CR. I dunno how good it is, but clearly they've thought about it. Test it before you invest billions of dollars into it, though.
    - DanielDent 6y ago
      What this describes is a mechanism for k8s to delete underlying resources prior to removing them from k8s. E.g. before completely removing the CRD representing an S3 bucket, this provides a mechanism for that S3 bucket to be deleted from AWS systems. Which I think is the opposite of what you were hoping.
    - dilyevsky 6y ago
      Finalizers dont do much for safety. They are simply there to ensure controller (in this case ACK) won’t miss the deletion event and leave the resource dangling. To actually prevent object from being deleted you need a validation webhook
      - robszumski 6y ago
        An interesting part of the Operator Lifecycle Manager (OLM) is the capability to scaffold webhooks for Operators, rotate their secrets, etc to make this type protection easy for everyone to provide. All you need to do is bring your validation logic specific to the app. https://olm.operatorframework.io/docs/advanced-tasks/adding-an-admission-webhook/ https://olm.operatorframework.io/docs/advanced-tasks/adding-...
        an_opabinia 6y ago
        Kubernetes complexity: The amount of indirection for “you must write a --force flag to delete something” is astounding.
  - nrmitchi 6y ago
    This is, in my opinion, the most dangerous thing about CRDs, especially for resources that are not scoped the the cluster. It is way to easy to accidently delete something you need. PV's were my first experience dealing with this (a chart upgrade recreated a PVC, the old one unbound, and was immediately cleaned up), and it's not a risk that I want to see extended to buckets, RDS instances, etc. The other side of this is that CRDs can lead to abandoned resources; if you find your cluster borked, or shut down improperly, any resources which existed as CRDs (or in cloud Kubernetes land, include LoadBalancers) probably did not get cleaned up, and will be abandoned (but left running). It's not clear that there is actually a good solution here that fits neatly in with existing CRD behaviour.
    - rektide 6y ago
      If you've used helm for more than one day you'd know about the helm.sh/resource-policy: keep, Which makes your first big "accidentally deleted" concern pretty near moot, so long as it's your ci/CD tools with permissions & they are using helm. So long as random cluster users aren't futzing around randomly poking at & deleting things. Yes, you have to go way further & prevent your idiot users from being dumb, if you are giving them cluster role permissions. But again, have you considered not doing that? Abandoned resources is no less of an issue. If there are external resources that don't have state (such as your Load balancer examp, the good news is you can delete them all & let the resource controllers recreate the real ones. In general I feel like you are letting the 1% of concerns dominate & dissuade you, & that most people can make it very far with nothing extra.
      - nrmitchi 6y ago
        > If you've used helm for more than one day I'm not sure why you're popping in to a thread from a week ago, starting with personal insinuations against my qualifications to have an opinion, and being a generally uncivil person. You should try to make your points without the condescending attitude.
k__ 6y ago
Is there something like CloudFormation console for EKS/ACK? last time I deleted a cluster it failed because of a NLB still being around but not accounted for as CFN resource, even though I provisioned the cluster with CFN.
takeda 6y ago
I thought whole point of running K8S in AWS was to have infrastructure not tied to their offerings. If you use something like that, why use K8S and not just use AWS services natively?
- scarface74 6y ago
  The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale. I have a feeling that anyone who thinks it is easy or even usually worth it has never been part of planning a large scale migration. At the same time, you usually end up spending more money and having worse results when you don’t go all in. As far as why use EKS vs ECS - the “native service”? They seem to have feature parity, ECS is easier to use for the unitiated. But, there are so many people who know k8s and your knowledge is portable. Which brings up my second point. Most software engineers don’t care about cloud mobility as much as they claim. They care about career mobility. There is a much better chance that you will leave a company and move to a company on a different provider than your company will. I’m not saying it’s a bad thing to focus on technologies that give you as an individual the most optionality.
  - dilyevsky 6y ago
    > The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale Pretty categorical statement and I don’t find this to be true at all if you avoid using anything managed except block devices and vms themselves. That is unless “any type or scale” is tens of thousands of VMs and petabytes of storage in which case it is indeed a moot endeavor since someone who is still on public cloud at that point apparently likes to give all their money to cloud vendors anyway.
    - scarface74 6y ago
      If all you’re doing is using a cloud provider to host a bunch of VMs, you’ve already lost the plot. Once you use any cloud provider as a glorified colo without changing any of your processes you’re already spending more than a colo and not getting any of the benefits of managed services. Have you ever budgeted a project plan involving a large migration.

by sytse 6y ago

rektide 6y ago

Every Kubernetes shop I've seen uses Helm to deploy their workloads, & now you can set up all your S3 Buckets, SQS, SNS stuff in that same Helm chart as your app. This is hella ideal. I also just generally love the thought of being able to manage any cloud resources at all via standard open-protocols & systems. Compounding the investment, rather than having to invest in a bunch of specific not-interconnected areas is going to lead to great things. It'll be interesting to see what if any architectural flourishes or innovations went in to ACK's control loops.

etxm 6y ago
It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?
- rektide 6y ago
  metadata: annotations: helm.sh/resource-policy: keep works for any & all resources!
- jrockway 6y ago
  Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RDS instances would have similar protection. (You might think that the k8s world is very mutable, but there are some immutable things hanging around. For cases exactly like this.) I don't use AWS so I didn't look into this thoroughly, but the word "finalizer" does show up in the code frequently: // MarkManaged places the supplied resource under the management of ACK. // What this typically means is that the resource manager will decorate the // underlying custom resource (CR) with a finalizer that indicates ACK is // managing the resource and the underlying CR may not be deleted until ACK // is finished cleaning up any backend AWS service resources associated // with the CR. I dunno how good it is, but clearly they've thought about it. Test it before you invest billions of dollars into it, though.
  - DanielDent 6y ago
    What this describes is a mechanism for k8s to delete underlying resources prior to removing them from k8s. E.g. before completely removing the CRD representing an S3 bucket, this provides a mechanism for that S3 bucket to be deleted from AWS systems. Which I think is the opposite of what you were hoping.
  - dilyevsky 6y ago
    Finalizers dont do much for safety. They are simply there to ensure controller (in this case ACK) won’t miss the deletion event and leave the resource dangling. To actually prevent object from being deleted you need a validation webhook
    - robszumski 6y ago
      An interesting part of the Operator Lifecycle Manager (OLM) is the capability to scaffold webhooks for Operators, rotate their secrets, etc to make this type protection easy for everyone to provide. All you need to do is bring your validation logic specific to the app. https://olm.operatorframework.io/docs/advanced-tasks/adding-an-admission-webhook/ https://olm.operatorframework.io/docs/advanced-tasks/adding-...
      - an_opabinia 6y ago
        Kubernetes complexity: The amount of indirection for “you must write a --force flag to delete something” is astounding.
- nrmitchi 6y ago
  This is, in my opinion, the most dangerous thing about CRDs, especially for resources that are not scoped the the cluster. It is way to easy to accidently delete something you need. PV's were my first experience dealing with this (a chart upgrade recreated a PVC, the old one unbound, and was immediately cleaned up), and it's not a risk that I want to see extended to buckets, RDS instances, etc. The other side of this is that CRDs can lead to abandoned resources; if you find your cluster borked, or shut down improperly, any resources which existed as CRDs (or in cloud Kubernetes land, include LoadBalancers) probably did not get cleaned up, and will be abandoned (but left running). It's not clear that there is actually a good solution here that fits neatly in with existing CRD behaviour.
  - rektide 6y ago
    If you've used helm for more than one day you'd know about the helm.sh/resource-policy: keep, Which makes your first big "accidentally deleted" concern pretty near moot, so long as it's your ci/CD tools with permissions & they are using helm. So long as random cluster users aren't futzing around randomly poking at & deleting things. Yes, you have to go way further & prevent your idiot users from being dumb, if you are giving them cluster role permissions. But again, have you considered not doing that? Abandoned resources is no less of an issue. If there are external resources that don't have state (such as your Load balancer examp, the good news is you can delete them all & let the resource controllers recreate the real ones. In general I feel like you are letting the 1% of concerns dominate & dissuade you, & that most people can make it very far with nothing extra.
    - nrmitchi 6y ago
      > If you've used helm for more than one day I'm not sure why you're popping in to a thread from a week ago, starting with personal insinuations against my qualifications to have an opinion, and being a generally uncivil person. You should try to make your points without the condescending attitude.

k__ 6y ago

Is there something like CloudFormation console for EKS/ACK? last time I deleted a cluster it failed because of a NLB still being around but not accounted for as CFN resource, even though I provisioned the cluster with CFN.

takeda 6y ago

I thought whole point of running K8S in AWS was to have infrastructure not tied to their offerings. If you use something like that, why use K8S and not just use AWS services natively?

scarface74 6y ago
The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale. I have a feeling that anyone who thinks it is easy or even usually worth it has never been part of planning a large scale migration. At the same time, you usually end up spending more money and having worse results when you don’t go all in. As far as why use EKS vs ECS - the “native service”? They seem to have feature parity, ECS is easier to use for the unitiated. But, there are so many people who know k8s and your knowledge is portable. Which brings up my second point. Most software engineers don’t care about cloud mobility as much as they claim. They care about career mobility. There is a much better chance that you will leave a company and move to a company on a different provider than your company will. I’m not saying it’s a bad thing to focus on technologies that give you as an individual the most optionality.
- dilyevsky 6y ago
  > The entire idea of your infrastructure not being tied to your provider is completely lost when you are at any type of scale Pretty categorical statement and I don’t find this to be true at all if you avoid using anything managed except block devices and vms themselves. That is unless “any type or scale” is tens of thousands of VMs and petabytes of storage in which case it is indeed a moot endeavor since someone who is still on public cloud at that point apparently likes to give all their money to cloud vendors anyway.
  - scarface74 6y ago
    If all you’re doing is using a cloud provider to host a bunch of VMs, you’ve already lost the plot. Once you use any cloud provider as a glorified colo without changing any of your processes you’re already spending more than a colo and not getting any of the benefits of managed services. Have you ever budgeted a project plan involving a large migration.