3 ms·

I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to sta

I haven't seen these failure modes in 2021. We do managed clusters at work and have created around 100,000 of them, and basically all the pods we intend to start start, and all the pods we intend to kill die -- even with autoscaling that provisions new instance types. Our biggest failure mode is TLS certificates failing to provision through Let's Encrypt, but that has nothing to do with Kubernetes (that is a layer above; what we run in Kubernetes). EKS continues to be painful. It has gotten better over the years, but it is a chore compared to GKE. I like to imagine that Jeff Bezos walked into someone's office, said "you're all fired if I don't have Kubersomethings in two weeks", and that's what they launched.

shaicoleman 5y ago
What are the EKS pain points?
- jrockway 5y ago
  The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)
  - bashinator 5y ago
    For what it's worth, there's an off-the-shelf terraform module for EKS that is far simpler to use than AWS' cloudformation tooling, which does allow you to pass in a custom AMI and multiple nodegroup configurations as input parameters. https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest https://registry.terraform.io/modules/terraform-aws-modules/...
  - blitzd 5y ago
    EKS has actually come a long way in the last couple of years. They now have 'eksctl' that helps a lot with provisioning/de-provisioning the node pools. They're also pretty up to date with supporting new versions of k8s (1.21.2 now, I think). I definitely know what you mean about the 'minimum viable product' and their overall views on k8s though, as our account rep definitely tried to steer us away from EKS when we ran into problems with CronJob. They also don't seem to have any particular relationship with k8s development, so we ended up not able to use CronJob for more than six months while our issue (with k8s repo) just sat waiting for someone to look at it.
cpach 5y ago
You have probably thought about this already but I must admit I’m curious: If you’re on AWS, can you not use Certificate Manager instead of Let’s Encrypt?
- jjoonathan 5y ago
  Certificate Manager pushes you (shoves you, really) in the direction of using AWS managed services. They make certificate installation/rotation really easy for their own services and unnecessarily difficult for any that you implement yourself. (This may have changed in the last year or two, but it was certainly this way when I tried it.)
  - bashinator 5y ago
    In my experience, it's difficult-to-impossible to use AWS' certificate management and LB termination in conjunction with Envoy-based networking like Istio or Ambassador.
    - jjoonathan 5y ago
      Yeah, the AWS LB has more issues than that, too. I'm pretty sure it's just nginx under the hood but they won't tweak the simplest parameters for you, even if you make a colossal stink, even if your company spends seven figures a year. I wonder if it isn't a decade-old duct-tape-and-bailing wire solution that shares the same config across literally every customer or something. Rolling our own was almost a relief -- the pile of awkward workarounds had grown pretty high by the point we bit the bullet.
- jrockway 5y ago
  I'm actually not on AWS, just used EKS extensively at my last job (and we still manually test our software against it). AWS burned me hard with forgetting to auto-renew certs at my last job. It just stopped working, the deadline passed, and only a support ticket and manual hacking on their side could make it work. cert-manager has been significantly more reliable and at least transparent. The mistake we make right now is asking for certificates on demand in the critical path of running our app -- but since we control the domain name, we could easily have a pool of domain names and certificates ready to go. Our mistake is having not done that yet.
debarshri 5y ago
This year, I have seen those issues popping up in statefulsets alot. I realised somebody in the team was force deleting it. It is actually well documented. https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/ https://kubernetes.io/docs/tasks/run-application/force-delet... I have seen few scenarios where people patching a statefulset actually screws up the volume mount. It is sometimes not evident where the error is for instance if it is the CSI or in the scheduler unless you deepdive into the issue.

shaicoleman 5y ago

What are the EKS pain points?

jrockway 5y ago
The biggest pain point is having to manually use cloudformation to create node pools. This is especially irritating when you just need to roll the Linux version on nodes -- takes half a day to do right. In GKE, it's just a button in the UI (or better, an easy-to-use API), and you can schedule maintenance windows for security updates (which are typically zero downtime anyway, assuming you have the right PodDisruptionBudgets). I think AWS fixed that. I remember when I used it, they said they had some new tool that would handle that, but you had to re-create the cluster from scratch. This was a couple years ago, and is probably decent nowadays. There are other warts, like certain storage classes being unavailable by default (gp3), the whole ENI thing for Pod IPs, the supported version being way out of date, etc. EKS has always felt like "minimum viable product" to me -- they really want you to use their proprietary stuff like ECS/Fargate, CloudFormation, etc. If you're already on AWS and want Kubernetes, it's just what you need. If you could pick any cloud provider for mainly Kubernetes, it wouldn't be my first choice. Having used EKS, GKE, and DOKS, I definitely prefer GKE. GKE is very feature-rich, and the API for managing clusters works well. The nodes are also cheaper than AWS. (I use DOKS for my personal stuff and I haven't had any problems, and it is free, but it's missing features like regional clusters that you probably want for things you make money off of.)
- bashinator 5y ago
  For what it's worth, there's an off-the-shelf terraform module for EKS that is far simpler to use than AWS' cloudformation tooling, which does allow you to pass in a custom AMI and multiple nodegroup configurations as input parameters. https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest https://registry.terraform.io/modules/terraform-aws-modules/...
- blitzd 5y ago
  EKS has actually come a long way in the last couple of years. They now have 'eksctl' that helps a lot with provisioning/de-provisioning the node pools. They're also pretty up to date with supporting new versions of k8s (1.21.2 now, I think). I definitely know what you mean about the 'minimum viable product' and their overall views on k8s though, as our account rep definitely tried to steer us away from EKS when we ran into problems with CronJob. They also don't seem to have any particular relationship with k8s development, so we ended up not able to use CronJob for more than six months while our issue (with k8s repo) just sat waiting for someone to look at it.

cpach 5y ago

You have probably thought about this already but I must admit I’m curious: If you’re on AWS, can you not use Certificate Manager instead of Let’s Encrypt?

jjoonathan 5y ago
Certificate Manager pushes you (shoves you, really) in the direction of using AWS managed services. They make certificate installation/rotation really easy for their own services and unnecessarily difficult for any that you implement yourself. (This may have changed in the last year or two, but it was certainly this way when I tried it.)
- bashinator 5y ago
  In my experience, it's difficult-to-impossible to use AWS' certificate management and LB termination in conjunction with Envoy-based networking like Istio or Ambassador.
  - jjoonathan 5y ago
    Yeah, the AWS LB has more issues than that, too. I'm pretty sure it's just nginx under the hood but they won't tweak the simplest parameters for you, even if you make a colossal stink, even if your company spends seven figures a year. I wonder if it isn't a decade-old duct-tape-and-bailing wire solution that shares the same config across literally every customer or something. Rolling our own was almost a relief -- the pile of awkward workarounds had grown pretty high by the point we bit the bullet.
jrockway 5y ago
I'm actually not on AWS, just used EKS extensively at my last job (and we still manually test our software against it). AWS burned me hard with forgetting to auto-renew certs at my last job. It just stopped working, the deadline passed, and only a support ticket and manual hacking on their side could make it work. cert-manager has been significantly more reliable and at least transparent. The mistake we make right now is asking for certificates on demand in the critical path of running our app -- but since we control the domain name, we could easily have a pool of domain names and certificates ready to go. Our mistake is having not done that yet.

debarshri 5y ago

This year, I have seen those issues popping up in statefulsets alot. I realised somebody in the team was force deleting it. It is actually well documented. https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/ https://kubernetes.io/docs/tasks/run-application/force-delet... I have seen few scenarios where people patching a statefulset actually screws up the volume mount. It is sometimes not evident where the error is for instance if it is the CSI or in the scheduler unless you deepdive into the issue.