3 ms·

I'm fine with forcing upgrades this way - however, from an operations standpoint, it is an absolute nightmare. For one, depending on your situation/CRD's/autom

I'm fine with forcing upgrades this way - however, from an operations standpoint, it is an absolute nightmare. For one, depending on your situation/CRD's/automation, doing these upgrades in-place can be next to impossible. Updating an EKS minor version can only be done one version at a time - e.g., if you want to go from 1.24 -> 1.28, you need to do 1.25, then 1.26, then 1.27, then 1.28. So teams without a lot of resources are probably in a tough spot depending on how far they are behind here. Often, it's far more efficient to build an entirely new cluster from scratch and then cut over - which seems ridiculous. Why are upgrading EKS versions such a pain? Well, if you're using any cluster add-ons, for one, all those need to be upgraded to the correct versions, and the compatibility matrix there can be rough. Stuff often breaks at this stage. Care needs to be taken around PV's, the CNI, and god help you if you have some helm charts or CRD's that rely on some deprecated EKS API - even if the upstream repository has a fix for it, you will often find this yak-shaving nightmare of fixing all the stuff that breaks on upgrading that, and then whatever downstream services THAT service breaks - etc. What is the solution? I don't know. I'm not a kubernetes architect, but I work with it a lot. I understand there are security patches and improvements constantly, but the release cycle, at least from an infrastructure/operations perspective, IME places considerable strain on teams, to the point where I have literally seen a role in a company whose primary responsibility was upgrading EKS cluster versions. I have a sneaking suspicion this is to try to encourage people to migrate to more expensive managed container orchestration services.

noctarius 3y ago
Didn't think of this suspicion beforehand, but doesn't sound like a total miss.
rho138 3y ago
I recently did the upgrade from 1.24->1.28 on a neglected cluster after testing the upgrade in a dev environment and it was honestly not that terrible. It really comes down to having the capability and man hours to manage the procedure. In reality the longest part was waiting for cluster nodes to upgrade to X version of k8s, but the complete upgrade only took 3 weeks of testing and a single 4 hour outage with no loss in processing over the period. Realistically those workloads being run would have been better suited in an horizontal-scaling EC2 deployment but that was a future goal that never came to fruition.
- JohnMakin 3y ago
  Like I said, it depends on your situation. Sometimes a /v1/beta api gets deprecated and causes complete chaos for a deployment. Sometimes your IAC is resistant to these kinds of frequent changes. There's really a billion scenarios. For reference, I have done upgrades from 1.12 -> 1.28 and most of the time if get into a messy project and I can get away with it, I will just rebuild a cluster from scratch.
cjk2 3y ago
Yeah this. My average day when I go near EKS upgrades: Waltz in, fuck up the ALB ingress controller in some new and interesting way, spend all day bouncing AWS support tickets around, find out it was AWS's fault, find half the manifest YAML schema in the universe is now deprecated, sob into my now soaking wet trousers and wonder why the fuck I ended up doing this for living. Yesterday I spent 3 hours trying to fix something and find it's an indent error somewhere.
watermelon0 3y ago
EKS release cycle is related to Kubernetes release cycle. I'm not sure it's fair to expect AWS to freely support outdated K8s versions, that don't have upstream support. If K8s would be backwards compatible, upgrading would be a lot easier, and if they would support LTS releases, like other projects, manual upgrades would be needed only every X years. For example, the reason that you can use PostgreSQL with the same major version for 5 years on RDS is due to PostgreSQL actively supporting it, and minor versions are non-breaking and can be seamlessly applied (restart or failover to standby replica is still needed during upgrade).
- JohnMakin 3y ago
  Completely understand why it is this way, and like I said I don't know the solution - unless AWS was able to or would want to fork Kubernetes in the same way that they did ElasticSearch, but that is understandable why they may not want to do that. Was mostly just griping that this process is a complete pain in the ass for tons of people (IME).
pid-1 3y ago
As K8s matures it's likely we will get some kind of LTS versioning scheme. Having new realeases so often for such a core infrastructure component is kinda insane unless it was explicitly architected to allow seeamless upgrades.
- noctarius 3y ago
  I hope you're right. Apart from that, yes I think it's necessary.
- mdaniel 3y ago
  There's a tiny bit of nuance there about "allow seamless upgrades" in that they do what I think is a fantastic job of version skew toleration between all the parts that interact (kubectl, kubelet, apiserver, etc). So that part, I think, is not the long pole in any such tent, especially because if the control-plane gets wiped out, kubelet will continue to manage the last state of affairs it knew about, and traffic will continue to flow to those pods The hairy bit is the rando junk that gets shoved into clutsers, without any sane packaging scheme to roll it up or back. I even recently had to learn the deep guts of the sh.helm.v1.foo secret because we accidentally left an old release in a cluster which no longer supported its apiVersion. No problem, says I, $(helm uninstall && helm install --version new-thing) but har-de-har-har helm uses that Secret to fully rehydrate the whole manifest of the release before deleting it so when helm tries (effectively) kubectl delete thing/v1beta1/oldthing and pukes, well, no uninstall for you, even if those objects are already gone
easton 3y ago
> try to encourage people to migrate to more expensive managed container orchestration services. The question is: what service? ECS is the competing Amazon built service and it’s entirely free for management, you just pay for compute. We don’t use k8s because ECS is free and we don’t plan on leaving AWS. Sure, you’re more locked in with ECS, but if you aren’t doing funky stuff with the APIs you can probably off ramp to k8s pretty easily. I know I could move us in a week or less, we’d have far bigger problems with the other AWS services we use.
- JohnMakin 3y ago
  Lightsail? I’m sure there’s more examples other than ecs.
- pas 3y ago
  how are you managing/configuring/monitoring/understanding ECS? (terraform?) to me it's a complete spaghetti with thick WTF-sauce. at least with k8s there's YAML and only YAML. (and there's CDK8S, and it supports tests, and spinning up a new cluster to test things is straightforward.) sure, I guess there's a whole industry that offers services to help manage AWS, but at that point the whole thing could be just spun off to the lowest bidder. (although I understand that people can make up a neverending list of reasons (excoughses) to keep paying the AWS tax.) ... okay, I'm probably too salty. if it works it works, if it's profitable and business is happy, it's hard to argue with the stack.
  - easton 3y ago
    CloudFormation for us. CF is wack, but the ECS concepts seemed as easy as k8s.

noctarius 3y ago

Didn't think of this suspicion beforehand, but doesn't sound like a total miss.

rho138 3y ago

I recently did the upgrade from 1.24->1.28 on a neglected cluster after testing the upgrade in a dev environment and it was honestly not that terrible. It really comes down to having the capability and man hours to manage the procedure. In reality the longest part was waiting for cluster nodes to upgrade to X version of k8s, but the complete upgrade only took 3 weeks of testing and a single 4 hour outage with no loss in processing over the period. Realistically those workloads being run would have been better suited in an horizontal-scaling EC2 deployment but that was a future goal that never came to fruition.

JohnMakin 3y ago
Like I said, it depends on your situation. Sometimes a /v1/beta api gets deprecated and causes complete chaos for a deployment. Sometimes your IAC is resistant to these kinds of frequent changes. There's really a billion scenarios. For reference, I have done upgrades from 1.12 -> 1.28 and most of the time if get into a messy project and I can get away with it, I will just rebuild a cluster from scratch.

cjk2 3y ago

Yeah this. My average day when I go near EKS upgrades: Waltz in, fuck up the ALB ingress controller in some new and interesting way, spend all day bouncing AWS support tickets around, find out it was AWS's fault, find half the manifest YAML schema in the universe is now deprecated, sob into my now soaking wet trousers and wonder why the fuck I ended up doing this for living. Yesterday I spent 3 hours trying to fix something and find it's an indent error somewhere.

watermelon0 3y ago

EKS release cycle is related to Kubernetes release cycle. I'm not sure it's fair to expect AWS to freely support outdated K8s versions, that don't have upstream support. If K8s would be backwards compatible, upgrading would be a lot easier, and if they would support LTS releases, like other projects, manual upgrades would be needed only every X years. For example, the reason that you can use PostgreSQL with the same major version for 5 years on RDS is due to PostgreSQL actively supporting it, and minor versions are non-breaking and can be seamlessly applied (restart or failover to standby replica is still needed during upgrade).

JohnMakin 3y ago
Completely understand why it is this way, and like I said I don't know the solution - unless AWS was able to or would want to fork Kubernetes in the same way that they did ElasticSearch, but that is understandable why they may not want to do that. Was mostly just griping that this process is a complete pain in the ass for tons of people (IME).

pid-1 3y ago

As K8s matures it's likely we will get some kind of LTS versioning scheme. Having new realeases so often for such a core infrastructure component is kinda insane unless it was explicitly architected to allow seeamless upgrades.

noctarius 3y ago
I hope you're right. Apart from that, yes I think it's necessary.
mdaniel 3y ago
There's a tiny bit of nuance there about "allow seamless upgrades" in that they do what I think is a fantastic job of version skew toleration between all the parts that interact (kubectl, kubelet, apiserver, etc). So that part, I think, is not the long pole in any such tent, especially because if the control-plane gets wiped out, kubelet will continue to manage the last state of affairs it knew about, and traffic will continue to flow to those pods The hairy bit is the rando junk that gets shoved into clutsers, without any sane packaging scheme to roll it up or back. I even recently had to learn the deep guts of the sh.helm.v1.foo secret because we accidentally left an old release in a cluster which no longer supported its apiVersion. No problem, says I, $(helm uninstall && helm install --version new-thing) but har-de-har-har helm uses that Secret to fully rehydrate the whole manifest of the release before deleting it so when helm tries (effectively) kubectl delete thing/v1beta1/oldthing and pukes, well, no uninstall for you, even if those objects are already gone

easton 3y ago

> try to encourage people to migrate to more expensive managed container orchestration services. The question is: what service? ECS is the competing Amazon built service and it’s entirely free for management, you just pay for compute. We don’t use k8s because ECS is free and we don’t plan on leaving AWS. Sure, you’re more locked in with ECS, but if you aren’t doing funky stuff with the APIs you can probably off ramp to k8s pretty easily. I know I could move us in a week or less, we’d have far bigger problems with the other AWS services we use.

JohnMakin 3y ago
Lightsail? I’m sure there’s more examples other than ecs.
pas 3y ago
how are you managing/configuring/monitoring/understanding ECS? (terraform?) to me it's a complete spaghetti with thick WTF-sauce. at least with k8s there's YAML and only YAML. (and there's CDK8S, and it supports tests, and spinning up a new cluster to test things is straightforward.) sure, I guess there's a whole industry that offers services to help manage AWS, but at that point the whole thing could be just spun off to the lowest bidder. (although I understand that people can make up a neverending list of reasons (excoughses) to keep paying the AWS tax.) ... okay, I'm probably too salty. if it works it works, if it's profitable and business is happy, it's hard to argue with the stack.
- easton 3y ago
  CloudFormation for us. CF is wack, but the ECS concepts seemed as easy as k8s.