5 ms·

Understanding AWS End of Service Life Is a Key FinOps Responsibility

53 points by noctarius 2y ago

noctarius 2y ago
Article by Mary Henry. I was shocked to see how much more the extended support (per hour) cost is for Kubernetes on AWS. Haven't had that situation myself on AWS yet, but ran into it a few times on Azure I can't remember to have paid extra on Azure though, but maybe we did. Certainly not 6x the price though. PS: not sure why it got flagged the first time, but I think because I used a different title. Sorry.
- qqtt 2y ago
  AWS also recently ended support for Mysql 5, so if you had an RDS instance with that version running past the cutoff, your support costs ballooned exorbitantly.
  - noctarius 2y ago
    Seems like I'm a lucky one. Neither using RDS nor MySQL. But seriously, ouch. I mean, I get why they want people to migrate to supported versions but ...
    - SteveNuts 2y ago
      I wish we could implement this internally via chargebacks. The teams that refuse to upgrade their stuff should be forced to pay for the externalities they cause.
  - VectorLock 2y ago
    Yup this one hit me hard. USE2-ExtendedSupport:Yr1-Yr2:MySQL5.7 sent my bill up 70%.
    - hughesjj 2y ago
      How long was it before the notice and you getting charged extra?
- res0nat0r 2y ago
  We just got emails yesterday about the EKS price increase. It's another reason we're trying to move the main app to the vendors SaaS because I don't have enough time and resources to be a fulltime k8s admin. The ecosystem moves way too fast and upgrades/deprecation happens way too quickly to keep up and to have time test / plan / rollout proper upgrades without breaking our critical production workloads.
chrisjj 2y ago
> running unsupported versions makes it harder to get help from a community that’s currently focused on the latest version Great example of misuse of that simple word 'that'. Should be 'which'.
- TecoAndJix 2y ago
  Always learning something new[1]: "The difference between which and that depends on whether the clause is restrictive or nonrestrictive. In a restrictive clause, use that. In a nonrestrictive clause, use which. Remember, which is as disposable as a sandwich wrapper. If you can remove the clause without destroying the meaning of the sentence, the clause is nonessential (another word for nonrestrictive), and you can use which. [1] https://www.grammarly.com/blog/which-vs-that/#:~:text=Which%20vs.%20that%3A%20What's%20the,a%20nonrestrictive%20clause%2C%20use%20which https://www.grammarly.com/blog/which-vs-that/#:~:text=Which%....
- pas 2y ago
  can you please explain the difference in semantics? what does this mean with 'that' and why that is inconsistent/incorrect/illogical compared to the meaning with 'which'? thanks!
  - chrisjj 2y ago
    https://writeanything.wordpress.com/2008/09/20/grammar-girl-ie-vs-eg/ https://writeanything.wordpress.com/2008/09/20/grammar-girl-...
htrp 2y ago
This is also the right way to deprecate. Charge people an arm and a leg to keep things running (and eventually force them to migrate).
- noctarius 2y ago
  True, but I guess it'll be a surprise to many. And, unfortunately, upgrading isn't always the easiest thing with deprecations and stuff
- solatic 2y ago
  100%. People are responsible for an ever-increasing amount of things; people will focus on business priorities and stuff that is working will be left the hell alone. As long as the bills are manageable and the business pays - the lights will be kept on forever. Passing increasing support costs to customers realigns interests between customer and provider without danger of user impact. And for Kubernetes, honestly, charging 6x for extended support is probably a bargain, considering the pace of change and difficulty of hiring engineers for unsexy maintenance work.
  - mdaniel 2y ago
    I do appreciate that the devil is always in the details, but I'll be straight that their new(?) "Upgrade insights" tab/api <https://docs.aws.amazon.com/eks/latest/userguide/cluster-insights.html https://docs.aws.amazon.com/eks/latest/userguide/cluster-ins...> goes a long way toward driving down the upgrade risk from a "well, what are we using that's going to get cut in the new version?" We just rolled off of their extended version and it was about 19 minutes to upgrade the control plane, no downtime, and then varying between 10 minutes and over an hour to upgrade the vpc-cni add-on. It seemed just completely random, and without any cancel button. We also had to manually patch kube-proxy container version, which OT1H, they did document, but OTOH, well, I didn't put those DaemonSets on the Nodes so why do I suddenly have to manage its version? Weird Touching CNI is always a potential downtime inducing event, but for the most part it was manageable
- TheP1000 2y ago
  Agreed. I would imagine the previous approach of forced upgrades ended up burning lots of customers in worse ways than just their pocketbook.
VectorLock 2y ago
Had this bite me for my small-scale personal AWS setup. Have an AWS account I run some personal sites on, a Mastodon instance, etc. Got some Billing Alarms I setup that my bill went from normally $100 to $180. Got a $75 charge for USE2-ExtendedSupport:Yr1-Yr2:MySQL5.7 I mean I'm very used to Amazon's ridiculous fee structure but even this one caught me for a loop.
- noctarius 2y ago
  Ouch. Glad you had the alarm (and that it reacted "early enough"). Anyhow, I think you may not be along with that surprise.
- steelaz 2y ago
  To be fair to AWS, they announced the deprecation of MySQL 5.7 in January 2021, and many emails warned of this change throughout 2024.
  - VectorLock 2y ago
    Deprecating a service is one thing. Charging an arm and a leg for a deprecated service is another.
neilv 2y ago
Sounds like an entropy problem. https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ > Which we get from EKS -- our entropy chaos service.
JohnMakin 2y ago
I'm fine with forcing upgrades this way - however, from an operations standpoint, it is an absolute nightmare. For one, depending on your situation/CRD's/automation, doing these upgrades in-place can be next to impossible. Updating an EKS minor version can only be done one version at a time - e.g., if you want to go from 1.24 -> 1.28, you need to do 1.25, then 1.26, then 1.27, then 1.28. So teams without a lot of resources are probably in a tough spot depending on how far they are behind here. Often, it's far more efficient to build an entirely new cluster from scratch and then cut over - which seems ridiculous. Why are upgrading EKS versions such a pain? Well, if you're using any cluster add-ons, for one, all those need to be upgraded to the correct versions, and the compatibility matrix there can be rough. Stuff often breaks at this stage. Care needs to be taken around PV's, the CNI, and god help you if you have some helm charts or CRD's that rely on some deprecated EKS API - even if the upstream repository has a fix for it, you will often find this yak-shaving nightmare of fixing all the stuff that breaks on upgrading that, and then whatever downstream services THAT service breaks - etc. What is the solution? I don't know. I'm not a kubernetes architect, but I work with it a lot. I understand there are security patches and improvements constantly, but the release cycle, at least from an infrastructure/operations perspective, IME places considerable strain on teams, to the point where I have literally seen a role in a company whose primary responsibility was upgrading EKS cluster versions. I have a sneaking suspicion this is to try to encourage people to migrate to more expensive managed container orchestration services.
- noctarius 2y ago
  Didn't think of this suspicion beforehand, but doesn't sound like a total miss.
- rho138 2y ago
  I recently did the upgrade from 1.24->1.28 on a neglected cluster after testing the upgrade in a dev environment and it was honestly not that terrible. It really comes down to having the capability and man hours to manage the procedure. In reality the longest part was waiting for cluster nodes to upgrade to X version of k8s, but the complete upgrade only took 3 weeks of testing and a single 4 hour outage with no loss in processing over the period. Realistically those workloads being run would have been better suited in an horizontal-scaling EC2 deployment but that was a future goal that never came to fruition.
thebeardisred 2y ago
This is something most people don't realize is an aspect of Red Hat's value. Extended Lifecycle Support (ELS) + Extended Update Support (EUS) are available _just in case_ you really can't figure out how to migrate off of those Red Hat Enterprise Linux 6 systems running on x86 (32 bit). https://access.redhat.com/support/policy/updates/errata https://access.redhat.com/support/policy/updates/errata
abrookewood 2y ago
Doesn't just apply to EKS - we are currently going through the same thing with MySQL on RDS. It's a big jump in support cost, but at the same time, I understand why they are doing it.
bushbaba 2y ago
Why the AWS hate when this is an issue of the k8s cncf team from constant churn. There needs to be a cncf blessed LTS release of k8s. AWS is just filling in a gap here with headaches involved of back porting security patches.

53 points by noctarius 2y ago

Article by Mary Henry. I was shocked to see how much more the extended support (per hour) cost is for Kubernetes on AWS. Haven't had that situation myself on AWS yet, but ran into it a few times on Azure I can't remember to have paid extra on Azure though, but maybe we did. Certainly not 6x the price though. PS: not sure why it got flagged the first time, but I think because I used a different title. Sorry.

qqtt 2y ago
AWS also recently ended support for Mysql 5, so if you had an RDS instance with that version running past the cutoff, your support costs ballooned exorbitantly.
- noctarius 2y ago
  Seems like I'm a lucky one. Neither using RDS nor MySQL. But seriously, ouch. I mean, I get why they want people to migrate to supported versions but ...
  - SteveNuts 2y ago
    I wish we could implement this internally via chargebacks. The teams that refuse to upgrade their stuff should be forced to pay for the externalities they cause.
- VectorLock 2y ago
  Yup this one hit me hard. USE2-ExtendedSupport:Yr1-Yr2:MySQL5.7 sent my bill up 70%.
  - hughesjj 2y ago
    How long was it before the notice and you getting charged extra?
res0nat0r 2y ago
We just got emails yesterday about the EKS price increase. It's another reason we're trying to move the main app to the vendors SaaS because I don't have enough time and resources to be a fulltime k8s admin. The ecosystem moves way too fast and upgrades/deprecation happens way too quickly to keep up and to have time test / plan / rollout proper upgrades without breaking our critical production workloads.

chrisjj 2y ago

> running unsupported versions makes it harder to get help from a community that’s currently focused on the latest version Great example of misuse of that simple word 'that'. Should be 'which'.

TecoAndJix 2y ago
Always learning something new[1]: "The difference between which and that depends on whether the clause is restrictive or nonrestrictive. In a restrictive clause, use that. In a nonrestrictive clause, use which. Remember, which is as disposable as a sandwich wrapper. If you can remove the clause without destroying the meaning of the sentence, the clause is nonessential (another word for nonrestrictive), and you can use which. [1] https://www.grammarly.com/blog/which-vs-that/#:~:text=Which%20vs.%20that%3A%20What's%20the,a%20nonrestrictive%20clause%2C%20use%20which https://www.grammarly.com/blog/which-vs-that/#:~:text=Which%....
pas 2y ago
can you please explain the difference in semantics? what does this mean with 'that' and why that is inconsistent/incorrect/illogical compared to the meaning with 'which'? thanks!
- chrisjj 2y ago
  https://writeanything.wordpress.com/2008/09/20/grammar-girl-ie-vs-eg/ https://writeanything.wordpress.com/2008/09/20/grammar-girl-...

htrp 2y ago

This is also the right way to deprecate. Charge people an arm and a leg to keep things running (and eventually force them to migrate).

noctarius 2y ago
True, but I guess it'll be a surprise to many. And, unfortunately, upgrading isn't always the easiest thing with deprecations and stuff
solatic 2y ago
100%. People are responsible for an ever-increasing amount of things; people will focus on business priorities and stuff that is working will be left the hell alone. As long as the bills are manageable and the business pays - the lights will be kept on forever. Passing increasing support costs to customers realigns interests between customer and provider without danger of user impact. And for Kubernetes, honestly, charging 6x for extended support is probably a bargain, considering the pace of change and difficulty of hiring engineers for unsexy maintenance work.
- mdaniel 2y ago
  I do appreciate that the devil is always in the details, but I'll be straight that their new(?) "Upgrade insights" tab/api <https://docs.aws.amazon.com/eks/latest/userguide/cluster-insights.html https://docs.aws.amazon.com/eks/latest/userguide/cluster-ins...> goes a long way toward driving down the upgrade risk from a "well, what are we using that's going to get cut in the new version?" We just rolled off of their extended version and it was about 19 minutes to upgrade the control plane, no downtime, and then varying between 10 minutes and over an hour to upgrade the vpc-cni add-on. It seemed just completely random, and without any cancel button. We also had to manually patch kube-proxy container version, which OT1H, they did document, but OTOH, well, I didn't put those DaemonSets on the Nodes so why do I suddenly have to manage its version? Weird Touching CNI is always a potential downtime inducing event, but for the most part it was manageable
TheP1000 2y ago
Agreed. I would imagine the previous approach of forced upgrades ended up burning lots of customers in worse ways than just their pocketbook.

VectorLock 2y ago

Had this bite me for my small-scale personal AWS setup. Have an AWS account I run some personal sites on, a Mastodon instance, etc. Got some Billing Alarms I setup that my bill went from normally $100 to $180. Got a $75 charge for USE2-ExtendedSupport:Yr1-Yr2:MySQL5.7 I mean I'm very used to Amazon's ridiculous fee structure but even this one caught me for a loop.

noctarius 2y ago
Ouch. Glad you had the alarm (and that it reacted "early enough"). Anyhow, I think you may not be along with that surprise.
steelaz 2y ago
To be fair to AWS, they announced the deprecation of MySQL 5.7 in January 2021, and many emails warned of this change throughout 2024.
- VectorLock 2y ago
  Deprecating a service is one thing. Charging an arm and a leg for a deprecated service is another.

neilv 2y ago

Sounds like an entropy problem. https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ > Which we get from EKS -- our entropy chaos service.

JohnMakin 2y ago

I'm fine with forcing upgrades this way - however, from an operations standpoint, it is an absolute nightmare. For one, depending on your situation/CRD's/automation, doing these upgrades in-place can be next to impossible. Updating an EKS minor version can only be done one version at a time - e.g., if you want to go from 1.24 -> 1.28, you need to do 1.25, then 1.26, then 1.27, then 1.28. So teams without a lot of resources are probably in a tough spot depending on how far they are behind here. Often, it's far more efficient to build an entirely new cluster from scratch and then cut over - which seems ridiculous. Why are upgrading EKS versions such a pain? Well, if you're using any cluster add-ons, for one, all those need to be upgraded to the correct versions, and the compatibility matrix there can be rough. Stuff often breaks at this stage. Care needs to be taken around PV's, the CNI, and god help you if you have some helm charts or CRD's that rely on some deprecated EKS API - even if the upstream repository has a fix for it, you will often find this yak-shaving nightmare of fixing all the stuff that breaks on upgrading that, and then whatever downstream services THAT service breaks - etc. What is the solution? I don't know. I'm not a kubernetes architect, but I work with it a lot. I understand there are security patches and improvements constantly, but the release cycle, at least from an infrastructure/operations perspective, IME places considerable strain on teams, to the point where I have literally seen a role in a company whose primary responsibility was upgrading EKS cluster versions. I have a sneaking suspicion this is to try to encourage people to migrate to more expensive managed container orchestration services.

noctarius 2y ago
Didn't think of this suspicion beforehand, but doesn't sound like a total miss.
rho138 2y ago
I recently did the upgrade from 1.24->1.28 on a neglected cluster after testing the upgrade in a dev environment and it was honestly not that terrible. It really comes down to having the capability and man hours to manage the procedure. In reality the longest part was waiting for cluster nodes to upgrade to X version of k8s, but the complete upgrade only took 3 weeks of testing and a single 4 hour outage with no loss in processing over the period. Realistically those workloads being run would have been better suited in an horizontal-scaling EC2 deployment but that was a future goal that never came to fruition.

thebeardisred 2y ago

This is something most people don't realize is an aspect of Red Hat's value. Extended Lifecycle Support (ELS) + Extended Update Support (EUS) are available _just in case_ you really can't figure out how to migrate off of those Red Hat Enterprise Linux 6 systems running on x86 (32 bit). https://access.redhat.com/support/policy/updates/errata https://access.redhat.com/support/policy/updates/errata

abrookewood 2y ago

Doesn't just apply to EKS - we are currently going through the same thing with MySQL on RDS. It's a big jump in support cost, but at the same time, I understand why they are doing it.

bushbaba 2y ago

Why the AWS hate when this is an issue of the k8s cncf team from constant churn. There needs to be a cncf blessed LTS release of k8s. AWS is just filling in a gap here with headaches involved of back porting security patches.