10 ms·
Kubernetes Needs an LTS
- chitraa 3y agoWhat are your thoughts on the feasibility of an LTS for Kubernetes? Do you think it's something the community would embrace?
- sofixa 3y agoThis, like a recent LTS discussion I saw for a different tool, ignores one tiny little detail that makes the whole discussion kind of moot. LTS doesn't mean it's immune to bugs or security vulnerabilities. It just means that the major release is updated and supported longer - but you still need to be able to apply patches and security fixes to that major release. Yes, it's easier to go from 1.20.1 to 1.20.5 than to 1.21, because there's less chance of breakage and less things that will change, but the process is pretty much the same - check for breaking changes, read changelogs, apply everything. The risk is less, might be slightly faster, but fundamentally, it's the same process. If the process is too heavy and takes you too long, having it be slightly faster won't be a gamechanger. So LTS brings slight advantages to the operator, while adding potentially significant complexity to the developer (generally backporting fixes into years old versions isn't fun). The specific proposed LTS falvour is also hardcore, without an upgrade path to the next LTS. The exact type of org that needs an LTS will be extremely reluctant to having to redo everything, 2 years later, with potentially drastic breaking changes making that change very hard.
- x86x87 3y agoThat's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. If you're at a point where a patch for LTS looks like an upgrade to the new version you've screwed up LTS. Also, getting to the point of having an LTS and actually providing the support is expensive. You need experts that can backport security fixes and know the product inside out.
- sofixa 3y ago> That's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. How do you do that on something as complex and with as many moving parts as Kubernetes? And how do you as an operator update that many things without checking there's no breaking changes in the patch?
- x86x87 3y agobingo! how do you do it? and do you want that kind of complexity to begin with?
- sofixa 3y agoDon't ask me, I'm firmly in the HashiCorp Nomad camp: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at... (Note: quite old, some things are no longer true most notably around Nomad downsides)
- rigrassm 3y agoI'm with you, Nomad is highly underrated!
- Rantenki 3y agoWe upgrade our distros pretty much fearlessly, all the time. While I have had breakage from Kernel upgrades, they've been very rare (and generally related to third party closed drivers). Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place.
- eddythompson80 3y ago> Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place. eh, the kernel is an incredibly mature project with 1 machine scope. The kernel also has decades of operating systems research and literature to build on. Kubernetes in comparison is new, distributed and exploring uncharted territory in terms of feature set and implementation. Sometimes bad decisions are made, and it's fair to not want to live with them forever. The kernel project looks very different today than it did in 1999. There is a happy medium though, especially that Kubernetes is kinda far from it.
- Karellen 3y ago> but the process is pretty much the same - check for breaking changes, Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. (...of course, there's no guarantee that you're not relying on buggy (or, at least, accidental) behaviour. People relying on `memcpy(3)` working as expected when the ranges overlap, simply because it happened to do so with historic versions of the `libc` implementation they most commonly happened to test with, is one example. But see also obxkcd https://xkcd.com/1172/ https://xkcd.com/1172/ and https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ )
- sofixa 3y ago> Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. Or a security vulnerability has forced a breaking change. Or any other issue, which is why you have to check.
- Karellen 3y ago> Or a security vulnerability has forced a breaking change. Theoretically, I suppose? Do you have a historic example in mind? I've been running Debian "stable" in its various incarnations on servers for over a decade, and I can't remember any time any service on any installation I've run had such an issue. But my memory is pretty bad, so I might have missed one. (Or even a dozen!) But I have `unattented-upgrades` installed on all my live servers right now, and don't lose a wink of sleep over it.
- sofixa 3y agoYes, I have an example in mind - https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-unattended-samba-upgrade-broke-our-setup https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-una... Yes, it's Ubuntu, but doesn't matter - sometimes security fixes require a breaking change and there's nothing that can be done to avoid it.
- natbennett 3y agoThis happens all the time on systems that are running hundreds of thousands of apps across hundreds of customers. The worst one I know: for a while basically all Cloud Foundry installations were stuck behind a patch release because the routing component upgraded their Go version and that Go version included an allegedly non-breaking-change that caused it to reject requests with certain kinds of malformed headers. The Spring example app has a header with the specific problem impacted. And the vast majority of Cloud Foundry apps are Spring apps, many of which got started by copying the Spring example app. So upgrading CF past this patch release required a code change to the apps running on the platform. Which the people running Cloud Foundry generally can’t get — there’s usually a team of like 12 people running them and then 1000s of app devs.
- barryrandall 3y agoNo open source package that's given away for free needs to or should pursue LTS releases. People who want LTS need a commercially-supported distribution so that they can pay people to maintain LTS versions of the product they're using.
- waynesonfire 3y agoMaybe my team of 15 engineers that manage the k8s stack can do it.
- watermelon0 3y agoNot saying that companies shouldn't pay for extended support, but many other open source software have LTS releases with multi-year support (e.g. Ubuntu/Debian 5 years for LTS releases, and Node.js for 2.5 years.) Additionally, I think one of the major reason for LTS is that K8s (and related software) regularly introduces breaking changes. Out of all the software that we use at work, K8s probably takes the most development time to upgrade.
- ses1984 3y agoPeople pay for longer versions of that, called extended support, ubuntu provides a cut down version for free.
- airocker 3y agoMaybe GKE and EKS should make LTS versions.
- stackskipton 3y agoAzure already has. https://learn.microsoft.com/en-us/azure/aks/long-term-support https://learn.microsoft.com/en-us/azure/aks/long-term-suppor...
- airocker 3y agoTwo years may or may not be enough "long term" depending on the product and the business.
- master_crab 3y agoYes K8s needs LTS. AWS released LTS version of EKS last month. 1.23 is on Extended support until next year for free. But later versions will cost money. https://aws.amazon.com/blogs/containers/amazon-eks-extended-support-for-kubernetes-versions-available-in-preview/ https://aws.amazon.com/blogs/containers/amazon-eks-extended-...
- oneplane 3y agoSo in other words: no it doesn't. The CNCF does its thing, and if you want something else, you can give money to AWS or Azure or GCP and have your cake and eat it too. I'd rather not see the resources in the Kubernetes project being re-directed to users who are in a situation where they aren't able to do a well-known action at planned intervals two or three times per year.
- master_crab 3y agoNo it definitely does. Two of the reasons we gave an ultimatum to teams to get off K8s is that CNCF doesn’t like LTS and loves introducing breaking changes about every other version. Unless you have a third party product with a helm chart that really needs to run on EKS, our dev teams have been told to use ECS
- oneplane 3y agoSounds like you are into some 90's top-down management mentality right there. Good luck with that. (in my experience, that no longer works well, and is a poor solution applied in highly regulated markets and mainframe land) That makes your lack of high speed software supply chain management not go away, and with ECS you'll just get mandatory Fargate update notifications, unless you're using ECS EC2 in which case, what you are even doing? We don't allow systems that cannot be mass-managed and don't have an active lifecycle that keeps up with demand. Sometimes that doesn't mean K8S, but often it does because it's the only large adopted system with a high speed reconciler. But the teams get to choose that on their own. Edit: let me rephrase that a little. It seems that software lifecycle speed and business speed interact to a degree where if the software is faster than the business, you need a different product or speed up the business. For me, that would mean that if either aren't an option, I'd change jobs because it doesn't align with my standards.
- JohnMakin 3y agoNot on the rough checklist - - pray to god whatever helm chart dependencies you have didn't break horribly, and if they did, that there's a patch available
- starttoaster 3y agoThere are tools to tell you if any of your deployed infrastructure uses a deprecated API. I mean, even if the tool didn't exist you could view the deprecation guide, scroll through the kubernetes versions you'll be upgrading through, and inspect your cluster for any objects using a Kind defined by any of those APIs. It's a burden but when is maintaining infrastructure not a burden? https://kubernetes.io/docs/reference/using-api/deprecation-guide/ https://kubernetes.io/docs/reference/using-api/deprecation-g...
- JohnMakin 3y agoCRD's can break in weird ways that aren't always picked up by a tool like that.
- starttoaster 3y agoI mean, CRD API changes/deprecations are, by nature, not reported by the kubernetes project because they're maintained by whatever software provider you installed them from. Unless you're saying that the CRD creates generic kubernetes resources in the cluster that is owned by the CR. In which case, the tool should definitely pick up the generic object at least, but not necessarily tell you that you need to update your version of the CRD, again because the CRD is owned by a third party.
- JohnMakin 3y agoThird party CRD’s are ubiquitous in every single kubernetes-based infra I’ve yet seen so far in my career. Nowhere was I making a statement vis a vis how it should be, but that dependencies like this have yet not to become a huge headache on every K8s upgrade I’ve done (started on 1.12, I just moved my current project to 1.27, and have done most upgrades in between). Given that the kubernetes dev cycle is so fast and API changes are so frequent, combined with the fact that the kubernetes API is designed to be built on top of, this tends to be a real hassle in the real world was my only point. Compare it to something like the “old” way of doing it, some monolith running on a container in a VM - that stuff could run until the cows come home with very few issues or intervention.
- yrro 3y agoFor comparison, Red Hat denote every other major release of OpenShift (4.8, 4.10, 4.12) as Extended Update Support releases, with 2 years of support.
- stbenjam 3y ago2 years is still not very long for large enterprises, they'll be conservative on the uptake - 3-6 months after it gets released, so they only have maybe 12-18 months before they're planning the next upgrade. It's better than vanilla kube, but compare that to RHEL where you get 10 years of maintenance, and can often extend that even further.
- gtirloni 3y agoKubernetes LTS goes by different names: AWS EKS, Azure AKS, Google GKE, SUSE Rancher, etc.
- watermelon0 3y agoNot sure about the others, but AWS EKS support closely follows upstream. Only recently they released paid extended support in preview, which extends support for an additional year.
- Arnavion 3y agoEKS still supports 1.24+ as part of "standard support". Upstream only supports 1.26+ (upstream's policy is "latest 3 versions").
- sciurus 3y agoEKS also added support for 1.24 seven months after the upstream release date. https://endoflife.date/amazon-eks https://endoflife.date/amazon-eks
- Kelkonosemmel 3y agoGKE does Auto upgrade and enforces them. You can't get a really old k8s version on gke
- mardifoufs 3y agoAks is usually like one version behind, and they deprecate older versions every 3 months IIRC. In a one year window of versions, that's not necessarily long term at all. The upgrade process is surprisingly kind of okay though, which is already a lot better than what I usually expect from Azure.
- emb3dded 3y agoAKS has LTS: https://azure.microsoft.com/en-us/updates/generally-available-long-term-support-version-in-aks/ https://azure.microsoft.com/en-us/updates/generally-availabl...
- superkuh 3y agoWhy don't you just put your Kubernetes in Kubernetes? Containers solve everything re: dependency management, right?
- deleted 3y ago[deleted]
- dilyevsky 3y agoWe actually did just that with just apiserver+sqlite-based storage and it works pretty well. Ofc you have the same problem with an outer cluster now but if you’re careful with locking your entire state into the “inner” cluster you can escape that
- geodel 3y agoFully agree. They can follow Cloud Native deployment best practices for guidance.
- nonameiguess 3y agoI think you're saying this sarcastically, but every Kubernetes component except the kubelet already can, and usually does in most distros, run as a Pod managed by Kubernetes itself. Edit the manifest, systemctl restart kubelet, and there you go, you just upgraded. Other than kube-proxy, which has a dependency on iptables, I don't even think any other component has any external dependencies, and the distroless containers they run in are pretty close to scratch. Running them in Pods is more for convenience than dependency management. The actual issue that needs to be addressed when upgrading is API deprecations and incompatibility. For better or worse, when people try to analogize this to the way Red Hat or Debian works, Kubernetes is not Linux. It may not always perfectly achieve the goal, but Linux development operates on the strict rule that no changes can ever break userspace. Kubernetes, in contrast, flat out tells you that generally available APIs will stay available, but if you use beta or alpha APIs, those may or may not exist in future releases. Use them at your own peril. In practice, this has meant virtually every cluster in existence and virtually every cluster management tool created by third parties, uses at least beta APIs. Nobody heeded the warning. So upgrades break stuff all the time. Should they have not ever released beta APIs at all? Maybe. They actually did start turning them off by default a few releases back, but the cultural norms are already there now. Everybody just turns them back on so Prometheus and cert-manager and Istio and what not will continue working.
- ckdarby 3y agoIt forces keeping up to date in a reasonable period of time. It is a feature, not a bug. I agree with comments in this thread that LTS should be a commercial bought option only.
- edude03 3y agoI know it rarely happens in practice, but I disagree than Kubernetes needs an LTS since the Kubernetes cluster itself should be a "cattle not a pet" and thus, you should "just" spin up a new cluster with the same version as your upgrade strategy.
- dilyevsky 3y agoWorks great for clusters that dont have any storage, loadbalancers, any configuration drift…
- simiones 3y agoThe whole point of Kubernetes is to move all of the machine management work to one place - the Kubernetes cluster nodes - so that your services are cattle, not pets. But there is no equivalent system for whole clusters. To transition from one cluster to another, you have to handle everything that k8s gives you without relying on k8s for it (restarting services, rerouting traffic, persistent storage etc). If you can do all that without k8s, why use k8s in the first place? So no, in practice clusters, just like any other stateful system, are not really cattle, they are pets you care for and carefully manage. Individual cluster nodes can come and go, but the whole cluster is pretty sacred.
- Kelkonosemmel 3y agoGlobal load balancer and traffic shifting is a real thing. But my preferred way is in place as it's easy: upgrade one Ctrl plane node after the other, than upgrade every node or create a new one and move your workload. The power of k8s is of course shit when you only move legacy garbage into the cloud as those are not ha at all. But lucky enough people slowly learn how to write apps cloud native.
- simiones 3y agoEven with the best cloud native apps that can really be easily moved from one node to another, it is not easy to do cluster upgrades. Upgrading one control plane node after another can still break your cluster when you have your control plane nodes running incompatible versions. If you get a big worker node failover during the control plane upgrade, which of the control plane nodes should provision the new worker nodes, which versions of containers should kubelet pull, etc? The upgrade process for a running multi-node solution is an extremely difficult technical challenge, and it has to be designed very carefully to actually have a good chance of working. Breaking changes are almost impossible to introduce if the goal is to upgrade gracefully - you probably need a special compatible version to run while the upgrade is still in progress, that is neither the old version nor the targeted new version, but can handle both. You need to make sure all APIs are versioned in a way where it's easy to distinguish a request coming from an old client (so you can serve the old format) or a new one etc. Kubernetes has done some work on this, but it still often requires manual work. And many common plugins have just not put in this work and will break if you try to upgrade a live cluster.
- mschuster91 3y agoAs someone working in managing a bunch of various Kubernetes clusters - on-prem and EKS - I agree a bit. Managing Kubernetes versions can be an utter PITA, especially keeping all of the various addons and integrations one needs to keep in sync with the current Kubernetes version. But: most of that can be mitigated by keeping a common structure and baseline templates. You only need to validate your common structure against a QA cluster and then roll out necessary changes onto the production cluster... but most organizations don't bother and let every team roll their own k8s, pipelines and whatnot. This will lead to tons of issues inevitably. Asking for a Kubernetes LTS is in many cases just papering over organizational deficiencies.
- HPsquared 3y agoTo borrow from reliability engineering, software failures in practice can approximate a "bathtub curve". That is: an initial high failure rate (teething problems), a low failure rate for most of the lifespan (when it's actively maintained), then gradually increasing failure rate (in hardware this is called wear-out). Unlike hardware, software doesn't wear out but the interfaces gradually shift and become obsolete. It's a kind of "gradually increasing risk of fatal incompatibility". Something like that. I wonder if anyone has done large-scale analysis of this type. Could maybe count CVEs, but that's just one type of failure.
- kevin_nisbet 3y agoFrom my perspective as a former developer on a kubernetes distribution that no longer exists. The model seems to largely be, the CNCF/Kubernetes authors have done a good job of writing clear expectations for the lifetime for their releases. But there are customers who for various reasons want extended support windows. This doesn't prevent the distribution from offering or selling extended support windows, so the customers of those distributions can put the pressure on those distribution authors. This is something we offered as a reason to use our distribution, that we can backport security fixes or other significant fixes to older versions of kubernetes. This was especially prevalent for the customers we focussed on, which were lots of clusters installed in places without remote access. This created a lot of work for us though, as whenever a big security announcement came out, I'd need to triage on whether we needed a backport. Even our extended support windows were in tension with customers, who wanted even longer windows, or would open support cases on releases out of support for more than a year. So I think the question really should be, should LTS be left to the distributions, many of which will select not to offer longer releases than upstream, but allow for some more commercial or narrow offerings where it's important enough to a customer to pay for it. Or whether it should be the responsibility of the Kubernetes authors and in that case what do you give up in project velocity with more work to do on offering and supporting LTS. I personally resonate with the argument that this can be left with the distributors, and if it's important enough customers to seek out, they can pay for it through their selected distribution, or switching distributions. But many customers lose out, because they're selecting distributions that don't offer this service, because it is time consuming and difficult to do.
- edude03 3y agoMaybe more importantly, you could get a distribution to support you but what about upstream projects? It'd be a big lift (if not impossible) to get projects like cert-manager cilium whatever to adopt the longer release cycle as well. Is it normal for a distribution to also package upstream projects that customers want?
- kevin_nisbet 3y ago> It'd be a big lift (if not impossible) to get projects like cert-manager cilium whatever to adopt the longer release cycle as well. It's a great point which probably should be part of the discussion. Say even if Kubernetes project offered LTS, how would that play into every other project that is pulled together. > Is it normal for a distribution to also package upstream projects that customers want? I suspect it differs by distribution. The distribution I worked on included a bunch of other projects, but it was also pretty niche.
- Kelkonosemmel 3y agoThat's the big advantage of kubernetes: You need to keep updating it! No legacy 10 year old garbage in the corner just because managers don't understand that this is shit. And for me it's the perfect excuse NOT having those garbage discussions. And honestly k8s doesn't change that much anyway
- simiones 3y agoHonestly, rather than an LTS, I think k8s needs a much better upgrade process. Right now it is really poorly supported, without even the ability to jump between versions. If you want to migrate from 1.24 to 1.28, you need to upgrade to 1.25, then 1.26, then 1.27, and only then can you go to 1.28. This alone is a significant impediment to the normal way a project upgrades (lag behind, then jump to latest), and would need to be fixed before any discussion of an actual LTS process.
- jbnorth 3y agoYour problem here is waiting until 1.28 to be released before starting your upgrades from 1.24. A version or two is one thing but 4 versions and you’re just asking to be constantly struggling to keep your head above water.
- FridgeSeal 3y agoI disagree. Software is a garden that needs to be tended. LTS (and to a lesser extent requirements for large amounts of backwards compatibility) arguments are the path the ossification and orgs running 10+ year out of date, unsupported legacy garbage that nobody wants to touch, and nobody can migrate off because it’s so out of whack. Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”. Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.
- iwontberude 3y agoKubernetes doesn't improve sufficiently to justify the broken compatibility anymore. Projects slow down and become mature. This isn't a bad thing.
- FridgeSeal 3y ago> Projects slow down and become mature. This isn't a bad thing I completely agree, but that’s also not-incompatible with my argument.
- jmspring 3y agoI happen to agree. Many "new features" aren't necessarily needed by a large segment of the community that uses K8s. Some of the attitudes in here about software needs to be constantly evolving is just odd. Many many systems run on legacy software and hardware because, for the most part, they just work. Innovation and evolution are important, yes. But churn for the sake of churn does not fit every (most) use case...
- MuffinFlavored 3y ago> long-term support > out of date, unsupported legacy garbage
- 3y ago
- airocker 3y agoThis is sorely needed. It causes service disruption for weeks because of weird upgrade schemes in GKE (or any other vendor) which works on top of Kubernetes. There are too many options with non-intuitive defaults related to how the control plane and the node pools will be repaired or upgraded. Clusters get upgraded without our knowledge and breaks service arbitrarily. Plus, if you are doing some infrastructure level changes, you have to put in extreme effort to keep upgrading. IMHO , infrastructure is too low level for frequent updates. Older versions need LTS.
- MuffinFlavored 3y agohow much of what you said is part of the complexity that comes with Kubernetes in general? at the end of the day it’s about cramming a ton of functionality into something like YAML is it not?
- airocker 3y agoOur cluster has reasonable complexity and yamls work well so far. Maybe we will see what you are saying if we scale further. It allows us to maintain a much more complex cluster with fewer developers. And the software is well tested/reliable. But not having LTS is really difficult. It is impossible to keep rewriting things. And all related components: terraform, helm, GKE etc also update too quickly for us to manage with a small team. We do some lower level infrastructure work so these problems bites us harder.
- tomjen3 3y agoLets Encrypt will only give you a cert that is good for 3 months, because they want you to automate the update. I don't think K8S should create an LTS, I think they should make it dirt simple to update.
- rafaelturk 3y agoIn my opinion if you need a stable production enviroment; MicroK8s stands out as the Kubernetes LTS choice. Canonical managed to create a successfull robust and stable Kubernetes ecosystem. Despite its name, MicroK8s isn't exactly 'micro'—it's more accurately a highly stable and somewhat opinionated version of Kubernetes.
- nezirus 3y agoThe cynical voice inside me says it works as intended. The purpose of k8s is not to help you run your business/project/whatever, but a way to ascend to DevOps Nirvana. That means never-ending cycle of upgrades for the purpose of upgrading. I guess too many people are using k8s where they should have used something simpler. It's fashionable to follow the "best practices" of FAANGs, but I'm not sure that's healthy for vast majority of other companies, which are simply not on the same scale and don't have armies of engineers (guardians of the holy "Platform")
- oneplane 3y agoNo it doesn't. If you can't keep up you have two options: 1. Pay someone else to do it for you (effectively an LTS) 2. Don't use it Software is imperfect, processes are imperfect. An LTS doesn't fix that, it just pushes problems forward. If you are in a situation where you need a frozen software product, Kubernetes simply doesn't fit the use case and that's okay. I suppose it's pretty much all about expectations and managing those instead of trying to hide mis-matches, bad choices and ineptitude. (most LTS use cases) It's essentially x509 certificate management all over again; if you can't do it right automatically, that's not the certificate lifetime's fault, it's the implementors fault. As for option 1: that can take many shapes, including abstracting away K8S entirely, replacing entire clusters instead of 'upgrading' them, or having someone do the actual manual upgrade. But in a world with control loops and automated reconciliation, adding a manual process seems a bit like missing the forest for the trees. I for one have not seen a successful use of K8S where it was treated like an application that you periodically manually patch. Not because it's not possible to do, but because it's a symptom of a certain company culture.
- mountainriver 3y agoThere have been numerous proposals for LTS releases within the k8s community. I’m curious if someone could chime in as to why those haven’t succeeded
- b7we5b7a 3y agoPerhaps it's because I work in a small software shop and we do only B2B, but 99% of our applications consist of a frontend (JS served by an nginx image), a middleware (RoR, C#, Rust), nginx ingress and cert-manager. Sometimes we have PersistentVolumes, for 1 project we have CronJobs. SQL DBs are provisioned via the cloud provider. We monitor via Grafana Cloud, and haven't felt the need for more complex tools yet (yes, we're about to deploy NetworkPolicies and perform other small changes to harden the setup a bit). In my experience: - AKS is the simplest to update: select "update cluster and nodes", click ok, wait ~15m (though I will always remember vividly the health probe path change for LBs in 1.24 - perhaps a giant red banner would have been a good idea in this case) - EKS requires you to manually perform all the steps AKS does for you, but it's still reasonably easy - All of this can be easily scripted I totally agree with the other comments here: LTS releases would doom the project to support >10y-old releases just because managers want to "create value", but don't want to spend a couple weeks a year to care for the stuff they use in production. Having reasonably up-to-date, maintainable infrastructure IS value to the business.
- cyrnel 3y agoGood overview! I'd personally rather have better tooling for upgrades. Recently the API changes have been minimal, but the real problem is the mandatory node draining that causes downtime/disruption. In theory, there's nothing stopping you from just updating the kubelet binary on every node. It will generally inherit the existing pods. Nomad even supports this[1]. But apparently there are no guarantees about this working between versions. And in fact some past upgrades have broken the way kubelet stores its own state, preventing this trick. All I ask is for this informal trick to be formalized in the e2e tests. I'd write a KEP but I'm too busy draining nodes! [1]: https://developer.hashicorp.com/nomad/docs/upgrade https://developer.hashicorp.com/nomad/docs/upgrade
- op00to 3y ago100% as someone who used to support Kubernetes commercially, long term support is an engineering nightmare with kube. My customers who could upgrade easily were more stable and easily supported. The customers that couldn’t handle an upgrade were the exact opposite - long support cases, complex request processes for troubleshooting information, the deck was probably stacked against them from the start. Anyway, make upgrades less scary and more routine and the risk bubbles away.
- linuxftw 3y ago> the real problem is the mandatory node draining that causes downtime/disruption. This sounds a lot like "We don't actually patch the OS" which is quite common among many companies. As a former enterprise kubernetes distro maintainer, I can tell you with certainty that most on-premise kubernetes customers aren't patching their machines between kubernetes releases, and try to stay on kubernetes releases for 18+ months.
- cyrnel 3y agoI have hope for that problem to be solved too, with some combination of minimal kernels and live patching. I really don't think running two copies of everything and hammering CPU/RAM/Disk/Network with constant drain operations is a permanent solution for applying patches.
- 3y ago
- voytec 3y ago> The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > This bot triages issues according to the following rules: ... > /close
- rdsubhas 3y agoMany comments here disagreeing about LTS just because it won't be up-to-date – are missing a critical point. When people run a rolling release on their server, their original intent is "Yay I'll force myself to be up-to-date". Reality is, they get conflicts on installed 3rd party software in each upgrade. What ends up happening is, they get frozen on some point in time, without even security patches for a long time. k8s is like an OS, it's not just the core components, there is <ingress/ gateway/ mesh/ overlays/ operators/ admission controllers/ 3rd party integrations like vault, autoscalers, etc>. Something or the other breaks with each rolling release. I've grown really tired of the way GKE v1.25 pretends to be a "minor" automated upgrade, when it removes or changes god knows how many APIs. This is what is happening in kubernetes land. The broken upgrade fatigue is real, but it's hampered by wishful thinking.
- houseofzeus 3y agoWell, that and there appears to at least be a commercial market for an LTS based on announcements from Red Hat, Azure, and now Amazon. Azure are the ones who effectively posed the question in the community of hey, since we all appear to be doing this is there any value in collaborating on it in the open by rebooting the LTS working group. Now that said, while you are absolutely right that third party solutions are the biggest boat anchor to users upgrading whether commercial or open source, they are also the reason an LTS doesn't necessarily fix much because then those things need an LTS too and/or they decide hey this is great we will fix on this one and it becomes an even bigger lift when the ISV does need to bring their stuff up to latest.
- vbezhenar 3y agoIt's not an issue with LTS, but just with the fact that kubernetes moves too fast and breaks backwards compatibility too often. I don't care about particular version as long as I can keep auto upgrade at night and don't care about breakage. May be kubernetes is too young at this point...
- sitkack 3y agoK8S needs a 1.0 and some stability. It is still a public PoC.
- fostware 3y agoFor a group so devoted to "cattle, not pets", so many responses here indicate an almost constant need for hands-on effort from upgrade testing, UAT, right through to post-upgrade hypercare. I'd like a slightly longer LTS purely so I'm not having to spend all my time spinning the plates to keep things up. I don't need 10 years LTS, I need three so I can work with the rest of the enterprise that moves even slower.
- mt42or 3y agoYour Kubernetes setup is just totally fuck off, fire the team and hire better ones.
- kristianpaul 3y agoOnce its stable a LTS could be of help
- solatic 3y agoThe economic math is very simple: organizations that release faster are more responsive to the market, have lower operational costs, and are therefore more efficient. Market players that release slower will get squeezed by more efficient players. It may happen later rather than sooner but it is inevitable. As a business, you can decide to become more efficient, or you can decide to try to support the status quo that enjoys internal political equilibrium. Smart businesses go for the former, most businesses go for the latter.
- pacoxu2023 3y agoIs there an option to add new cluster with new version each year? Keep 3-4 clusters. Deploy your new application with new addons in new cluster. Keep old application running on old ones.
- essdodson 3y agoKubernetes WG-LTS is seeking feedback, both from those who would and would not make use of longer support cycles. Survey at https://bit.ly/k8s-upgrade-survey https://bit.ly/k8s-upgrade-survey
- belter 3y agoI found it a little weird the acronym was not spell out in full term.[1] After all, we could also looks for a discussion in the article about the need for an STS, and how that should defined before GA or after the RC. Definitely at least before the EA versions of Kubernetes and at least before EOL of the product.... [1] https://en.wikipedia.org/wiki/Long-term_support https://en.wikipedia.org/wiki/Long-term_support