8 ms·

I disagree. Software is a garden that needs to be tended. LTS (and to a lesser extent requirements for large amounts of backwards compatibility) arguments are

I disagree. Software is a garden that needs to be tended. LTS (and to a lesser extent requirements for large amounts of backwards compatibility) arguments are the path the ossification and orgs running 10+ year out of date, unsupported legacy garbage that nobody wants to touch, and nobody can migrate off because it’s so out of whack. Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”. Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself.

iwontberude 3y ago
Kubernetes doesn't improve sufficiently to justify the broken compatibility anymore. Projects slow down and become mature. This isn't a bad thing.
- FridgeSeal 3y ago
  > Projects slow down and become mature. This isn't a bad thing I completely agree, but that’s also not-incompatible with my argument.
- jmspring 3y ago
  I happen to agree. Many "new features" aren't necessarily needed by a large segment of the community that uses K8s. Some of the attitudes in here about software needs to be constantly evolving is just odd. Many many systems run on legacy software and hardware because, for the most part, they just work. Innovation and evolution are important, yes. But churn for the sake of churn does not fit every (most) use case...
MuffinFlavored 3y ago
> long-term support > out of date, unsupported legacy garbage
- jen20 3y ago
  You can probably remove the "unsupported" part without changing the overall correctness of the statement.
baby_souffle 3y ago
> Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”. You can't see me, but I'm violently nodding in agreement. Faithfully adhering to these best practices isn't always possible, though; management gonna manage how they manage and now how your ops team wants them to manage. > Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself. How different could things be if k8s had a "you don't break userland!" policy akin to the way the Linux kernel operates? Is there a better balance between new stuff replacing the old and never shipping new stuff that would make more cluster operators more comfortable with upgrades?
mfer 3y ago
Consider where people would use something for a long time and want to keep it relatively stable for long periods? Planes, trains, and automobiles are just a few of the examples. How should over the air updates for these kinds of things work? Where are all the places that k8s is being used? If we only think of open source in datacenters we limit our thinking to just one of the many places it's used.
- oneplane 3y ago
  Or, they could use something that is not Kubernetes. If you are working with a system that is super static, Podman comes to mind.
- samus 3y ago
  You don't need Kubernetes for over-the-air updates. Kubernetes is also not suitable for that. It scales up alright, but not down. As TA expounds, there are simply too many moving parts that require expertise to operate. And that's fine. Not every software has to be able to accommodate all use cases.
  - Palomides 3y ago
    they run kubernetes on fighter jets now
    - fungalcretin007 3y ago
      That is because the pressure on defense to adopt an approach and technologies that 'work' in commercial space has become dogma. Agile is a disease that fuels recipe approaches. Recently worked with a team doing edge architecture and design that has had at least 5 iterations of k8s design and vendors. Why is that? What was the solution? My opinion at the end (even though I love cloud managed k8s for commodity/insecure in the commercial space) is not to use k8s for edge and secure. Stop pretending. It's a mess and it won't get better.
    - _w1tm 3y ago
      What kind of pilot would upgrade their fighter jet cluster midair? These are made up problems.
incahoots 3y ago
I agree on the very principal of what you're laying out here, but the reality is often rare if not at all in tandem with principals and "best practices" Manufacturing comes to mind, shuttering a machine down to apply patches monthly is going to piss off the graph babysitters, especially if the business is a 24/7 operation, and most are currently. In an ideal world there would be times every month to do proper machine maintenance, but that doesn't translate to big money gains for the glutton of shareholders who don't understand anything, let alone understand that maintenance prolongs processes as opposed to running everything ragged.
- dwattttt 3y ago
  You can also run your car for longer if you never stop to take it to a mechanic too.
  - Volundr 3y ago
    I don't think the comparison to taking your car to a mechanic is a good one. When I take my car to a mechanic they are returning it to a stock configuration. They don't need to update to a new, slightly different power steering this month, new brakes next, then injectors....
    - dwattttt 3y ago
      Certainly. But it's being compared to not taking systems down to do maintenance at all. > if the business is a 24/7 operation > In an ideal world there would be times every month to do proper machine maintenance
    - ClumsyPilot 3y ago
      > They don't need to update to a new, slightly different power steering this month, new brakes next, then injectors.... Then they find a compatability problem and the car n3eds to be rebuilt
wouldbecouldbe 3y ago
Kubernetes feels like Javascript has reached the sysadmins, new updates, libraries, build tools every week. It's mainly good for keeping high paid people employed, not keeping your servers stable. I ran a cluster 1.5 years in production, took me so much energy. Especially that one night where digital ocean managed cluster forced an update that crashed all servers; and there was no sane way to fix it. I'm back to stability with old school VPS; it just works. Every now and then you run a few patches. Simple & fast deploys; what a blessing.
- hot_gril 3y ago
  Javascript stuff tends to stay working, unless you're talking about React which breaks all the react-whatever libs every time.
- firesteelrain 3y ago
  I am not understanding how you say you ran a cluster when you were using a managed instance so you weren’t really running it. Now , going to a VPS makes you effectively the MSP. I don’t see how that addresses the LTS issues since k8s is updating so often that you have to keep pace with the updates (that you now need to do). I use a managed instance and manage my own private instance.
  - sitkack 3y ago
    You do understand, please don’t use that rhetorical device here.
    - firesteelrain 3y ago
      Please don’t be a dick here
  - btasker 3y ago
    > I don’t see how that addresses the LTS issues since k8s is updating so often that you have to keep pace with the updates (that you now need to do). They're not saying they moved to running k8s on a VPS - they're saying they moved to using a VPS instead of k8s (i.e. have escaped the k8s upgrade cycle).
jl6 3y ago
That’s great if you have control over all the moving parts, but a lot of real-world (i.e. not exclusively software-based) orgs have interfaces to components and external entities that aren’t so amenable to change. Maybe you can upgrade your cluster without anybody noticing. Maybe you’re a wizard and upgrades never go wrong for you. More likely, you will be constrained by a patchwork quilt of contracts with customers and suppliers, or by regulatory and statutory requirements, or technology risk policies, and to get approval you’ll need to schedule end-to-end testing with a bunch of stakeholders whose incentives aren’t necessarily aligned to yours. That all adds up to $$$, and that’s why there’s a demand for stability and LTS editions.
- Alcatros552 3y ago
  @ji6 I have to tell you, we are using Kubernetes since the very early versions available. If you automate everything you will never have an issue with K8, you can deploy all the required dependencies in one go. You can run the tests, if done correctly this literally requires 10 minutes. This argument to have an LTS Version for me is like air. It's the kind of Nuclear Reactor argument. We did have the money to buy Uranium to burn but now we can't afford to dispose it normally. Kubernetes is for flexibility not for big companies which want to use their software development processes of the 90's...
  - JamesonNetworks 3y ago
    Yeah? Through the RBAC changes? Because thats where I gave up
  - dvfjsdhgfv 3y ago
    I'm sorry, this is simply not true. K8s relatively often introduces breaking changes. They are announced well in advance, which is very nice, but the solution will not take 10 minutes. Take Pod Security Policies and migration to Kyverno of OPA Gatekeeper. Even if you have a relatively simple cluster and choose to stick to defaults rather than write your own rules, it will still take time to understand the new system, to choose the policies you need, to update and test it. In complex cluster this takes weeks and in some cases even months, especially if you are legally obliged to follow change management.
Waterluvian 3y ago
Sometimes I get this feeling that a lot of developers kind of want to be in their code all the time, tending to it. But it’s just not a good use of my time. I want it to work so I can ignore it and move on to the next best use of my time. I trial upgraded a years old Django project today to 5.0 and it took almost zero work. I hadn’t touched versions (other than patch)in over a year. That’s the way I want it. Admittedly this was less about an LTS and more about sensible design with upgrading in mind.
- selcuka 3y ago
  > I trial upgraded a years old Django project today to 5.0 and it took almost zero work. Django is great at backwards compatibility, but to be honest they haven't added many revolutionary features to Django for years, except a half-baked implementation of async support.
maximinus_thrax 3y ago
To paraphrase someone else's reaction, I'm violently shaking my head in disagreement. What you're saying only works when you have 100% full control of everything (including the customer data). As someone who spent years in the enterprise space, what you're describing is akin to 'Ivory Tower Architecture'. LTS is a commitment. That is all. If someone is uncomfortable with such a commitment, then that's fine, let me free market sort it out. But what LTS does is it tells everyone (including paying customers) that the formats/schemas/APIs/etc.. in that version will be supported for a very long time and if I adopt it, I won't have to think about it or budget too much for its maintenance for a period of time measured in months/years. I would go the extra mile here and say that offline format should be supported FOREVER. None of that LTS bs for offline data, ever. LTS means that you're accountable for some of the costs in your partnership with the customers. If you move fast and break things, they will have to work extra just to keep up. If you move fast but mindful with back-compat, you will work extra but your customer will be happier. That is all. Re-reading your comment gives me chills and reinforces my belief that I will never pay money to Google (they have a similar gung-ho attitude against 'legacy garbage') or have any parts of my business depend on stuff which reserves the liberty of breaking shit early and often.
- JackSlateur 3y ago
  I worked in a public cloud compagny We had very few control over the customers stuff We did not had to : avoid breaking changes (or impose them) and transparent upgrades were the key
hot_gril 3y ago
Even if you could get everyone to tend the garden, it encourages unnecessary breaking changes or fragile design. LTS provides some much-needed backpressure. If a new feature truly requires breaking things, people who really want it will put in the effort.
- p_l 3y ago
  That's why Kubernetes provides versioned APIs with proper lifecycles and levels of stability. A stable API is not going to change randomly on you, and of it does, it's going to come with deprecation policy so you have time to do it. Personally I did have few "painful" k8s upgrades, but every time it happened it was due to long-announced large scale change... That another team didn't want to upgrade over despite delaying so long the last version compatible became EOL
  - hot_gril 3y ago
    Right, but the support is not very long. You still get basically forced to update at least every 14 months according to the article.
jvans 3y ago
I've spent so much time chasing down performance problems and bugs where the problem was due to an outdated dependency. Simply spending a few hours a month upgrading dependencies is a big win to avoid those situations
cramjabsyn 3y ago
Its not a garden. Gardens are horizontal with no dependencies between plants. Infrastructure is a high rise building. Long term planning and careful maintenance is needed. And it shouldn't be necessary to replace the foundation every year or two.
- cratermoon 3y ago
  > Gardens are horizontal with no dependencies between plants. Two words: companion plantings Also, there's a world of incompatible plants. Some plants actively harm the growth of others, and some simply can't grow in the same place because of different soil and nutrient requirements.
- oceanplexian 3y ago
  Yeah imagine if someone built a bridge and was like, yeah, we need to do regular updates or else it will fail and come crashing down. I feel like software engineers who preach that everything must be connected to the internet and update from the mothership regularly are fundamentally disconnected from reality. If your design is robust to begin with you should be able to depend on it without constantly fiddling with everything.
  - terom 3y ago
    Bridges need regular maintenance or they will fail and come crashing down.
    - mcmcmc 3y ago
      Regular maintenance for bridges means tensioning cables, tightening loose bolts, repainting exposed parts. Not redoing the foundations every month.
      - pas 3y ago
        k8s doesn't require redoing foundations every month this LTS hysteria is completely made-up it benefits from updates every year, also who uses naked k8s (the hard way?), folks use a distribution with an updater
zzyzxd 3y ago
Yup. At my last company, Kubernetes was the only place where software was not 2+ years behind upstream because of this and I appreciate it a lot. And its API deprecation policy [1] made upgrade not so painful (if upgrades frequently break your stuff, check if your infra people really understand this policy, and your software vendors are compliant). 1. https://kubernetes.io/docs/reference/using-api/deprecation-policy https://kubernetes.io/docs/reference/using-api/deprecation-p...
cpeterso 3y ago
At the far end of the update frequency spectrum from LTS is Google's "live at head" philosophy. Google's Abseil C++ library, for example, recommends that consumers update to Abseil's latest commit from the master branch as often as possible. Abseil have no tagged releases besides master and an LTS branch (updated every 6-9 months). https://abseil.io/about/philosophy#we-recommend-that-you-choose-to-live-at-head https://abseil.io/about/philosophy#we-recommend-that-you-cho...
- waynesonfire 3y ago
  The tag is the commit. So what. Same thing.
quickthrower2 3y ago
I like the idea, and with something like say Chrome, this is excellent. However each upgrade of k8s needs planning. Need to check through the list to see what APIs are broken, and figure out if anything needs changing, preparing for it, including stuff the cloud provider has chucked in the mix. Probably to the point I feel like a cluster of clusters would be safer, so you can slowly roll it out.
- p_l 3y ago
  Use multiple clusters for checking out upgrade path at least. Plan regular upgrade so it's moved in small chunks and so that teams are aware of where they stand with APIs etc.
  - fungalcretin007 3y ago
    This is garbage path compared to controlled quantity. In 2008 no one would have countenanced this for good reason. What is the reason now?
cultofmetatron 3y ago
> Do your upgrades and releases frequently this sounds great in theory. after 5years of running a startup, you learn to pick your battles. that db library you upgraded? works great except for this one edge case where they changed the interface in an obscure portion that affects 5% of your users. it got past QA but thaat 5% of users get REAL VOCAL about it. now its in production and you're planning an emrgency revert of that dependncy after you verify that you aren't using any of the new features of the library. sounds doable? now multiply that by the number of dependencies your app has. I agree that it good to periodically update deps and allocate time for keeping your system up to date but Its easy to let tracking all your deps suck up all the time your startup would be better off spending on adding features that bring new users in and add to your bottom line.
- waynesonfire 3y ago
  Absolutely, it's clear the OP hasn't maintained anything at scale. Last thing I want to do is upgrade a rock solid library just because.
  - FridgeSeal 3y ago
    I suggested upgrading like it’s the wild-west where? When your rock solid dependency releases a new major version, when you’re ready you update it, and fix all the things you need to in a single PR, you take that to prod when you’re ready, and roll it back if it’s unhappy, or your favourite variation of out-and-back. What I’m advocating against, is picking a version, and then sitting on it for so long that it inevitably goes out of support, and then find yourself having a really bad, un-fun time-probably at some awful time of day- and discovering the work you need to do to get out of the current system is so much more than you initially realised, because you’re so far behind. Breaking changes happen anyway at some point, you may as well stay abreast of them.
jacurtis 3y ago
I both agree and disagree with you. In theory, I agree. Tend your garden, do small upgrades often instead of major upgrades less often. In a homelab this is easy to implement. In a small organization it is too. But in the facetious "real world", things are a lot more complicated. I work as an SRE Manager and my team is basically ALWAYS upgrading Kubernetes. New releases drop about as fast as we can upgrade the last ones. When you work on a large cluster, doing an upgrade isn't a simple process. It requires a ton of testing, several steps, and being very slow and methodical to make sure it is all done properly. Where I currently work, we have 2 week sprints and infrastructure changes must align with sprint cycles. So to promote an upgrade at the fastest possible schedule it requires: - Week 0: Upgrade Dev environment - Week 2: Upgrade QA Environment - Week 4: Upgrade Sandbox Environment - Week 6: Upgrade Prod Environment That is the fastest possible schedule. That assumes we do a cluster upgrade every sprint, which is 2 weeks. It also ignores other clusters for other business units. We have 4 primary product lines, so multiply all that work times 4. Plus we have supporting products (like self-hosted gitlab, codescene, and custom tools running in k8s clusters). I say fastest possible schedule because, we can't keep up with this schedule, but even if we could it is the fastest we could go and still maintain our deployment and infrastructure promotion policies. With new releases every 3-4 months (12-16 weeks), we are essentially in a constant state of always upgrading kubernetes. Right now my team is 2 versions behind. Skipping versions doesn't make sense because you can't guarantee a safe upgrade when skipping versions. This is why LTS releases are nice. When you run systems at large scale, it is impractical to upgrade that often. I'd prefer to limit upgrades to no more than twice a year and personally I find annual upgrade cycles to be the best balance between "tending the garden" and "not drowning in upgrade work". LTS releases are usually tests to skip upgrades so that companies can go from LTS release to LTS release, without the need to worry about upgrading every minor version in sync. Remember, upgrading K8s clusters isn't what my bosses want to hear my team spends our time. They want to know that observability is improving, devs are getting infrastructure support, we are building out new systems, deploying hardware for the product team, running our resiliency tests, etc.. Sure upgrading is part of the job, but i can't be ALWAYS upgrading. I have a lot of other responsibilities.
- solatic 3y ago
  I don't know what your internal architecture looks like, but on the face of it, arguably you should be running Dev and QA loads on the same cluster. Either you have an organization where an SRE team is responsible for running clusters for teams, in which case why not run Dev and QA on the same cluster, or you are responsible for last-line-of-support for teams responsible for running their own clusters, in which case you say, here's a new version of Kubernetes (e.g. a Terraform module) and in this sprint you are responsible for deploying it through dev, QA, pre-prod, and production clusters. Especially if you have separate Kubernetes clusters per product line (do you really need separate dev Kubernetes clusters per product line?).
stouset 3y ago
100% agreed. Make upgrades easy, automate the tedious parts, and do them as often as possible. If you do upgrades once per several years, yes, it is going to be excruciating.
deleted 3y ago
[deleted]
npacenop 3y ago
This, and very much this. At the end of course the decision will be based on business and market analysis, as even the cloud native foundation abides more or less to the same rules as everyone else does... Introducing LTS for kubernetes however will be a huge step towards pushing it down the enterprise products alley, where software is selected more for the amount of people available in the market, capable to work with it / operate it, and the running costs generated, rather than satisfying an actual need of the business. Chances are that if your org needs LTS for kubernetes, then kubernetes is not the right solution to your problems. Which is probably the case anyway... but that's a whole different story.

iwontberude 3y ago

Kubernetes doesn't improve sufficiently to justify the broken compatibility anymore. Projects slow down and become mature. This isn't a bad thing.

FridgeSeal 3y ago
> Projects slow down and become mature. This isn't a bad thing I completely agree, but that’s also not-incompatible with my argument.
jmspring 3y ago
I happen to agree. Many "new features" aren't necessarily needed by a large segment of the community that uses K8s. Some of the attitudes in here about software needs to be constantly evolving is just odd. Many many systems run on legacy software and hardware because, for the most part, they just work. Innovation and evolution are important, yes. But churn for the sake of churn does not fit every (most) use case...

MuffinFlavored 3y ago

> long-term support > out of date, unsupported legacy garbage

jen20 3y ago
You can probably remove the "unsupported" part without changing the overall correctness of the statement.

baby_souffle 3y ago

> Don’t do this. Tend your garden. Do your upgrades and releases frequently, ensure that everything in your stack is well understood and don’t let any part of your stack ossify and “crust over”. You can't see me, but I'm violently nodding in agreement. Faithfully adhering to these best practices isn't always possible, though; management gonna manage how they manage and now how your ops team wants them to manage. > Upgrades (even breaking ones) are easier to handle when you do them early and often. If you let them pile up, and then have to upgrade all at once because something finally gave way, then you’re simply inflicting unnecessary pain on yourself. How different could things be if k8s had a "you don't break userland!" policy akin to the way the Linux kernel operates? Is there a better balance between new stuff replacing the old and never shipping new stuff that would make more cluster operators more comfortable with upgrades?

mfer 3y ago

Consider where people would use something for a long time and want to keep it relatively stable for long periods? Planes, trains, and automobiles are just a few of the examples. How should over the air updates for these kinds of things work? Where are all the places that k8s is being used? If we only think of open source in datacenters we limit our thinking to just one of the many places it's used.

oneplane 3y ago
Or, they could use something that is not Kubernetes. If you are working with a system that is super static, Podman comes to mind.
samus 3y ago
You don't need Kubernetes for over-the-air updates. Kubernetes is also not suitable for that. It scales up alright, but not down. As TA expounds, there are simply too many moving parts that require expertise to operate. And that's fine. Not every software has to be able to accommodate all use cases.
- Palomides 3y ago
  they run kubernetes on fighter jets now
  - fungalcretin007 3y ago
    That is because the pressure on defense to adopt an approach and technologies that 'work' in commercial space has become dogma. Agile is a disease that fuels recipe approaches. Recently worked with a team doing edge architecture and design that has had at least 5 iterations of k8s design and vendors. Why is that? What was the solution? My opinion at the end (even though I love cloud managed k8s for commodity/insecure in the commercial space) is not to use k8s for edge and secure. Stop pretending. It's a mess and it won't get better.
  - _w1tm 3y ago
    What kind of pilot would upgrade their fighter jet cluster midair? These are made up problems.

incahoots 3y ago

I agree on the very principal of what you're laying out here, but the reality is often rare if not at all in tandem with principals and "best practices" Manufacturing comes to mind, shuttering a machine down to apply patches monthly is going to piss off the graph babysitters, especially if the business is a 24/7 operation, and most are currently. In an ideal world there would be times every month to do proper machine maintenance, but that doesn't translate to big money gains for the glutton of shareholders who don't understand anything, let alone understand that maintenance prolongs processes as opposed to running everything ragged.

dwattttt 3y ago
You can also run your car for longer if you never stop to take it to a mechanic too.
- Volundr 3y ago
  I don't think the comparison to taking your car to a mechanic is a good one. When I take my car to a mechanic they are returning it to a stock configuration. They don't need to update to a new, slightly different power steering this month, new brakes next, then injectors....
  - dwattttt 3y ago
    Certainly. But it's being compared to not taking systems down to do maintenance at all. > if the business is a 24/7 operation > In an ideal world there would be times every month to do proper machine maintenance
  - ClumsyPilot 3y ago
    > They don't need to update to a new, slightly different power steering this month, new brakes next, then injectors.... Then they find a compatability problem and the car n3eds to be rebuilt

wouldbecouldbe 3y ago

Kubernetes feels like Javascript has reached the sysadmins, new updates, libraries, build tools every week. It's mainly good for keeping high paid people employed, not keeping your servers stable. I ran a cluster 1.5 years in production, took me so much energy. Especially that one night where digital ocean managed cluster forced an update that crashed all servers; and there was no sane way to fix it. I'm back to stability with old school VPS; it just works. Every now and then you run a few patches. Simple & fast deploys; what a blessing.

hot_gril 3y ago
Javascript stuff tends to stay working, unless you're talking about React which breaks all the react-whatever libs every time.
firesteelrain 3y ago
I am not understanding how you say you ran a cluster when you were using a managed instance so you weren’t really running it. Now , going to a VPS makes you effectively the MSP. I don’t see how that addresses the LTS issues since k8s is updating so often that you have to keep pace with the updates (that you now need to do). I use a managed instance and manage my own private instance.
- sitkack 3y ago
  You do understand, please don’t use that rhetorical device here.
  - firesteelrain 3y ago
    Please don’t be a dick here
- btasker 3y ago
  > I don’t see how that addresses the LTS issues since k8s is updating so often that you have to keep pace with the updates (that you now need to do). They're not saying they moved to running k8s on a VPS - they're saying they moved to using a VPS instead of k8s (i.e. have escaped the k8s upgrade cycle).

jl6 3y ago

That’s great if you have control over all the moving parts, but a lot of real-world (i.e. not exclusively software-based) orgs have interfaces to components and external entities that aren’t so amenable to change. Maybe you can upgrade your cluster without anybody noticing. Maybe you’re a wizard and upgrades never go wrong for you. More likely, you will be constrained by a patchwork quilt of contracts with customers and suppliers, or by regulatory and statutory requirements, or technology risk policies, and to get approval you’ll need to schedule end-to-end testing with a bunch of stakeholders whose incentives aren’t necessarily aligned to yours. That all adds up to $$$, and that’s why there’s a demand for stability and LTS editions.

Alcatros552 3y ago
@ji6 I have to tell you, we are using Kubernetes since the very early versions available. If you automate everything you will never have an issue with K8, you can deploy all the required dependencies in one go. You can run the tests, if done correctly this literally requires 10 minutes. This argument to have an LTS Version for me is like air. It's the kind of Nuclear Reactor argument. We did have the money to buy Uranium to burn but now we can't afford to dispose it normally. Kubernetes is for flexibility not for big companies which want to use their software development processes of the 90's...
- JamesonNetworks 3y ago
  Yeah? Through the RBAC changes? Because thats where I gave up
- dvfjsdhgfv 3y ago
  I'm sorry, this is simply not true. K8s relatively often introduces breaking changes. They are announced well in advance, which is very nice, but the solution will not take 10 minutes. Take Pod Security Policies and migration to Kyverno of OPA Gatekeeper. Even if you have a relatively simple cluster and choose to stick to defaults rather than write your own rules, it will still take time to understand the new system, to choose the policies you need, to update and test it. In complex cluster this takes weeks and in some cases even months, especially if you are legally obliged to follow change management.

Waterluvian 3y ago

Sometimes I get this feeling that a lot of developers kind of want to be in their code all the time, tending to it. But it’s just not a good use of my time. I want it to work so I can ignore it and move on to the next best use of my time. I trial upgraded a years old Django project today to 5.0 and it took almost zero work. I hadn’t touched versions (other than patch)in over a year. That’s the way I want it. Admittedly this was less about an LTS and more about sensible design with upgrading in mind.

selcuka 3y ago
> I trial upgraded a years old Django project today to 5.0 and it took almost zero work. Django is great at backwards compatibility, but to be honest they haven't added many revolutionary features to Django for years, except a half-baked implementation of async support.

maximinus_thrax 3y ago

To paraphrase someone else's reaction, I'm violently shaking my head in disagreement. What you're saying only works when you have 100% full control of everything (including the customer data). As someone who spent years in the enterprise space, what you're describing is akin to 'Ivory Tower Architecture'. LTS is a commitment. That is all. If someone is uncomfortable with such a commitment, then that's fine, let me free market sort it out. But what LTS does is it tells everyone (including paying customers) that the formats/schemas/APIs/etc.. in that version will be supported for a very long time and if I adopt it, I won't have to think about it or budget too much for its maintenance for a period of time measured in months/years. I would go the extra mile here and say that offline format should be supported FOREVER. None of that LTS bs for offline data, ever. LTS means that you're accountable for some of the costs in your partnership with the customers. If you move fast and break things, they will have to work extra just to keep up. If you move fast but mindful with back-compat, you will work extra but your customer will be happier. That is all. Re-reading your comment gives me chills and reinforces my belief that I will never pay money to Google (they have a similar gung-ho attitude against 'legacy garbage') or have any parts of my business depend on stuff which reserves the liberty of breaking shit early and often.

JackSlateur 3y ago
I worked in a public cloud compagny We had very few control over the customers stuff We did not had to : avoid breaking changes (or impose them) and transparent upgrades were the key

hot_gril 3y ago

Even if you could get everyone to tend the garden, it encourages unnecessary breaking changes or fragile design. LTS provides some much-needed backpressure. If a new feature truly requires breaking things, people who really want it will put in the effort.

p_l 3y ago
That's why Kubernetes provides versioned APIs with proper lifecycles and levels of stability. A stable API is not going to change randomly on you, and of it does, it's going to come with deprecation policy so you have time to do it. Personally I did have few "painful" k8s upgrades, but every time it happened it was due to long-announced large scale change... That another team didn't want to upgrade over despite delaying so long the last version compatible became EOL
- hot_gril 3y ago
  Right, but the support is not very long. You still get basically forced to update at least every 14 months according to the article.

jvans 3y ago

I've spent so much time chasing down performance problems and bugs where the problem was due to an outdated dependency. Simply spending a few hours a month upgrading dependencies is a big win to avoid those situations

cramjabsyn 3y ago

Its not a garden. Gardens are horizontal with no dependencies between plants. Infrastructure is a high rise building. Long term planning and careful maintenance is needed. And it shouldn't be necessary to replace the foundation every year or two.

cratermoon 3y ago
> Gardens are horizontal with no dependencies between plants. Two words: companion plantings Also, there's a world of incompatible plants. Some plants actively harm the growth of others, and some simply can't grow in the same place because of different soil and nutrient requirements.
oceanplexian 3y ago
Yeah imagine if someone built a bridge and was like, yeah, we need to do regular updates or else it will fail and come crashing down. I feel like software engineers who preach that everything must be connected to the internet and update from the mothership regularly are fundamentally disconnected from reality. If your design is robust to begin with you should be able to depend on it without constantly fiddling with everything.
- terom 3y ago
  Bridges need regular maintenance or they will fail and come crashing down.
  - mcmcmc 3y ago
    Regular maintenance for bridges means tensioning cables, tightening loose bolts, repainting exposed parts. Not redoing the foundations every month.
    - pas 3y ago
      k8s doesn't require redoing foundations every month this LTS hysteria is completely made-up it benefits from updates every year, also who uses naked k8s (the hard way?), folks use a distribution with an updater

zzyzxd 3y ago

Yup. At my last company, Kubernetes was the only place where software was not 2+ years behind upstream because of this and I appreciate it a lot. And its API deprecation policy [1] made upgrade not so painful (if upgrades frequently break your stuff, check if your infra people really understand this policy, and your software vendors are compliant). 1. https://kubernetes.io/docs/reference/using-api/deprecation-policy https://kubernetes.io/docs/reference/using-api/deprecation-p...

cpeterso 3y ago

At the far end of the update frequency spectrum from LTS is Google's "live at head" philosophy. Google's Abseil C++ library, for example, recommends that consumers update to Abseil's latest commit from the master branch as often as possible. Abseil have no tagged releases besides master and an LTS branch (updated every 6-9 months). https://abseil.io/about/philosophy#we-recommend-that-you-choose-to-live-at-head https://abseil.io/about/philosophy#we-recommend-that-you-cho...

waynesonfire 3y ago
The tag is the commit. So what. Same thing.

quickthrower2 3y ago

I like the idea, and with something like say Chrome, this is excellent. However each upgrade of k8s needs planning. Need to check through the list to see what APIs are broken, and figure out if anything needs changing, preparing for it, including stuff the cloud provider has chucked in the mix. Probably to the point I feel like a cluster of clusters would be safer, so you can slowly roll it out.

p_l 3y ago
Use multiple clusters for checking out upgrade path at least. Plan regular upgrade so it's moved in small chunks and so that teams are aware of where they stand with APIs etc.
- fungalcretin007 3y ago
  This is garbage path compared to controlled quantity. In 2008 no one would have countenanced this for good reason. What is the reason now?

cultofmetatron 3y ago

> Do your upgrades and releases frequently this sounds great in theory. after 5years of running a startup, you learn to pick your battles. that db library you upgraded? works great except for this one edge case where they changed the interface in an obscure portion that affects 5% of your users. it got past QA but thaat 5% of users get REAL VOCAL about it. now its in production and you're planning an emrgency revert of that dependncy after you verify that you aren't using any of the new features of the library. sounds doable? now multiply that by the number of dependencies your app has. I agree that it good to periodically update deps and allocate time for keeping your system up to date but Its easy to let tracking all your deps suck up all the time your startup would be better off spending on adding features that bring new users in and add to your bottom line.

waynesonfire 3y ago
Absolutely, it's clear the OP hasn't maintained anything at scale. Last thing I want to do is upgrade a rock solid library just because.
- FridgeSeal 3y ago
  I suggested upgrading like it’s the wild-west where? When your rock solid dependency releases a new major version, when you’re ready you update it, and fix all the things you need to in a single PR, you take that to prod when you’re ready, and roll it back if it’s unhappy, or your favourite variation of out-and-back. What I’m advocating against, is picking a version, and then sitting on it for so long that it inevitably goes out of support, and then find yourself having a really bad, un-fun time-probably at some awful time of day- and discovering the work you need to do to get out of the current system is so much more than you initially realised, because you’re so far behind. Breaking changes happen anyway at some point, you may as well stay abreast of them.

jacurtis 3y ago

I both agree and disagree with you. In theory, I agree. Tend your garden, do small upgrades often instead of major upgrades less often. In a homelab this is easy to implement. In a small organization it is too. But in the facetious "real world", things are a lot more complicated. I work as an SRE Manager and my team is basically ALWAYS upgrading Kubernetes. New releases drop about as fast as we can upgrade the last ones. When you work on a large cluster, doing an upgrade isn't a simple process. It requires a ton of testing, several steps, and being very slow and methodical to make sure it is all done properly. Where I currently work, we have 2 week sprints and infrastructure changes must align with sprint cycles. So to promote an upgrade at the fastest possible schedule it requires: - Week 0: Upgrade Dev environment - Week 2: Upgrade QA Environment - Week 4: Upgrade Sandbox Environment - Week 6: Upgrade Prod Environment That is the fastest possible schedule. That assumes we do a cluster upgrade every sprint, which is 2 weeks. It also ignores other clusters for other business units. We have 4 primary product lines, so multiply all that work times 4. Plus we have supporting products (like self-hosted gitlab, codescene, and custom tools running in k8s clusters). I say fastest possible schedule because, we can't keep up with this schedule, but even if we could it is the fastest we could go and still maintain our deployment and infrastructure promotion policies. With new releases every 3-4 months (12-16 weeks), we are essentially in a constant state of always upgrading kubernetes. Right now my team is 2 versions behind. Skipping versions doesn't make sense because you can't guarantee a safe upgrade when skipping versions. This is why LTS releases are nice. When you run systems at large scale, it is impractical to upgrade that often. I'd prefer to limit upgrades to no more than twice a year and personally I find annual upgrade cycles to be the best balance between "tending the garden" and "not drowning in upgrade work". LTS releases are usually tests to skip upgrades so that companies can go from LTS release to LTS release, without the need to worry about upgrading every minor version in sync. Remember, upgrading K8s clusters isn't what my bosses want to hear my team spends our time. They want to know that observability is improving, devs are getting infrastructure support, we are building out new systems, deploying hardware for the product team, running our resiliency tests, etc.. Sure upgrading is part of the job, but i can't be ALWAYS upgrading. I have a lot of other responsibilities.

solatic 3y ago
I don't know what your internal architecture looks like, but on the face of it, arguably you should be running Dev and QA loads on the same cluster. Either you have an organization where an SRE team is responsible for running clusters for teams, in which case why not run Dev and QA on the same cluster, or you are responsible for last-line-of-support for teams responsible for running their own clusters, in which case you say, here's a new version of Kubernetes (e.g. a Terraform module) and in this sprint you are responsible for deploying it through dev, QA, pre-prod, and production clusters. Especially if you have separate Kubernetes clusters per product line (do you really need separate dev Kubernetes clusters per product line?).

stouset 3y ago

100% agreed. Make upgrades easy, automate the tedious parts, and do them as often as possible. If you do upgrades once per several years, yes, it is going to be excruciating.

deleted 3y ago

[deleted]

npacenop 3y ago

This, and very much this. At the end of course the decision will be based on business and market analysis, as even the cloud native foundation abides more or less to the same rules as everyone else does... Introducing LTS for kubernetes however will be a huge step towards pushing it down the enterprise products alley, where software is selected more for the amount of people available in the market, capable to work with it / operate it, and the running costs generated, rather than satisfying an actual need of the business. Chances are that if your org needs LTS for kubernetes, then kubernetes is not the right solution to your problems. Which is probably the case anyway... but that's a whole different story.