4 ms·

This, like a recent LTS discussion I saw for a different tool, ignores one tiny little detail that makes the whole discussion kind of moot. LTS doesn't mean it

by sofixa 3y ago

This, like a recent LTS discussion I saw for a different tool, ignores one tiny little detail that makes the whole discussion kind of moot. LTS doesn't mean it's immune to bugs or security vulnerabilities. It just means that the major release is updated and supported longer - but you still need to be able to apply patches and security fixes to that major release. Yes, it's easier to go from 1.20.1 to 1.20.5 than to 1.21, because there's less chance of breakage and less things that will change, but the process is pretty much the same - check for breaking changes, read changelogs, apply everything. The risk is less, might be slightly faster, but fundamentally, it's the same process. If the process is too heavy and takes you too long, having it be slightly faster won't be a gamechanger. So LTS brings slight advantages to the operator, while adding potentially significant complexity to the developer (generally backporting fixes into years old versions isn't fun). The specific proposed LTS falvour is also hardcore, without an upgrade path to the next LTS. The exact type of org that needs an LTS will be extremely reluctant to having to redo everything, 2 years later, with potentially drastic breaking changes making that change very hard.

x86x87 3y ago
That's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. If you're at a point where a patch for LTS looks like an upgrade to the new version you've screwed up LTS. Also, getting to the point of having an LTS and actually providing the support is expensive. You need experts that can backport security fixes and know the product inside out.
- sofixa 3y ago
  > That's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. How do you do that on something as complex and with as many moving parts as Kubernetes? And how do you as an operator update that many things without checking there's no breaking changes in the patch?
  - x86x87 3y ago
    bingo! how do you do it? and do you want that kind of complexity to begin with?
    - sofixa 3y ago
      Don't ask me, I'm firmly in the HashiCorp Nomad camp: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at... (Note: quite old, some things are no longer true most notably around Nomad downsides)
      - rigrassm 3y ago
        I'm with you, Nomad is highly underrated!
  - Rantenki 3y ago
    We upgrade our distros pretty much fearlessly, all the time. While I have had breakage from Kernel upgrades, they've been very rare (and generally related to third party closed drivers). Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place.
    - eddythompson80 3y ago
      > Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place. eh, the kernel is an incredibly mature project with 1 machine scope. The kernel also has decades of operating systems research and literature to build on. Kubernetes in comparison is new, distributed and exploring uncharted territory in terms of feature set and implementation. Sometimes bad decisions are made, and it's fair to not want to live with them forever. The kernel project looks very different today than it did in 1999. There is a happy medium though, especially that Kubernetes is kinda far from it.
      - x86x87 3y ago
- freedomben 3y ago
  I don't see anywhere that GP said an LTS patch would take effort. They said the upgrade path to the next LTS would. If you are talking about upgrade from LTS to LTS, can you give an example project where that is effortless? And if so, how do they manage to innovate and modernize without ever breaking backwards compatibility?
  - x86x87 3y ago
    Here: "it's easier to go from 1.20.1 to 1.20.5 than to 1.21, because there's less chance of breakage and less things that will change, but the process is pretty much the same" LTS to LTS is another story. But the point is that L=LongTerm so in theory you're only going to do this exercise twice in a decade. > manage to innovate and modernize without ever breaking backwards yeah. fuck backwards compatibility. that is for suckers. how about stopping the madness for a second and thinking about what you are building when you build it?
    - pixl97 3y ago
      > in theory you're only going to do this exercise twice in a decade. So I've seen things like this in corporations many times and it typically works like this... Well trained team sets up environment. Over time team members leave and only less senior members remain. They are capable of patching the system and keeping it running. Eventually the number of staff even capable of patching the system diminishes. System reaches end of life and vendor demands upgrading. System falls out of security compliance and everything around it is an organizational exception in one way or another. Eventually at massive cost from outside contractors the system gets upgraded and the cycle begins all over again. Not being able to upgrade these systems is about the lack of and loss of capable internal staff.
      - x86x87 3y ago
        What is the cost of keeping it "up to date" vs doing this exercise once every say 7 years? Are most software systems even around for 7 years?
Karellen 3y ago
> but the process is pretty much the same - check for breaking changes, Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. (...of course, there's no guarantee that you're not relying on buggy (or, at least, accidental) behaviour. People relying on `memcpy(3)` working as expected when the ranges overlap, simply because it happened to do so with historic versions of the `libc` implementation they most commonly happened to test with, is one example. But see also obxkcd https://xkcd.com/1172/ https://xkcd.com/1172/ and https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ )
- sofixa 3y ago
  > Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. Or a security vulnerability has forced a breaking change. Or any other issue, which is why you have to check.
  - Karellen 3y ago
    > Or a security vulnerability has forced a breaking change. Theoretically, I suppose? Do you have a historic example in mind? I've been running Debian "stable" in its various incarnations on servers for over a decade, and I can't remember any time any service on any installation I've run had such an issue. But my memory is pretty bad, so I might have missed one. (Or even a dozen!) But I have `unattented-upgrades` installed on all my live servers right now, and don't lose a wink of sleep over it.
    - sofixa 3y ago
      Yes, I have an example in mind - https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-unattended-samba-upgrade-broke-our-setup https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-una... Yes, it's Ubuntu, but doesn't matter - sometimes security fixes require a breaking change and there's nothing that can be done to avoid it.
    - natbennett 3y ago
      This happens all the time on systems that are running hundreds of thousands of apps across hundreds of customers. The worst one I know: for a while basically all Cloud Foundry installations were stuck behind a patch release because the routing component upgraded their Go version and that Go version included an allegedly non-breaking-change that caused it to reject requests with certain kinds of malformed headers. The Spring example app has a header with the specific problem impacted. And the vast majority of Cloud Foundry apps are Spring apps, many of which got started by copying the Spring example app. So upgrading CF past this patch release required a code change to the apps running on the platform. Which the people running Cloud Foundry generally can’t get — there’s usually a team of like 12 people running them and then 1000s of app devs.

by sofixa 3y ago

x86x87 3y ago

That's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. If you're at a point where a patch for LTS looks like an upgrade to the new version you've screwed up LTS. Also, getting to the point of having an LTS and actually providing the support is expensive. You need experts that can backport security fixes and know the product inside out.

sofixa 3y ago
> That's not how LTS is supposed to work. You should be able to uograde effortlessly with minimum risk. How do you do that on something as complex and with as many moving parts as Kubernetes? And how do you as an operator update that many things without checking there's no breaking changes in the patch?
- x86x87 3y ago
  bingo! how do you do it? and do you want that kind of complexity to begin with?
  - sofixa 3y ago
    Don't ask me, I'm firmly in the HashiCorp Nomad camp: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at... (Note: quite old, some things are no longer true most notably around Nomad downsides)
    - rigrassm 3y ago
      I'm with you, Nomad is highly underrated!
- Rantenki 3y ago
  We upgrade our distros pretty much fearlessly, all the time. While I have had breakage from Kernel upgrades, they've been very rare (and generally related to third party closed drivers). Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place.
  - eddythompson80 3y ago
    > Kubernetes is _not_ more complicated than the Linux kernel, but it is much more dangerous to upgrade in place. eh, the kernel is an incredibly mature project with 1 machine scope. The kernel also has decades of operating systems research and literature to build on. Kubernetes in comparison is new, distributed and exploring uncharted territory in terms of feature set and implementation. Sometimes bad decisions are made, and it's fair to not want to live with them forever. The kernel project looks very different today than it did in 1999. There is a happy medium though, especially that Kubernetes is kinda far from it.
    - x86x87 3y ago
freedomben 3y ago
I don't see anywhere that GP said an LTS patch would take effort. They said the upgrade path to the next LTS would. If you are talking about upgrade from LTS to LTS, can you give an example project where that is effortless? And if so, how do they manage to innovate and modernize without ever breaking backwards compatibility?
- x86x87 3y ago
  Here: "it's easier to go from 1.20.1 to 1.20.5 than to 1.21, because there's less chance of breakage and less things that will change, but the process is pretty much the same" LTS to LTS is another story. But the point is that L=LongTerm so in theory you're only going to do this exercise twice in a decade. > manage to innovate and modernize without ever breaking backwards yeah. fuck backwards compatibility. that is for suckers. how about stopping the madness for a second and thinking about what you are building when you build it?
  - pixl97 3y ago
    > in theory you're only going to do this exercise twice in a decade. So I've seen things like this in corporations many times and it typically works like this... Well trained team sets up environment. Over time team members leave and only less senior members remain. They are capable of patching the system and keeping it running. Eventually the number of staff even capable of patching the system diminishes. System reaches end of life and vendor demands upgrading. System falls out of security compliance and everything around it is an organizational exception in one way or another. Eventually at massive cost from outside contractors the system gets upgraded and the cycle begins all over again. Not being able to upgrade these systems is about the lack of and loss of capable internal staff.
    - x86x87 3y ago
      What is the cost of keeping it "up to date" vs doing this exercise once every say 7 years? Are most software systems even around for 7 years?

Karellen 3y ago

> but the process is pretty much the same - check for breaking changes, Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. (...of course, there's no guarantee that you're not relying on buggy (or, at least, accidental) behaviour. People relying on `memcpy(3)` working as expected when the ranges overlap, simply because it happened to do so with historic versions of the `libc` implementation they most commonly happened to test with, is one example. But see also obxkcd https://xkcd.com/1172/ https://xkcd.com/1172/ and https://www.hyrumslaw.com/ https://www.hyrumslaw.com/ )

sofixa 3y ago
> Unless you're relying on buggy behaviour, there should be no breaking changes in an LTS update. Or a security vulnerability has forced a breaking change. Or any other issue, which is why you have to check.
- Karellen 3y ago
  > Or a security vulnerability has forced a breaking change. Theoretically, I suppose? Do you have a historic example in mind? I've been running Debian "stable" in its various incarnations on servers for over a decade, and I can't remember any time any service on any installation I've run had such an issue. But my memory is pretty bad, so I might have missed one. (Or even a dozen!) But I have `unattented-upgrades` installed on all my live servers right now, and don't lose a wink of sleep over it.
  - sofixa 3y ago
    Yes, I have an example in mind - https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-unattended-samba-upgrade-broke-our-setup https://askubuntu.com/questions/1376118/ubuntu-20-04-lts-una... Yes, it's Ubuntu, but doesn't matter - sometimes security fixes require a breaking change and there's nothing that can be done to avoid it.
  - natbennett 3y ago
    This happens all the time on systems that are running hundreds of thousands of apps across hundreds of customers. The worst one I know: for a while basically all Cloud Foundry installations were stuck behind a patch release because the routing component upgraded their Go version and that Go version included an allegedly non-breaking-change that caused it to reject requests with certain kinds of malformed headers. The Spring example app has a header with the specific problem impacted. And the vast majority of Cloud Foundry apps are Spring apps, many of which got started by copying the Spring example app. So upgrading CF past this patch release required a code change to the apps running on the platform. Which the people running Cloud Foundry generally can’t get — there’s usually a team of like 12 people running them and then 1000s of app devs.