3 ms·

Every single time I’ve seen companies use LTS versions of operating systems, upgrading is more painful and consumes double or more the overall time than upgradi

Every single time I’ve seen companies use LTS versions of operating systems, upgrading is more painful and consumes double or more the overall time than upgrading regularly would have. Plus it inevitably happens that teams will need something only available on newer platforms. With LTS OSes, this becomes an enormous PITA. Someone needs a newer version of some thing, that needs newer versions of system libraries, that turns out to use a new kernel feature, and now you’re in the hell of trying to figure all that out. The only solution I have ever found is: upgrade everything constantly. Yes, it consumes time. But that time will get consumed anyway at some point down the line, and you’re better off doing it now. It’ll take less time and you’ll be able to deliver new feature your teams are asking for. The only time I’d ever consider an LTS anything at this point is for early-stage startups that just cannot pay that cost.

worthless-trash 2y ago
From my limited experience I don't see many containerized apps end up using newer syscalls. We're only recently getting io-uring (with its array of security defeating side affects) in containers. Business software moves much slower than consumer software, i expect that LTS ends up sucking less when they can control the userspace around their application.
- tapoxi 2y ago
  eBPF potentially? There's a lot of security tooling out there (Wiz agent etc) that would assume a newer kernel on the node.
  - worthless-trash 2y ago
    Yeah, eBPF is another one of those speculative execution improvement tools. I haven't had a chance to look at Wiz, I would assume they would make their own kernel module that inspects things based on a very slim ABI requirement. Their customer base (according to their page) is fortune 500 companies, most of those only move off unsupported releases when they are forced to, it is unlikely they use new syscalls.
- inahga 2y ago
  OTOH off the shelf software distributed for Kubernetes IME does tend to pick up on new control plane features fairly regularly. I’ve historically tracked n-1 or n-2 of k8s, and it is not infrequent that we can’t take a helm chart update until we update k8s.
fujinghg 2y ago
We jumped I think 8 year old CentOS versions without too much issues. It depends how badly you fucked up in the first place really.
- bityard 2y ago
  This is a good point. The distance between versions can matter, but it usually matters a lot less than the eng/ops processes and culture of the specific organization. For my part, every time I have done a major OS distribution upgrade, 75% of my time was spent fixing all of the things that were deployed into a highly fragile state to begin with, 25% direct upgrade-related issues (dependencies, config, behavior changes), and 5% actually performing the upgrade.
dralley 2y ago
You CAN upgrade RHEL (or CentOS Stream, or Alma, or Ubuntu) every 3 years if you want. Some people do it. Nobody forces you to live out the entire lifecycle on one release. Some people appreciate having the option to pick and choose when the transition happens though.
- stouset 2y ago
  Funny, because we’ve been on RHEL. After one round of calcification, upgrading our datacenters for each future RHEL release took years of concerted efforts. We actually lost ground every year.
  - hedora 2y ago
    I've supported others that use RHEL. If you like software archeology, it's the OS for you: > This bug was fixed in 2002, then regressed in 2005 and fixed again in 2007 in mainline, but in RHEL, they botched the '02 and '05 back port, so '02 breaks it, but '05 doesn't. Now, let me tell you about secondary effects of patches in unrelated subsystems. We have three interesting "type A" examples introduced in ... > ... and that's why it's completely acceptable to run half the fortune 500 off of Linux 2.6.19-RHEL.... in 2025. > What, you're running 2.6.20? What a hoot!!! We like to kid. You're not serious? You are? You work at the local nuclear plant? > backs out of conference room slowly, then bolts down the hallway screaming "run for your lives!" once they have clear view of the elevator
    - dralley 2y ago
      > What, you're running 2.6.20? What a hoot!!! We like to kid. You're not serious? You are? You work at the local nuclear plant? The local nuclear power plant is hopefully not connected to the internet. It shouldn't be surprising that such systems run old software.
      - hedora 2y ago
        Not all mission critical bugs are security holes.
synack 2y ago
LTS makes sense for services with a finite lifecycle that are meant to be replaced at EOL, not upgraded.
- stouset 2y ago
  I can agree with this.
- SahAssar 2y ago
  How often do you know that that far ahead? If someone asked me to guarantee that a service will be replaced in 4 years that'd be hard. Ask me about 12 years and it's basically impossible.
  - deleted 2y ago
    [deleted]
  - notatoad 2y ago
    if it's a project for your employer that you have staff in charge of maintaining, then it's probably a bad choice. but for projects delivered by contractors who provide a fixed service and maintenance term, it's reasonable to provide a fixed end date for the project as a whole, and that's probably less than 12 years.
mystifyingpoi 2y ago
I'm a big fan of the opposite, nuclear approach - never upgrade. Always build new system/cluster/database/whatever on the side, test it as long as you like, then flip the traffic over. It is both much easier and hella lot harder (at different times in the lifecycle), but in the end it pays off. It's not always possible, but when it is, I'm on it.
- stouset 2y ago
  This is the approach I prefer for upgrading when able. But you should still be exercising this approach regularly when possible. You’re still upgrading to new versions, you’re just not upgrading live systems. At least with k8s, this approach is often sadly somewhat tough.
lexicality 2y ago
On the other hand, every time I've been in an "update everything constantly" situation there have been extremely painful outages every 6 months because people put dramatically too much faith in their unit tests to make updating "quick and painless". Personally I go for having a recurring calendar entry every few months that blocks off a few days to very carefully read all the patch notes before updating
2OEH8eoCRo0 2y ago
Those "LTS" distros will often backport support for new HW.
- remram 2y ago
  This is usually done by just shipping new kernels. E.g. Ubuntu's "hardware enablement" (hwe) kernels are just new kernel releases, not old kernels with backported drivers.
OptionOfT 2y ago
I'm also in favor of constant updates. When you update relatively soon after the update was released, the internet's memory is very fresh. Errors that you get related to the versions you're updating from/to.
- koolba 2y ago
  Not too soon though. There’s a sweet spot for this where you’re not the guinea pig, but it’s recent enough that the target you’re upgrading to has not yet changed. Once you’re more than one major version back, you risk hitting all kinds of issues.
packetlost 2y ago
I've found LTS releases of Ubuntu to be less prone to breakages when going LTS=>LTS vs LTS=>non-LTS=>LTS, but only slightly. That being said, the one time I've had critical bootloader broken to the point of looking at GRUB code was a simple `apt dist-upgrade` on a LTS release of Ubuntu 22.04, so there's that. These days I think my preference is for immutable core + userspace package management or going all in on something like NixOS. I'd really love to see a distro that takes this approach to its logical conclusion using OverlayFS or something (but strictly not containers)
eduction 2y ago
Honestly this is why I'm intrigued by Fedora Server, which at first I took to be some kind of sick joke, but now I see from using regular Fedora long enough that it's good to be in regular practice of handling upgrades. Like you said it keeps those muscles from atrophying.
solatic 2y ago
> The only time I’d ever consider an LTS anything at this point is for early-stage startups that just cannot pay that cost. You're touching on the deeper truth here. Ultimately you are forced to touch production all the time for at least security updates. The question is whether you have people on staff who know the system, at all levels of the stack. If you do, then there is only pain by taking the LTS route, because you will have a fundamental divide between the people who need the latest features and the people who know the specific LTS release. But if you have people who treat the lower level like a black box, and nobody on staff knows how it works, then LTS helps you derisk this black box that nobody knows by restricting the updates to security updates only and nothing that is supposed to break existing behavior. Then, when upgrading LTS releases, you treat the LTS upgrade as a project in and of itself (full QA, the whole nine yards) instead of some day-to-day maintenance running on autopilot. Most SaaS stacks run by startups / grow-ups should have people familiar with every level of the stack that they're building on top of. If you don't, build on top of a managed service instead, and be familiar with the managed service's APIs instead.
wink 2y ago
counterpoint: I've been running my own mail server for 20 years and until a while ago it was: Install Ubuntu LTS, install the needed packages, don't touch the machine except for security upgrades until LTS runs out and I would (after skipping one LTS) use the next one for 4-5 years. Yes, not a company, but I can imagine many more scenarios like this.

worthless-trash 2y ago

From my limited experience I don't see many containerized apps end up using newer syscalls. We're only recently getting io-uring (with its array of security defeating side affects) in containers. Business software moves much slower than consumer software, i expect that LTS ends up sucking less when they can control the userspace around their application.

tapoxi 2y ago
eBPF potentially? There's a lot of security tooling out there (Wiz agent etc) that would assume a newer kernel on the node.
- worthless-trash 2y ago
  Yeah, eBPF is another one of those speculative execution improvement tools. I haven't had a chance to look at Wiz, I would assume they would make their own kernel module that inspects things based on a very slim ABI requirement. Their customer base (according to their page) is fortune 500 companies, most of those only move off unsupported releases when they are forced to, it is unlikely they use new syscalls.
inahga 2y ago
OTOH off the shelf software distributed for Kubernetes IME does tend to pick up on new control plane features fairly regularly. I’ve historically tracked n-1 or n-2 of k8s, and it is not infrequent that we can’t take a helm chart update until we update k8s.

fujinghg 2y ago

We jumped I think 8 year old CentOS versions without too much issues. It depends how badly you fucked up in the first place really.

bityard 2y ago
This is a good point. The distance between versions can matter, but it usually matters a lot less than the eng/ops processes and culture of the specific organization. For my part, every time I have done a major OS distribution upgrade, 75% of my time was spent fixing all of the things that were deployed into a highly fragile state to begin with, 25% direct upgrade-related issues (dependencies, config, behavior changes), and 5% actually performing the upgrade.

dralley 2y ago

You CAN upgrade RHEL (or CentOS Stream, or Alma, or Ubuntu) every 3 years if you want. Some people do it. Nobody forces you to live out the entire lifecycle on one release. Some people appreciate having the option to pick and choose when the transition happens though.

stouset 2y ago
Funny, because we’ve been on RHEL. After one round of calcification, upgrading our datacenters for each future RHEL release took years of concerted efforts. We actually lost ground every year.
- hedora 2y ago
  I've supported others that use RHEL. If you like software archeology, it's the OS for you: > This bug was fixed in 2002, then regressed in 2005 and fixed again in 2007 in mainline, but in RHEL, they botched the '02 and '05 back port, so '02 breaks it, but '05 doesn't. Now, let me tell you about secondary effects of patches in unrelated subsystems. We have three interesting "type A" examples introduced in ... > ... and that's why it's completely acceptable to run half the fortune 500 off of Linux 2.6.19-RHEL.... in 2025. > What, you're running 2.6.20? What a hoot!!! We like to kid. You're not serious? You are? You work at the local nuclear plant? > backs out of conference room slowly, then bolts down the hallway screaming "run for your lives!" once they have clear view of the elevator
  - dralley 2y ago
    > What, you're running 2.6.20? What a hoot!!! We like to kid. You're not serious? You are? You work at the local nuclear plant? The local nuclear power plant is hopefully not connected to the internet. It shouldn't be surprising that such systems run old software.
    - hedora 2y ago
      Not all mission critical bugs are security holes.

synack 2y ago

LTS makes sense for services with a finite lifecycle that are meant to be replaced at EOL, not upgraded.

stouset 2y ago
I can agree with this.
SahAssar 2y ago
How often do you know that that far ahead? If someone asked me to guarantee that a service will be replaced in 4 years that'd be hard. Ask me about 12 years and it's basically impossible.
- deleted 2y ago
  [deleted]
- notatoad 2y ago
  if it's a project for your employer that you have staff in charge of maintaining, then it's probably a bad choice. but for projects delivered by contractors who provide a fixed service and maintenance term, it's reasonable to provide a fixed end date for the project as a whole, and that's probably less than 12 years.

mystifyingpoi 2y ago

I'm a big fan of the opposite, nuclear approach - never upgrade. Always build new system/cluster/database/whatever on the side, test it as long as you like, then flip the traffic over. It is both much easier and hella lot harder (at different times in the lifecycle), but in the end it pays off. It's not always possible, but when it is, I'm on it.

stouset 2y ago
This is the approach I prefer for upgrading when able. But you should still be exercising this approach regularly when possible. You’re still upgrading to new versions, you’re just not upgrading live systems. At least with k8s, this approach is often sadly somewhat tough.

lexicality 2y ago

On the other hand, every time I've been in an "update everything constantly" situation there have been extremely painful outages every 6 months because people put dramatically too much faith in their unit tests to make updating "quick and painless". Personally I go for having a recurring calendar entry every few months that blocks off a few days to very carefully read all the patch notes before updating

2OEH8eoCRo0 2y ago

Those "LTS" distros will often backport support for new HW.

remram 2y ago
This is usually done by just shipping new kernels. E.g. Ubuntu's "hardware enablement" (hwe) kernels are just new kernel releases, not old kernels with backported drivers.

OptionOfT 2y ago

I'm also in favor of constant updates. When you update relatively soon after the update was released, the internet's memory is very fresh. Errors that you get related to the versions you're updating from/to.

koolba 2y ago
Not too soon though. There’s a sweet spot for this where you’re not the guinea pig, but it’s recent enough that the target you’re upgrading to has not yet changed. Once you’re more than one major version back, you risk hitting all kinds of issues.

packetlost 2y ago

I've found LTS releases of Ubuntu to be less prone to breakages when going LTS=>LTS vs LTS=>non-LTS=>LTS, but only slightly. That being said, the one time I've had critical bootloader broken to the point of looking at GRUB code was a simple `apt dist-upgrade` on a LTS release of Ubuntu 22.04, so there's that. These days I think my preference is for immutable core + userspace package management or going all in on something like NixOS. I'd really love to see a distro that takes this approach to its logical conclusion using OverlayFS or something (but strictly not containers)

eduction 2y ago

Honestly this is why I'm intrigued by Fedora Server, which at first I took to be some kind of sick joke, but now I see from using regular Fedora long enough that it's good to be in regular practice of handling upgrades. Like you said it keeps those muscles from atrophying.

solatic 2y ago

> The only time I’d ever consider an LTS anything at this point is for early-stage startups that just cannot pay that cost. You're touching on the deeper truth here. Ultimately you are forced to touch production all the time for at least security updates. The question is whether you have people on staff who know the system, at all levels of the stack. If you do, then there is only pain by taking the LTS route, because you will have a fundamental divide between the people who need the latest features and the people who know the specific LTS release. But if you have people who treat the lower level like a black box, and nobody on staff knows how it works, then LTS helps you derisk this black box that nobody knows by restricting the updates to security updates only and nothing that is supposed to break existing behavior. Then, when upgrading LTS releases, you treat the LTS upgrade as a project in and of itself (full QA, the whole nine yards) instead of some day-to-day maintenance running on autopilot. Most SaaS stacks run by startups / grow-ups should have people familiar with every level of the stack that they're building on top of. If you don't, build on top of a managed service instead, and be familiar with the managed service's APIs instead.

wink 2y ago

counterpoint: I've been running my own mail server for 20 years and until a while ago it was: Install Ubuntu LTS, install the needed packages, don't touch the machine except for security upgrades until LTS runs out and I would (after skipping one LTS) use the next one for 4-5 years. Yes, not a company, but I can imagine many more scenarios like this.