15 ms·

Linux gives up on 6-year LTS kernels, says they’re too much work

274 points by rascul 3y ago

SuperNinKenDo 3y ago
6 years seems like an incredibly long time to me. It's a bit of a shame, but looks like industry really didn't come out to support the idea. 2 years on the other hand seems incredibly short? Am I wrong?
- LinuxBender 3y ago
  That's 2 years of the upstream LTS kernel. I would expect that major Linux distributions such as Redhat RHEL and Canonical's Ubuntu would continue to do their extended patch cycles against one of the upstream snapshots as they have done in the past. I think 2 years for upstream LTS is probably fine if the vendor patching methodology remains true. This also assumes that the usage of smaller distributions such as Alpine are more commonly used in very agile environments such as K8's, Docker Swarm, etc... Perhaps that is a big assumption on my part.
- inferiorhuman 3y ago
  6 years may seem like a long time, but check out what the competition is doing. Oracle is supporting Solaris 10 for 20 years, 11.4 for 16 years (23 years if you lump it in with 11.0). HP-UX 11i versions seem to get around 15 years of support. It really depends on what you're doing, a lot of industries may not need such long-term support. 6 years seems like a happy medium to me, but then again I'm not the one supporting it. I expect the kernel devs would be singing a different tune if people were willing to pay for that extended support. https://upload.wikimedia.org/wikipedia/en/timeline/276jjn0uo65zngws02n9iabdtnhic97.png https://upload.wikimedia.org/wikipedia/en/timeline/276jjn0uo...
  - wkat4242 3y ago
    Are those really competition anymore? They're just legacy now IMO and their long term support requirements are a result of this, companies that haven't gotten rid of them by now aren't likely to do it any time soon. I hate seeing them go. I wasn't such a fan of Solaris but I was of HP-UX. But its days are over. It doesn't even run on x86 or x64 and HP has been paying Intel huge money to keep itanium on life support, which is running out now if it hasn't already. At least Solaris had an Intel port but it too is very rare now.
    - kube-system 3y ago
      Theres a lot of people who haven't got rid of old Linux systems these days too. RHEL 6 from 2010 is still eligible for extended support.
      - structural 3y ago
        There's still a decent population of RHEL 5 systems in the wild. Last year I was offered an engagement (turned down for a few reasons) to help a company upgrade several hundred systems from RHEL 5 to RHEL 6 and start planning for a future rollout of RHEL 7. Outside of tech focused companies, 10+ year old systems really are the norm.
        SoftTalker 3y ago
        > Outside of tech focused companies, 10+ year old systems really are the norm. It's because outside of tech companies, nobody cares about new features. They care about things continuing to work. Companies don't buy software for the fun of exploring new versions, especially frustratingly pointless cosmetic changes that keep hitting their training budgets. Many companies would be happy with RHEL5 or Windows XP today from a feature standpoint, if it weren't a security vulnerability.
        wkat4242 3y ago
        The problem about "things continuing to work" is really that many security fixes require updated architecture too. This is really why it's so hard to do LTS. It's not only about wanting new features.
        worthless-trash 3y ago
        What feature did they need in el7 that wasn't there in el9 ? What was their logic ?
        inferiorhuman 3y ago
        At megacorp (years ago) we were transitioning to CentOS 7 (from 6) and just starting to wind down our 32-bit windows stuff in AWS. I'm sure there are plenty of legacy Linux systems out there, but I wonder how many folks are actually paying for them. CentOS/RHEL 6 was already pretty long in the tooth, but being the contrarian I am, I was not looking forward to the impending systemd nonsense.
- kube-system 3y ago
  Depends on where the computer is at, I guess. On a desk, 6 years is a pretty long time. In an industrial setting, 6 years is not very long of a lifecycle.
  - dathinab 3y ago
    consider that it's 6 years after release of the kernel version so likely <5 years since release of the hardware in the US likely <4 years since release of hardware outside of the US likely <3 years since you bought the hardware and if you buy older phones having only a year or so of proper security updates is not that unlikely So for phones you would need more something like a 8 or 10 year LTS, or well, proper driver updates for proprietary hardware. In which case 2 years can be just fine because in general/normally only drivers are affected by kernel updates.
- Gigachad 3y ago
  What kind of breaking change would take longer than 2 years to deal with? The reality is that people wait out the entire 6 year period and then do the required months work at the end. If you make the support period 2 years they will just start working on it sooner.
  - Retric 3y ago
    Many perhaps most projects don’t last 6 years, thus punting can save people a great deal of time.
- dathinab 3y ago
  6 years is way to short for some use-cases, like phones sadly it's not rare that some phone hardware proprietary drivers are basically written once and then hardly maintained and in turn only work for a small number of Linux kernel versions. so for some hardware it might mean that when it gets released you only have noticeable less then 6 years of kernel support then between starting to build a phone and releasing it 2 years might easily pass so that means from release you can provide _at most_ 4 years of kernel security patches etc. but dates tend to not align that grate so maybe it's just 3 years but then you sell your phone for more then one year, right? in which case less then 2 years can be between the customer buying your product and you no longer providing kernel updates that is a huge issue I mean think about it if someone is a bit thigh on money and buy a slightly older phone they probably aren't aware that their phone stops getting security updates in a year or so (3 years software support since release but it's a 2 year old phone) at which point using it is a liability. EDIT: The answer is in my opinion not to have even longer LTS but to proper maintain drivers and in turn being able to do full kernel updates
  - xvilka 3y ago
    > sadly it's not rare that some phone hardware proprietary drivers are basically written once and then hardly maintained and in turn only work for a small number of Linux kernel versions. These manufacturers should be punished by the lack of LTS and need to upgrade precisely because of that lazyness and incompetence.
    - charcircuit 3y ago
      Why not blame the kernel developers for being lazy and incompetent for not offering a stable API for driver developers to use? You don't see Windows driver developers having their drivers broken by updates every few months.
      - mschuster91 3y ago
        > You don't see Windows driver developers having their drivers broken by updates every few months. At the cost of Windows kernel development being a huge PITA because effectively everything in the driver development kit becomes an ossified API that can't ever change, no matter if there are bugs or there are more efficient ways to get something done. The Linux kernel developers can do whatever they want to get the best (most performant, most energy saving, ...) system because they don't need to worry about breaking someone else's proprietary code. Device manufacturers can always do the right thing and provide well-written modules to upstream - but many don't because (rightfully) the Linux kernel team demands good code quality which is expensive AF. Just look at the state of most non-Pixel/Samsung code dumps, if you're dedicated enough you'll find tons of vulnerabilities and code smell.
- Ekaros 3y ago
  All comes to cycle. When do you enter that 6 year LTS? Is there new LTS every year or every other year? If you enter 2 years in or even 4 years in. How much have you support left? Do you jump LTS releases. So one you are on is ending and there is brand new available? Or do you go to one before and have possibly only 2 or 4 years left...
jauntywundrkind 3y ago
Google showed up & offered to pay a huge amount of money to extend LTS support 3x, iirc. At the time, Linux 4.14 was shipping I think. Personally it made me a bit sad, because it created real permission for Android to never upgrade kernels. I'd hoped eventually Android would start upgrading kernels, thought the short LTS would surely force them to become a respectable ecosystem that takes maintenance seriously. Making Super LTS was a dodge; suddenly it was ok that 4.14 straggles along until 2024. Also an interesting note, supposedly there's a "Civil Infrastructure Platform" that supposedly will have some support until 2029 for 4.14! The past never is gone, eh? Supposedly Project Treble is a whole new driver abstraction in I think mostly userland (maybe?) whose intent is to allow kernel upgrades without having to rewrite drivers (not sure if that is the primary/express goal or was just often mentioned). I'm not sure if Android has yet shipped a kernel upgrade to any phones though; anyone know of an model running an upgraded kernel?
- yjftsjthsd-h 3y ago
  > Personally it made me a bit sad, because it created real permission for Android to never upgrade kernels. I would argue that it was still better than not doing that, because the vendors weren't going to properly keep up with kernels either way; the choice wasn't "support 3.x for longer or move to 4.x", it was "support 3.x for longer or watch Android devices stay on 3.x without patches".
  - jauntywundrkind 3y ago
    As the old adage goes, "chips without software is just expensive sand." Yes what happened & happens is often monstrously unsupportable & terrible. For the past 6 years, Google rolling up with a dump truck full of bills has been justification to keep doing nothing, to keep letting kernel devices be bad. Your history isn't even good or right. Old releases at the time didn't get official longer support. Canonical just opted to maintain a basically Super LTS 3.16 until 2020, regardless of the Google dump-truck-of-money thing going on here. Old phones got nothing directly from this payoff. Google was just paying for a way forward to keep doing nothing, to justify their ongoing delinquency & inactivity. Which was unprincipled terrible and awful before, but which they basically bribed gregkh to suddenly make acceptable by at least paying for the privilege of being negligent do nothing delinquents on.
josephcsible 3y ago
With the "never break userspace" guarantee, is there ever a reason to want to be on an LTS kernel instead of the latest stable one, other than proprietary kernel modules?
- craftkiller 3y ago
  Yes, ZFS. When new, incompatible kernels are released the dkms build for the ZFS kernel modules will fail. By switching to an LTS kernel, I no longer have to worry because my kernel lags so far behind. The alternative is using a pre-built package repo which will prevent the kernel from updating to an incompatible version using the package dependencies. I lived that way for years and it is an awful experience.
  - XorNot 3y ago
    As a ZFS user, this really isn't a good enough reason IMO. Never break userspace really should be enough for most people.
  - phone8675309 3y ago
    This wouldn't be a problem if OpenZFS didn't have a dogshit license.
    - craftkiller 3y ago
      The same could be said for Linux. This isn't a problem on FreeBSD.
      - josephcsible 3y ago
        > The same could be said for Linux. No, because the CDDL was intentionally written to be incompatible with the GPL.
        craftkiller 3y ago
        The original intent of the license authors is irrelevant. The aspect of the CDDL that makes it incompatible with GPL is present in the GPL too. Neither license is more or less "dogshit" than the other, they are the same. The difference is the CDDL only applies to code written under the CDDL, whereas the GPL spreads to everything it touches.
blibble 3y ago
I figured that with the random bugs and policy changes (i.e. break ZFS again) that seem to appear in "LTS" mode on a regular basis
1970-01-01 3y ago
Going back to 2017, some of you knew this was not a tenable solution: https://news.ycombinator.com/item?id=15428340 https://news.ycombinator.com/item?id=15428340
zshrc 3y ago
They are a lot of work. And expensive to maintain. You’re not getting just 10 years of support from Red Hat, but 10 years of kernel support as well.
- worthless-trash 3y ago
  14 years of RHEL 7 for ELS.
userbinator 3y ago
It's really disturbing that software is the only thing whose stability seems to be decreasing very significantly over time, and dragging down everything that it's embedded in. Who wants constant changes and breakage? Who wants software that's in constant need of updating? I'm pretty sure it's not the users.
- imtringued 3y ago
  That is how C rolls. In theory you need to formally verify every single C program to ensure that it does not violate memory safety. Yes, that is akin to the same straightjacket that people complain about in that iron oxide language.
- titzer 3y ago
  There's always time to refactor and tinker with incomplete migration, always time to debug and patch and fuss with updates, but never time to sit down, gather all the requirements, redesign, and reimplement.
  - arcticbull 3y ago
    Ultimately it's just a question of objectives. Usually software isn't written for its own sake. It's written to achieve a goal, meet a customer need, generate revenue, prove a market, etc. You can achieve those goals without gathering all the requirements, redesigning and re-implementing most of the time. Not in aerospace, or biomedical maybe, where we are willing to pay the outlandish velocity penalties. But most of the things we do aren't that. Generally, if you want to build a pristine, perfect snowflake, a work of art, then you'll be the only one working on it, on your own time while listening to German electronica, in your house. [1] Nothing wrong with that - I have a few of those projects myself - but I think it's important to remember. Linux is hoping to adjust the velocity-quality equilibrium a little closer to velocity, and a little further from quality. That's okay too. Linux doesn't have to be everything to everyone. It doesn't have to be flawless to meet the needs of any given person. [1] https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks
athrun 3y ago
As an ex-Novell/SUSE employee this makes sense to me. Upstream is supposed to keep marching onwards. Backporting is _so_ much work. And it's unfortunately not sexy work either, so it's always going to be hard attracting unpaid contributors for it. If you need stability and long term support as a customer, you have companies like RedHat or SUSE whose entire point is providing 10+ years maintenance on these components.
- structural 3y ago
  Unfortunately, none of these companies providing 10+ years of maintenance are doing so for most embedded devices. We either need to get SoC vendors to update their kernel baselines regularly This is hard, we've been trying for a decade and not seen much progress. Alternately, get them to backport fixes and patches (there's actually been quite a bit of progress here in getting them to actually take updates from the stable kernel at all! And that's getting thrown away now...)
  - worthless-trash 3y ago
    If they got their fixes upstream , they would be pulled into the same build tree that centos and rhel uses.
    - dcow 3y ago
      Which really makes you understand why the lgpl was a defeat.
      - tremon 3y ago
        What do you mean? The Linux kernel isn't licensed under the LGPL.
  - qznc 3y ago
    The effect of EU’s Cyber Resilience Act will be interesting. It requires maintenance, at least for security fixes.
pmontra 3y ago
My phone is running a minor version of 4.4 from March 2023. Kernel 4.4 is originally from 2016. This means that they are still patching the kernel after 7 years even if it's not a LTS version.
- brnt 3y ago
  Is Google's Android Linux tree something distros could use as LTS?
  - phh 3y ago
    Google pays gregkh to make LTS (not all LTS are made by him though), and he works mainline first, so it's already working that way
  - pjmlp 3y ago
    Hardly, given how the Linux kernel is an implementation detail, Linux drivers are considered legacy (all modern drivers should be in userspace since Treble, written in Java/C++/Rust), and the NDK doesn't expose Linux APIs as stable interface. So not something to build GNU/Linux distributions on top of.
    - redleader55 3y ago
      The drivers in userspace are part of the GKI initiative[0], not Treble [1]. Treble deals with separation between Vendor, System and OEM. It establishes a process (CTS & VTS tests) to ensure system components (HALs) stay compatible with whatever updates Google makes to Android, but it deals with the base Android, not the Kernel specifically. [0] - https://source.android.com/docs/core/architecture/kernel/generic-kernel-image https://source.android.com/docs/core/architecture/kernel/gen... [1] - https://android-developers.googleblog.com/2017/05/here-comes-treble-modular-base-for.html https://android-developers.googleblog.com/2017/05/here-comes...
      - pjmlp 3y ago
        Historically, Treble predates GKI, created after OEMs disregarded Treble, as Google had the clever idea to leave Treble updates as optional for OEMs. > Binderized HALs. HALs expressed in HAL interface definition language (HIDL) or Android interface definition language (AIDL). These HALs replace both conventional and legacy HALs used in earlier versions of Android. In a Binderized HAL, the Android framework and HALs communicate with each other using binder inter-process communication (IPC) calls. All devices launching with Android 8.0 or later must support binderized HALs only. https://source.android.com/docs/core/architecture/hal https://source.android.com/docs/core/architecture/hal GKI only became a thing in Android 12 to fix Treble adoption issues, as you can also easily check, and GSI was introduced in Android 9, after userspace drivers became a requirement in Android 8 as per link above. https://arstechnica.com/gadgets/2017/05/ars-talks-android-googlers-chat-about-project-treble-os-updates-and-linux/ https://arstechnica.com/gadgets/2017/05/ars-talks-android-go... https://arstechnica.com/gadgets/2021/11/android-12-the-ars-technica-review/3/ https://arstechnica.com/gadgets/2021/11/android-12-the-ars-t...
TowerTall 3y ago
Microsoft is famed for their backwards combability. How do they achieve this? By hard work and a lot of "if version == x" spread throughout the code? Or is because of their development process or do they plan and design for backwards combability from day one?
- zik 3y ago
  A friend who worked at MS tells me that there's a huge amount of "if version" in their code. Apparently it's at the level where it's a big maintenance headache.
  - somsak2 3y ago
    maintenance burden has to be felt somewhere. it's nice when the vendor does it for you.
    - rcme 3y ago
      Maybe for a while. But when you add the maintenance burden to the code, it stays there, forever being felt. Over time, this degrades the product for everyone. And indeed, Windows can be unpleasant to use, not least of all because it feels like glued together legacy systems.
  - spondyl 3y ago
    A running theory was that Windows 9 was skipped because of all the "if version contains 9" logic for 95, 98 etc I don't think it was ever anything more than speculation though
    - noirscape 3y ago
      IIRC it was partially confirmed when some Windows 11 beta builds started causing issues with software thinking it was being executed on 1.1.x (whose identifier internally apparently is 11).
    - GrumpySloth 3y ago
dathinab 3y ago
in my opinion for anything internet connected not updating the kernel is a liability security patches of an LTS kernel are as much updates as moving to a newer kernel version custom non in-tree drivers are generally an anti-pattern the kernel interface is quite stable automated testing tools have come quite a way ===> you should fully update the kernel LTS isn't needed the only offenders which makes this hard are certain hardware vendors mostly related to phones and IoT which provide proprietary drivers only and also do not update them even with LTS kernels this has caused ton's of problems over time maybe 6-years LTS being absconded in combination with some legislatures starting to require security updates for devices for 2-5 years *after sold* (i.e > released) this will put enough pressure on to change this for a better approach (weather that are user land drivers, in-tree drivers or better driver support in general)
polskibus 3y ago
Won’t the new right to repair regulations both in eu and us affect this stance?
- ghaff 3y ago
  Why? You have the right to backport fixes yourself to your heart’s delight. Right to repair doesn’t require someone else to do the work for you for free.
  - diffeomorphism 3y ago
    Because. The "new right to repair regulations" aim to improve sustainability and that includes repair and support and upgrades, e.g. see https://www.androidauthority.com/eu-smartphone-updates-rules-3205112/ https://www.androidauthority.com/eu-smartphone-updates-rules... > Right to repair doesn’t require someone else to do the work for you for free. The term maybe not, but the proposed legislation totally does. Same as warranties or customer protection or not using toxic materials or ... ; none of that is "for free" to the manufacturer, but it is mandatory if you want to be allowed to sell your product.
    - ghaff 3y ago
      First of all this isn’t about a product. But if legislation would actually require the Linux kernel, say, to have LTS for??? every major release? Every point release? It’s bad law and should absolutely not exist. If I’m running a community project I have a big FU for anyone trying to impose support requirements on me. Which was actually a rather hot topic at an open source conference I was just at in Europe.
      - diffeomorphism 3y ago
        > It’s bad law and should absolutely not exist. Whatever you came up with in your mind sounds very weird and, yeah, obviously should not exist. That has nothing to do with the actual law though. > I have a big FU for anyone trying to impose support requirements on me Nobody talked about any of that? Product: Samsung phone. Requirement: Samsung needs to keep that device usable for N years. To meet that requirement Samsung will also need kernel updates. Whether that means doing them in house, paying someone else, making updates more seamless to easily upgrade or ... . The requirement to find a way to make that work is on Samsung, not you.
vermaden 3y ago
On the contrary the FreeBSD project offers stable 'LTS' release for 5 years each. What I mean by that is each 'major' version with stable API/ABI has a life span of about 5 years - like 5 years of 12.x version, 5 years of 13.x version, etc. ... and all that with having about only 1/10 of the Linux people count (rough estimate).
- pengaru 3y ago
  > ... and all that with having about only 1/10 of the Linux people count (rough estimate). It's easy to offer a long LTS release cadence when your project is stagnant.
  - ktaylora 3y ago
    It's pretty active, actually. Look at the release notes for FreeBSD major versions. Some folks think the release engineering team is too active and that major versions should be supported for more than ~5 years.
  - vermaden 3y ago
    If this [1] is stagnant then I do not know what is not ... [1] https://cgit.freebsd.org/src/log/ https://cgit.freebsd.org/src/log/
    - pengaru 3y ago
      That's all of FreeBSD including userspace, not just the kernel. In Linux land you see more than that level of activity in just the kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... I didn't say FreeBSD was abandonware, its kernel development has just been relatively stagnant for decades vs. Linux. Which shouldn't come as particularly surprising considering how much more adoption and investment there's been surrounding Linux over that time period.
- ladyanita22 3y ago
  Yeah FreeBSD is not happening, sorry.
runeks 3y ago
> The other big problem is the burnout from maintainers, which are often unpaid and could use a lot more support from the billion-dollar companies that benefit from using Linux. This is the crux of the issue: putting the maintenance burden on unpaid volunteers instead of having the burden be carried by the companies that profit from the 6-year LTS.
- justincormack 3y ago
  To be fair some companies do maintain LTS kernels like Red Hat but they choose different kernel versions than upstream LTS and backport different things so it doesn’t have much crossover with these ones now.
- INTPenis 3y ago
  Spot on, I mean who wants 6 year LTS support? Must be big old slow enterprises and possibly banks, government agencies, so pay up.
  - cvwright 3y ago
    In practice, that would require those companies to pay for a distro like RedHat, who supports a lot of the kernel dev work. But unfortunately everyone would rather find a way to get RH packages for free, and RH/IBM has only made the situation worse by alienating people.
    - red-iron-pine 3y ago
      aye. the second IBM took over it was clear it would be going to shit. it hasn't, totally, but im not loving the long-term direction
  - supertrope 3y ago
    Windows is supported for 10 years if you upgrade on day one. No special support contract needed.
    - 3y ago
sylware 3y ago
My opinion is that 6 years is not enough, I would target 10 years. But I guess the core of the issue is planned obsolescence in linux internals. Namely the fear of missing out some features which require significant internal changes and which if linux is without could lower its attractiveness in some ways. It all depends on Linus T.: arbitration of "internals breaking changes".
- worksonmine 3y ago
  Do you have the resources to achieve that target and still move the project forward? If I could ask anything from Linus it would be to be a little more relaxed about the "never break userspace" rule. Allow for some innovation and improvements. There are bugs in the kernel that have become documented features because some userspace program took advantage of that.
  - sylware 3y ago
    Arbitration, which is ulitmately in the hands of Linus T. Where ABI stability is paramount for Linus T., ABI bugs will become features. The glibc/libgcc/libstdc++ found a way around it... which did end up even worse: GNU symbol versioning. Basically, "fixed" symbols get a new version, BUT sourceware binutils is always linking with the latest symbol version, which makes generating "broad glibc version spectrum compatibility" binaries an abomination (I am trying to stay polite)... because glibc/libgcc/libstdc++ devs are grossely abusing GNU symbol versioning. Game/game engine devs are hit super hard. It is actually a real mess, valve is trying technical mitigations, but there are all shabby and a pain to maintain. Basically, they are killing native elf/linux gaming with this kind of abuse.
ralferoo 3y ago
Some comments are saying that the 6-year LTS is needed to support older Android devices. Also, in practice most vendors don't bother releasing updates to phones after the first 2 years, other than security updates. One possible nice side-effect of not maintaining kernels for so long and allowing people to stay on out-of-date systems would be to encourage vendors to allow users to upgrade to the newer versions of Android for more than the current 2 year life span. They are then more likely to put pressure on their component vendors to get kernel support for their chipsets into mainline so they don't have the excuse that they can't provide updates because the hardware isn't support by modern firmware.
compiler-devel 3y ago
Could this be a reaction to RedHat’s latest position on RHEL rebuilders? Did kernel devs see RH as freeloading on the LTS kernels?
- tuna74 3y ago
  Absolutely, especially since RHEL never used the LTS Linux kernels. /s
dingi 3y ago
Good move! At least, this will push those pesky Android OEMs to make their drivers available upstream. Its time we have a standardized environment for mobile/embedded use cases like the one we have for PCs. All these stupid little devices filling the junk yards because some greedy OEM didn't want to update their drivers is ridiculous.
ksec 3y ago
And yet most of the Synology NAS are still running on 4.x kernel.
- tuna74 3y ago
  Mine runs on 5.10...
kashyapc 3y ago
(Disclaimer: I work for Red Hat, but I don't work on the kernel. I'm a user-land mammal, and sometimes work with kernel maintainers to debug issues.) It makes total sense and I support this. I've met some of the upstream Linux maintainers at conferences over the years. Some (many?) of them are really oversubscribed and still plug away. They need relief from the drudgery of LTS stuff at some point. Anyone here involved in backporting fixes to many stable branches in a user-land project will relate to the problem here. It's time-consuming and tedious work. This kind of work is what "Enterprise Linux" companies get paid for by customers.
phendrenad2 3y ago
This is why I like MINIX. They last LTS release was 9 years ago and it's still just as unsupported as ever!

274 points by rascul 3y ago

6 years seems like an incredibly long time to me. It's a bit of a shame, but looks like industry really didn't come out to support the idea. 2 years on the other hand seems incredibly short? Am I wrong?

LinuxBender 3y ago
That's 2 years of the upstream LTS kernel. I would expect that major Linux distributions such as Redhat RHEL and Canonical's Ubuntu would continue to do their extended patch cycles against one of the upstream snapshots as they have done in the past. I think 2 years for upstream LTS is probably fine if the vendor patching methodology remains true. This also assumes that the usage of smaller distributions such as Alpine are more commonly used in very agile environments such as K8's, Docker Swarm, etc... Perhaps that is a big assumption on my part.
inferiorhuman 3y ago
6 years may seem like a long time, but check out what the competition is doing. Oracle is supporting Solaris 10 for 20 years, 11.4 for 16 years (23 years if you lump it in with 11.0). HP-UX 11i versions seem to get around 15 years of support. It really depends on what you're doing, a lot of industries may not need such long-term support. 6 years seems like a happy medium to me, but then again I'm not the one supporting it. I expect the kernel devs would be singing a different tune if people were willing to pay for that extended support. https://upload.wikimedia.org/wikipedia/en/timeline/276jjn0uo65zngws02n9iabdtnhic97.png https://upload.wikimedia.org/wikipedia/en/timeline/276jjn0uo...
- wkat4242 3y ago
  Are those really competition anymore? They're just legacy now IMO and their long term support requirements are a result of this, companies that haven't gotten rid of them by now aren't likely to do it any time soon. I hate seeing them go. I wasn't such a fan of Solaris but I was of HP-UX. But its days are over. It doesn't even run on x86 or x64 and HP has been paying Intel huge money to keep itanium on life support, which is running out now if it hasn't already. At least Solaris had an Intel port but it too is very rare now.
  - kube-system 3y ago
    Theres a lot of people who haven't got rid of old Linux systems these days too. RHEL 6 from 2010 is still eligible for extended support.
    - structural 3y ago
      There's still a decent population of RHEL 5 systems in the wild. Last year I was offered an engagement (turned down for a few reasons) to help a company upgrade several hundred systems from RHEL 5 to RHEL 6 and start planning for a future rollout of RHEL 7. Outside of tech focused companies, 10+ year old systems really are the norm.
      - SoftTalker 3y ago
        > Outside of tech focused companies, 10+ year old systems really are the norm. It's because outside of tech companies, nobody cares about new features. They care about things continuing to work. Companies don't buy software for the fun of exploring new versions, especially frustratingly pointless cosmetic changes that keep hitting their training budgets. Many companies would be happy with RHEL5 or Windows XP today from a feature standpoint, if it weren't a security vulnerability.
        wkat4242 3y ago
        The problem about "things continuing to work" is really that many security fixes require updated architecture too. This is really why it's so hard to do LTS. It's not only about wanting new features.
      - worthless-trash 3y ago
        What feature did they need in el7 that wasn't there in el9 ? What was their logic ?
      - inferiorhuman 3y ago
        At megacorp (years ago) we were transitioning to CentOS 7 (from 6) and just starting to wind down our 32-bit windows stuff in AWS. I'm sure there are plenty of legacy Linux systems out there, but I wonder how many folks are actually paying for them. CentOS/RHEL 6 was already pretty long in the tooth, but being the contrarian I am, I was not looking forward to the impending systemd nonsense.
kube-system 3y ago
Depends on where the computer is at, I guess. On a desk, 6 years is a pretty long time. In an industrial setting, 6 years is not very long of a lifecycle.
- dathinab 3y ago
  consider that it's 6 years after release of the kernel version so likely <5 years since release of the hardware in the US likely <4 years since release of hardware outside of the US likely <3 years since you bought the hardware and if you buy older phones having only a year or so of proper security updates is not that unlikely So for phones you would need more something like a 8 or 10 year LTS, or well, proper driver updates for proprietary hardware. In which case 2 years can be just fine because in general/normally only drivers are affected by kernel updates.
Gigachad 3y ago
What kind of breaking change would take longer than 2 years to deal with? The reality is that people wait out the entire 6 year period and then do the required months work at the end. If you make the support period 2 years they will just start working on it sooner.
- Retric 3y ago
  Many perhaps most projects don’t last 6 years, thus punting can save people a great deal of time.
dathinab 3y ago
6 years is way to short for some use-cases, like phones sadly it's not rare that some phone hardware proprietary drivers are basically written once and then hardly maintained and in turn only work for a small number of Linux kernel versions. so for some hardware it might mean that when it gets released you only have noticeable less then 6 years of kernel support then between starting to build a phone and releasing it 2 years might easily pass so that means from release you can provide _at most_ 4 years of kernel security patches etc. but dates tend to not align that grate so maybe it's just 3 years but then you sell your phone for more then one year, right? in which case less then 2 years can be between the customer buying your product and you no longer providing kernel updates that is a huge issue I mean think about it if someone is a bit thigh on money and buy a slightly older phone they probably aren't aware that their phone stops getting security updates in a year or so (3 years software support since release but it's a 2 year old phone) at which point using it is a liability. EDIT: The answer is in my opinion not to have even longer LTS but to proper maintain drivers and in turn being able to do full kernel updates
- xvilka 3y ago
  > sadly it's not rare that some phone hardware proprietary drivers are basically written once and then hardly maintained and in turn only work for a small number of Linux kernel versions. These manufacturers should be punished by the lack of LTS and need to upgrade precisely because of that lazyness and incompetence.
  - charcircuit 3y ago
    Why not blame the kernel developers for being lazy and incompetent for not offering a stable API for driver developers to use? You don't see Windows driver developers having their drivers broken by updates every few months.
    - mschuster91 3y ago
      > You don't see Windows driver developers having their drivers broken by updates every few months. At the cost of Windows kernel development being a huge PITA because effectively everything in the driver development kit becomes an ossified API that can't ever change, no matter if there are bugs or there are more efficient ways to get something done. The Linux kernel developers can do whatever they want to get the best (most performant, most energy saving, ...) system because they don't need to worry about breaking someone else's proprietary code. Device manufacturers can always do the right thing and provide well-written modules to upstream - but many don't because (rightfully) the Linux kernel team demands good code quality which is expensive AF. Just look at the state of most non-Pixel/Samsung code dumps, if you're dedicated enough you'll find tons of vulnerabilities and code smell.
Ekaros 3y ago
All comes to cycle. When do you enter that 6 year LTS? Is there new LTS every year or every other year? If you enter 2 years in or even 4 years in. How much have you support left? Do you jump LTS releases. So one you are on is ending and there is brand new available? Or do you go to one before and have possibly only 2 or 4 years left...

jauntywundrkind 3y ago

Google showed up & offered to pay a huge amount of money to extend LTS support 3x, iirc. At the time, Linux 4.14 was shipping I think. Personally it made me a bit sad, because it created real permission for Android to never upgrade kernels. I'd hoped eventually Android would start upgrading kernels, thought the short LTS would surely force them to become a respectable ecosystem that takes maintenance seriously. Making Super LTS was a dodge; suddenly it was ok that 4.14 straggles along until 2024. Also an interesting note, supposedly there's a "Civil Infrastructure Platform" that supposedly will have some support until 2029 for 4.14! The past never is gone, eh? Supposedly Project Treble is a whole new driver abstraction in I think mostly userland (maybe?) whose intent is to allow kernel upgrades without having to rewrite drivers (not sure if that is the primary/express goal or was just often mentioned). I'm not sure if Android has yet shipped a kernel upgrade to any phones though; anyone know of an model running an upgraded kernel?

yjftsjthsd-h 3y ago
> Personally it made me a bit sad, because it created real permission for Android to never upgrade kernels. I would argue that it was still better than not doing that, because the vendors weren't going to properly keep up with kernels either way; the choice wasn't "support 3.x for longer or move to 4.x", it was "support 3.x for longer or watch Android devices stay on 3.x without patches".
- jauntywundrkind 3y ago
  As the old adage goes, "chips without software is just expensive sand." Yes what happened & happens is often monstrously unsupportable & terrible. For the past 6 years, Google rolling up with a dump truck full of bills has been justification to keep doing nothing, to keep letting kernel devices be bad. Your history isn't even good or right. Old releases at the time didn't get official longer support. Canonical just opted to maintain a basically Super LTS 3.16 until 2020, regardless of the Google dump-truck-of-money thing going on here. Old phones got nothing directly from this payoff. Google was just paying for a way forward to keep doing nothing, to justify their ongoing delinquency & inactivity. Which was unprincipled terrible and awful before, but which they basically bribed gregkh to suddenly make acceptable by at least paying for the privilege of being negligent do nothing delinquents on.

josephcsible 3y ago

With the "never break userspace" guarantee, is there ever a reason to want to be on an LTS kernel instead of the latest stable one, other than proprietary kernel modules?

craftkiller 3y ago
Yes, ZFS. When new, incompatible kernels are released the dkms build for the ZFS kernel modules will fail. By switching to an LTS kernel, I no longer have to worry because my kernel lags so far behind. The alternative is using a pre-built package repo which will prevent the kernel from updating to an incompatible version using the package dependencies. I lived that way for years and it is an awful experience.
- XorNot 3y ago
  As a ZFS user, this really isn't a good enough reason IMO. Never break userspace really should be enough for most people.
- phone8675309 3y ago
  This wouldn't be a problem if OpenZFS didn't have a dogshit license.
  - craftkiller 3y ago
    The same could be said for Linux. This isn't a problem on FreeBSD.
    - josephcsible 3y ago
      > The same could be said for Linux. No, because the CDDL was intentionally written to be incompatible with the GPL.
      - craftkiller 3y ago
        The original intent of the license authors is irrelevant. The aspect of the CDDL that makes it incompatible with GPL is present in the GPL too. Neither license is more or less "dogshit" than the other, they are the same. The difference is the CDDL only applies to code written under the CDDL, whereas the GPL spreads to everything it touches.

blibble 3y ago

I figured that with the random bugs and policy changes (i.e. break ZFS again) that seem to appear in "LTS" mode on a regular basis

1970-01-01 3y ago

Going back to 2017, some of you knew this was not a tenable solution: https://news.ycombinator.com/item?id=15428340 https://news.ycombinator.com/item?id=15428340

zshrc 3y ago

They are a lot of work. And expensive to maintain. You’re not getting just 10 years of support from Red Hat, but 10 years of kernel support as well.

worthless-trash 3y ago
14 years of RHEL 7 for ELS.

userbinator 3y ago

It's really disturbing that software is the only thing whose stability seems to be decreasing very significantly over time, and dragging down everything that it's embedded in. Who wants constant changes and breakage? Who wants software that's in constant need of updating? I'm pretty sure it's not the users.

imtringued 3y ago
That is how C rolls. In theory you need to formally verify every single C program to ensure that it does not violate memory safety. Yes, that is akin to the same straightjacket that people complain about in that iron oxide language.
titzer 3y ago
There's always time to refactor and tinker with incomplete migration, always time to debug and patch and fuss with updates, but never time to sit down, gather all the requirements, redesign, and reimplement.
- arcticbull 3y ago
  Ultimately it's just a question of objectives. Usually software isn't written for its own sake. It's written to achieve a goal, meet a customer need, generate revenue, prove a market, etc. You can achieve those goals without gathering all the requirements, redesigning and re-implementing most of the time. Not in aerospace, or biomedical maybe, where we are willing to pay the outlandish velocity penalties. But most of the things we do aren't that. Generally, if you want to build a pristine, perfect snowflake, a work of art, then you'll be the only one working on it, on your own time while listening to German electronica, in your house. [1] Nothing wrong with that - I have a few of those projects myself - but I think it's important to remember. Linux is hoping to adjust the velocity-quality equilibrium a little closer to velocity, and a little further from quality. That's okay too. Linux doesn't have to be everything to everyone. It doesn't have to be flawless to meet the needs of any given person. [1] https://www.stilldrinking.org/programming-sucks https://www.stilldrinking.org/programming-sucks

athrun 3y ago

As an ex-Novell/SUSE employee this makes sense to me. Upstream is supposed to keep marching onwards. Backporting is _so_ much work. And it's unfortunately not sexy work either, so it's always going to be hard attracting unpaid contributors for it. If you need stability and long term support as a customer, you have companies like RedHat or SUSE whose entire point is providing 10+ years maintenance on these components.

structural 3y ago
Unfortunately, none of these companies providing 10+ years of maintenance are doing so for most embedded devices. We either need to get SoC vendors to update their kernel baselines regularly This is hard, we've been trying for a decade and not seen much progress. Alternately, get them to backport fixes and patches (there's actually been quite a bit of progress here in getting them to actually take updates from the stable kernel at all! And that's getting thrown away now...)
- worthless-trash 3y ago
  If they got their fixes upstream , they would be pulled into the same build tree that centos and rhel uses.
  - dcow 3y ago
    Which really makes you understand why the lgpl was a defeat.
    - tremon 3y ago
      What do you mean? The Linux kernel isn't licensed under the LGPL.
- qznc 3y ago
  The effect of EU’s Cyber Resilience Act will be interesting. It requires maintenance, at least for security fixes.

pmontra 3y ago

My phone is running a minor version of 4.4 from March 2023. Kernel 4.4 is originally from 2016. This means that they are still patching the kernel after 7 years even if it's not a LTS version.

brnt 3y ago
Is Google's Android Linux tree something distros could use as LTS?
- phh 3y ago
  Google pays gregkh to make LTS (not all LTS are made by him though), and he works mainline first, so it's already working that way
- pjmlp 3y ago
  Hardly, given how the Linux kernel is an implementation detail, Linux drivers are considered legacy (all modern drivers should be in userspace since Treble, written in Java/C++/Rust), and the NDK doesn't expose Linux APIs as stable interface. So not something to build GNU/Linux distributions on top of.
  - redleader55 3y ago
    The drivers in userspace are part of the GKI initiative[0], not Treble [1]. Treble deals with separation between Vendor, System and OEM. It establishes a process (CTS & VTS tests) to ensure system components (HALs) stay compatible with whatever updates Google makes to Android, but it deals with the base Android, not the Kernel specifically. [0] - https://source.android.com/docs/core/architecture/kernel/generic-kernel-image https://source.android.com/docs/core/architecture/kernel/gen... [1] - https://android-developers.googleblog.com/2017/05/here-comes-treble-modular-base-for.html https://android-developers.googleblog.com/2017/05/here-comes...
    - pjmlp 3y ago
      Historically, Treble predates GKI, created after OEMs disregarded Treble, as Google had the clever idea to leave Treble updates as optional for OEMs. > Binderized HALs. HALs expressed in HAL interface definition language (HIDL) or Android interface definition language (AIDL). These HALs replace both conventional and legacy HALs used in earlier versions of Android. In a Binderized HAL, the Android framework and HALs communicate with each other using binder inter-process communication (IPC) calls. All devices launching with Android 8.0 or later must support binderized HALs only. https://source.android.com/docs/core/architecture/hal https://source.android.com/docs/core/architecture/hal GKI only became a thing in Android 12 to fix Treble adoption issues, as you can also easily check, and GSI was introduced in Android 9, after userspace drivers became a requirement in Android 8 as per link above. https://arstechnica.com/gadgets/2017/05/ars-talks-android-googlers-chat-about-project-treble-os-updates-and-linux/ https://arstechnica.com/gadgets/2017/05/ars-talks-android-go... https://arstechnica.com/gadgets/2021/11/android-12-the-ars-technica-review/3/ https://arstechnica.com/gadgets/2021/11/android-12-the-ars-t...

TowerTall 3y ago

Microsoft is famed for their backwards combability. How do they achieve this? By hard work and a lot of "if version == x" spread throughout the code? Or is because of their development process or do they plan and design for backwards combability from day one?

zik 3y ago
A friend who worked at MS tells me that there's a huge amount of "if version" in their code. Apparently it's at the level where it's a big maintenance headache.
- somsak2 3y ago
  maintenance burden has to be felt somewhere. it's nice when the vendor does it for you.
  - rcme 3y ago
    Maybe for a while. But when you add the maintenance burden to the code, it stays there, forever being felt. Over time, this degrades the product for everyone. And indeed, Windows can be unpleasant to use, not least of all because it feels like glued together legacy systems.
- spondyl 3y ago
  A running theory was that Windows 9 was skipped because of all the "if version contains 9" logic for 95, 98 etc I don't think it was ever anything more than speculation though
  - noirscape 3y ago
    IIRC it was partially confirmed when some Windows 11 beta builds started causing issues with software thinking it was being executed on 1.1.x (whose identifier internally apparently is 11).
  - GrumpySloth 3y ago

dathinab 3y ago

in my opinion for anything internet connected not updating the kernel is a liability security patches of an LTS kernel are as much updates as moving to a newer kernel version custom non in-tree drivers are generally an anti-pattern the kernel interface is quite stable automated testing tools have come quite a way ===> you should fully update the kernel LTS isn't needed the only offenders which makes this hard are certain hardware vendors mostly related to phones and IoT which provide proprietary drivers only and also do not update them even with LTS kernels this has caused ton's of problems over time maybe 6-years LTS being absconded in combination with some legislatures starting to require security updates for devices for 2-5 years *after sold* (i.e > released) this will put enough pressure on to change this for a better approach (weather that are user land drivers, in-tree drivers or better driver support in general)

polskibus 3y ago

Won’t the new right to repair regulations both in eu and us affect this stance?

ghaff 3y ago
Why? You have the right to backport fixes yourself to your heart’s delight. Right to repair doesn’t require someone else to do the work for you for free.
- diffeomorphism 3y ago
  Because. The "new right to repair regulations" aim to improve sustainability and that includes repair and support and upgrades, e.g. see https://www.androidauthority.com/eu-smartphone-updates-rules-3205112/ https://www.androidauthority.com/eu-smartphone-updates-rules... > Right to repair doesn’t require someone else to do the work for you for free. The term maybe not, but the proposed legislation totally does. Same as warranties or customer protection or not using toxic materials or ... ; none of that is "for free" to the manufacturer, but it is mandatory if you want to be allowed to sell your product.
  - ghaff 3y ago
    First of all this isn’t about a product. But if legislation would actually require the Linux kernel, say, to have LTS for??? every major release? Every point release? It’s bad law and should absolutely not exist. If I’m running a community project I have a big FU for anyone trying to impose support requirements on me. Which was actually a rather hot topic at an open source conference I was just at in Europe.
    - diffeomorphism 3y ago
      > It’s bad law and should absolutely not exist. Whatever you came up with in your mind sounds very weird and, yeah, obviously should not exist. That has nothing to do with the actual law though. > I have a big FU for anyone trying to impose support requirements on me Nobody talked about any of that? Product: Samsung phone. Requirement: Samsung needs to keep that device usable for N years. To meet that requirement Samsung will also need kernel updates. Whether that means doing them in house, paying someone else, making updates more seamless to easily upgrade or ... . The requirement to find a way to make that work is on Samsung, not you.

vermaden 3y ago

On the contrary the FreeBSD project offers stable 'LTS' release for 5 years each. What I mean by that is each 'major' version with stable API/ABI has a life span of about 5 years - like 5 years of 12.x version, 5 years of 13.x version, etc. ... and all that with having about only 1/10 of the Linux people count (rough estimate).

pengaru 3y ago
> ... and all that with having about only 1/10 of the Linux people count (rough estimate). It's easy to offer a long LTS release cadence when your project is stagnant.
- ktaylora 3y ago
  It's pretty active, actually. Look at the release notes for FreeBSD major versions. Some folks think the release engineering team is too active and that major versions should be supported for more than ~5 years.
- vermaden 3y ago
  If this [1] is stagnant then I do not know what is not ... [1] https://cgit.freebsd.org/src/log/ https://cgit.freebsd.org/src/log/
  - pengaru 3y ago
    That's all of FreeBSD including userspace, not just the kernel. In Linux land you see more than that level of activity in just the kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... I didn't say FreeBSD was abandonware, its kernel development has just been relatively stagnant for decades vs. Linux. Which shouldn't come as particularly surprising considering how much more adoption and investment there's been surrounding Linux over that time period.
ladyanita22 3y ago
Yeah FreeBSD is not happening, sorry.

runeks 3y ago

> The other big problem is the burnout from maintainers, which are often unpaid and could use a lot more support from the billion-dollar companies that benefit from using Linux. This is the crux of the issue: putting the maintenance burden on unpaid volunteers instead of having the burden be carried by the companies that profit from the 6-year LTS.

justincormack 3y ago
To be fair some companies do maintain LTS kernels like Red Hat but they choose different kernel versions than upstream LTS and backport different things so it doesn’t have much crossover with these ones now.
INTPenis 3y ago
Spot on, I mean who wants 6 year LTS support? Must be big old slow enterprises and possibly banks, government agencies, so pay up.
- cvwright 3y ago
  In practice, that would require those companies to pay for a distro like RedHat, who supports a lot of the kernel dev work. But unfortunately everyone would rather find a way to get RH packages for free, and RH/IBM has only made the situation worse by alienating people.
  - red-iron-pine 3y ago
    aye. the second IBM took over it was clear it would be going to shit. it hasn't, totally, but im not loving the long-term direction
- supertrope 3y ago
  Windows is supported for 10 years if you upgrade on day one. No special support contract needed.
  - 3y ago

sylware 3y ago

My opinion is that 6 years is not enough, I would target 10 years. But I guess the core of the issue is planned obsolescence in linux internals. Namely the fear of missing out some features which require significant internal changes and which if linux is without could lower its attractiveness in some ways. It all depends on Linus T.: arbitration of "internals breaking changes".

worksonmine 3y ago
Do you have the resources to achieve that target and still move the project forward? If I could ask anything from Linus it would be to be a little more relaxed about the "never break userspace" rule. Allow for some innovation and improvements. There are bugs in the kernel that have become documented features because some userspace program took advantage of that.
- sylware 3y ago
  Arbitration, which is ulitmately in the hands of Linus T. Where ABI stability is paramount for Linus T., ABI bugs will become features. The glibc/libgcc/libstdc++ found a way around it... which did end up even worse: GNU symbol versioning. Basically, "fixed" symbols get a new version, BUT sourceware binutils is always linking with the latest symbol version, which makes generating "broad glibc version spectrum compatibility" binaries an abomination (I am trying to stay polite)... because glibc/libgcc/libstdc++ devs are grossely abusing GNU symbol versioning. Game/game engine devs are hit super hard. It is actually a real mess, valve is trying technical mitigations, but there are all shabby and a pain to maintain. Basically, they are killing native elf/linux gaming with this kind of abuse.

ralferoo 3y ago

Some comments are saying that the 6-year LTS is needed to support older Android devices. Also, in practice most vendors don't bother releasing updates to phones after the first 2 years, other than security updates. One possible nice side-effect of not maintaining kernels for so long and allowing people to stay on out-of-date systems would be to encourage vendors to allow users to upgrade to the newer versions of Android for more than the current 2 year life span. They are then more likely to put pressure on their component vendors to get kernel support for their chipsets into mainline so they don't have the excuse that they can't provide updates because the hardware isn't support by modern firmware.

compiler-devel 3y ago

Could this be a reaction to RedHat’s latest position on RHEL rebuilders? Did kernel devs see RH as freeloading on the LTS kernels?

tuna74 3y ago
Absolutely, especially since RHEL never used the LTS Linux kernels. /s

dingi 3y ago

Good move! At least, this will push those pesky Android OEMs to make their drivers available upstream. Its time we have a standardized environment for mobile/embedded use cases like the one we have for PCs. All these stupid little devices filling the junk yards because some greedy OEM didn't want to update their drivers is ridiculous.

ksec 3y ago

And yet most of the Synology NAS are still running on 4.x kernel.

tuna74 3y ago
Mine runs on 5.10...

kashyapc 3y ago

(Disclaimer: I work for Red Hat, but I don't work on the kernel. I'm a user-land mammal, and sometimes work with kernel maintainers to debug issues.) It makes total sense and I support this. I've met some of the upstream Linux maintainers at conferences over the years. Some (many?) of them are really oversubscribed and still plug away. They need relief from the drudgery of LTS stuff at some point. Anyone here involved in backporting fixes to many stable branches in a user-land project will relate to the problem here. It's time-consuming and tedious work. This kind of work is what "Enterprise Linux" companies get paid for by customers.

phendrenad2 3y ago

This is why I like MINIX. They last LTS release was 9 years ago and it's still just as unsupported as ever!