5 ms·

Why don't Red Hat et al use the LTS kernels?

by baggy_trough 6y ago

pjmlp 6y ago
Probably drivers.
- bubblethink 6y ago
  What about them ? I don't think the OP means using the upstream kernel directly, just why they don't use the same version as the LTS version.
  - tssva 6y ago
    Reasons I can think of include: Not wanting to be tied to a kernel LTS release cycle you don't control. The need to backport features and hardware enablement to the LTS kernel significantly offsets some of the benefits of using a LTS kernel. The length of support for LTS kernels is significantly shorter than Redhat's length of support which means they would end up supporting it on their own for a significant timeframe anyways.
gregkh 6y ago
They feel like they know better and do not want all of the fixes that the LTS kernels provide for some crazy reason. I suggest you contact them if you rely on a RHEL kernel to ask them why they do this, it's always seemed crazy. Note, I'm the person who does the LTS kernel releases, maybe they just don't like me :)
- rwmj 6y ago
  I'm certain that Red Hat and our kernel developers have no animosity towards you, in fact it's completely the opposite. You're well known in the community not just for this but for maintaining and writing countless drivers and loads of other great work in the kernel. Fedora does use the LTS kernels.
- stonogo 6y ago
  It could also be motivated by the fact that it's not entirely wise to base an entire multibillion dollar business around whatever text a non-employee happens to push to a repo you don't control.
  - gregkh 6y ago
    Android doesn't seem to mind, they require the LTS updates to be taken for their devices (well, "require" is a strong word, they are pushing harder now than they were in the past, "required" will be happening in the future, hopefully...) As the number of systems running RHEL is really just a rounding error compared to the number of Android systems out there, maybe it doesn't really matter :)
    - azinman2 6y ago
      I would guess that RHEL has more stringent requirements than your average me-too Android manufacturer.
    - radiator 6y ago
      But do the devices running Android generally operate with the same requirements as the servers running RHEL?
      - nordsieck 6y ago
        > But do the devices running Android generally operate with the same requirements as the servers running RHEL? I guess it depends on the device and the business. How many days of downtime on Amazon.com is the cost of bricking 1M phones? Or 100M?
    - bonzini 6y ago
      Android and RHEL are completely different scenarios. The target of Android, which is who Google has to deal with, is multiple manufacturers creating kernels for custom hardware (often without upstream drivers), with very short product life, relatively little experience with upstream contribution and few needs for new features for a given major release of Android. RHEL is developed by a single company with a 10 years life cycle, only 3-4 kernel versions to juggle but almost non-overlapping lifecycle (as far as the initial development-heavy phase is concerned). Development occurs upstream first and quite a few engineers are upstream developers or maintainers, so that the number of non-upstream features is very small and almost going down over time, see for example stuff like the secure boot lockdown patches that Matthew Garrett started when he was at Red Hat. And even though the product is not the kernel, we need to backport more features than what goes into LTS, because userspace needs them (user namespaces, driver updates, networking or virtualization optimizations, enablement for new processors, etc.) So it's only natural that there are completely trade-offs to make.
- iforgotpassword 6y ago
  > They feel like they know better and do not want all of the fixes that the LTS kernels provide for some crazy reason. It's even crazier; they sometimes backport changes to their kernel that the LTS kernels don't get. We use a custom kernel module that contains a bunch of #if #endif blocks that check the kernel version for stuff that changed. Doesn't work on RedHat since you actually need the branch that's for more recent kernels in some places.
  - orf 6y ago
    Isn’t the whole autoconf stuff supposed to avoid the need for if macro soup, by using feature detection?
    - iforgotpassword 6y ago
      There would be cleaner ways to achieve this, maybe not specifically autoconf since I think that's more tailored towards "normal" (user space) stuff. Macros are convenient to quickly check the version in your code without adding another layer of tooling... Until you end up with said macro soup of course. It's actually a legacy module we're about to phase out for 5.x so don't worry too much. The new and shiny replacement will probably use git branches for whenever something changes.
gulikoza 6y ago
Red Hat sometimes also backports new features, not just security fixes. In the first years of RHEL life cycle new features and certain upgrades are usually added to the point releases. This is actually nicer than being locked into same feature kernel for 10 years...for those that can't afford the changes, you have the Extended Support for each point release which then gets only the security fixes.
Conan_Kudo 6y ago
Note that Red Hat, SUSE, and Canonical all don't use them. They maintain their own kernel trees with their modifications. These kernel teams have different priorities: * stable ABI within a timespan (RH, SUSE) * feature backports based on customer demand (all three) * out of tree goop (Canonical) I've also heard from others (though notably, not folks specifically at these companies) that longterm kernels are iffier than regular stable kernels because people come out of the woodwork to push weird stuff for longterm kernels, which destabilizes things more than with regular stable kernels.
- bluedays 6y ago
  Seems like even Arch maintains it's own kernel. Not uncommon for distros to do it at all.
  - artistsvoid 6y ago
    Wrong, or at the least misleading. What do you mean by 'maintain' here? Because Arch is trying to stay on the bleeding edge side of things in a reasonable manner and is rolling release, they almost stay 1:1 with upstream, "maintains its own kernel" makes it sound like they would do some heavy patching and do active maintenance with certain kernel versions, et cetera. You can have a look yourself if you want to https://git.archlinux.org/linux.git/commit/?h=v5.7.5-arch1&id=35f8fcadda6fd6d6eb582a7750ee307b2b822686 https://git.archlinux.org/linux.git/commit/?h=v5.7.5-arch1&i...
    - bluedays 6y ago
      They have their own kernel as can be seen here: https://i.imgur.com/jP6Pdvq.png https://i.imgur.com/jP6Pdvq.png Also if you look at the diffstat you can see that there were a number of changes here: https://i.imgur.com/J2em78v.png https://i.imgur.com/J2em78v.png It may be align with the majority of the kernel, but these patches exist in every version of the kernel that is released by Arch. What else could that be called aside from maintaining your own kernel? These changes are extremely small within scope which is the nature of Arch, but they still exist. A more hands-on distro will see many more changes than Arch, naturally. However, as evidenced from the above you can see that Arch does have it's own kernel and I cited Arch as an example because it strives so hard to maintain 1:1 with upstream. Btw, I use Arch.
drewg123 6y ago
RHEL maintains a KABI to support binary device drivers. Given how frequently the linux kernel changes internal APIs, this would be much harder or even impossible if they changed kernel versions regularly. Instead, they take features and bugfixes from upstream and merge them back into their kernel version, auditing them for changes that would not be binary compatible.
heavyset_go 6y ago
Ubuntu does the same thing. I'm only pointing it out because if you can't find a reason RH does it, you might be able to find the reason Canonical does it, though.

Why don't Red Hat et al use the LTS kernels?

pjmlp 6y ago

Probably drivers.

bubblethink 6y ago
What about them ? I don't think the OP means using the upstream kernel directly, just why they don't use the same version as the LTS version.
- tssva 6y ago
  Reasons I can think of include: Not wanting to be tied to a kernel LTS release cycle you don't control. The need to backport features and hardware enablement to the LTS kernel significantly offsets some of the benefits of using a LTS kernel. The length of support for LTS kernels is significantly shorter than Redhat's length of support which means they would end up supporting it on their own for a significant timeframe anyways.

gregkh 6y ago

They feel like they know better and do not want all of the fixes that the LTS kernels provide for some crazy reason. I suggest you contact them if you rely on a RHEL kernel to ask them why they do this, it's always seemed crazy. Note, I'm the person who does the LTS kernel releases, maybe they just don't like me :)

rwmj 6y ago
I'm certain that Red Hat and our kernel developers have no animosity towards you, in fact it's completely the opposite. You're well known in the community not just for this but for maintaining and writing countless drivers and loads of other great work in the kernel. Fedora does use the LTS kernels.
stonogo 6y ago
It could also be motivated by the fact that it's not entirely wise to base an entire multibillion dollar business around whatever text a non-employee happens to push to a repo you don't control.
- gregkh 6y ago
  Android doesn't seem to mind, they require the LTS updates to be taken for their devices (well, "require" is a strong word, they are pushing harder now than they were in the past, "required" will be happening in the future, hopefully...) As the number of systems running RHEL is really just a rounding error compared to the number of Android systems out there, maybe it doesn't really matter :)
  - azinman2 6y ago
    I would guess that RHEL has more stringent requirements than your average me-too Android manufacturer.
  - radiator 6y ago
    But do the devices running Android generally operate with the same requirements as the servers running RHEL?
    - nordsieck 6y ago
      > But do the devices running Android generally operate with the same requirements as the servers running RHEL? I guess it depends on the device and the business. How many days of downtime on Amazon.com is the cost of bricking 1M phones? Or 100M?
  - bonzini 6y ago
    Android and RHEL are completely different scenarios. The target of Android, which is who Google has to deal with, is multiple manufacturers creating kernels for custom hardware (often without upstream drivers), with very short product life, relatively little experience with upstream contribution and few needs for new features for a given major release of Android. RHEL is developed by a single company with a 10 years life cycle, only 3-4 kernel versions to juggle but almost non-overlapping lifecycle (as far as the initial development-heavy phase is concerned). Development occurs upstream first and quite a few engineers are upstream developers or maintainers, so that the number of non-upstream features is very small and almost going down over time, see for example stuff like the secure boot lockdown patches that Matthew Garrett started when he was at Red Hat. And even though the product is not the kernel, we need to backport more features than what goes into LTS, because userspace needs them (user namespaces, driver updates, networking or virtualization optimizations, enablement for new processors, etc.) So it's only natural that there are completely trade-offs to make.
iforgotpassword 6y ago
> They feel like they know better and do not want all of the fixes that the LTS kernels provide for some crazy reason. It's even crazier; they sometimes backport changes to their kernel that the LTS kernels don't get. We use a custom kernel module that contains a bunch of #if #endif blocks that check the kernel version for stuff that changed. Doesn't work on RedHat since you actually need the branch that's for more recent kernels in some places.
- orf 6y ago
  Isn’t the whole autoconf stuff supposed to avoid the need for if macro soup, by using feature detection?
  - iforgotpassword 6y ago
    There would be cleaner ways to achieve this, maybe not specifically autoconf since I think that's more tailored towards "normal" (user space) stuff. Macros are convenient to quickly check the version in your code without adding another layer of tooling... Until you end up with said macro soup of course. It's actually a legacy module we're about to phase out for 5.x so don't worry too much. The new and shiny replacement will probably use git branches for whenever something changes.

gulikoza 6y ago

Red Hat sometimes also backports new features, not just security fixes. In the first years of RHEL life cycle new features and certain upgrades are usually added to the point releases. This is actually nicer than being locked into same feature kernel for 10 years...for those that can't afford the changes, you have the Extended Support for each point release which then gets only the security fixes.

Conan_Kudo 6y ago

Note that Red Hat, SUSE, and Canonical all don't use them. They maintain their own kernel trees with their modifications. These kernel teams have different priorities: * stable ABI within a timespan (RH, SUSE) * feature backports based on customer demand (all three) * out of tree goop (Canonical) I've also heard from others (though notably, not folks specifically at these companies) that longterm kernels are iffier than regular stable kernels because people come out of the woodwork to push weird stuff for longterm kernels, which destabilizes things more than with regular stable kernels.

bluedays 6y ago
Seems like even Arch maintains it's own kernel. Not uncommon for distros to do it at all.
- artistsvoid 6y ago
  Wrong, or at the least misleading. What do you mean by 'maintain' here? Because Arch is trying to stay on the bleeding edge side of things in a reasonable manner and is rolling release, they almost stay 1:1 with upstream, "maintains its own kernel" makes it sound like they would do some heavy patching and do active maintenance with certain kernel versions, et cetera. You can have a look yourself if you want to https://git.archlinux.org/linux.git/commit/?h=v5.7.5-arch1&id=35f8fcadda6fd6d6eb582a7750ee307b2b822686 https://git.archlinux.org/linux.git/commit/?h=v5.7.5-arch1&i...
  - bluedays 6y ago
    They have their own kernel as can be seen here: https://i.imgur.com/jP6Pdvq.png https://i.imgur.com/jP6Pdvq.png Also if you look at the diffstat you can see that there were a number of changes here: https://i.imgur.com/J2em78v.png https://i.imgur.com/J2em78v.png It may be align with the majority of the kernel, but these patches exist in every version of the kernel that is released by Arch. What else could that be called aside from maintaining your own kernel? These changes are extremely small within scope which is the nature of Arch, but they still exist. A more hands-on distro will see many more changes than Arch, naturally. However, as evidenced from the above you can see that Arch does have it's own kernel and I cited Arch as an example because it strives so hard to maintain 1:1 with upstream. Btw, I use Arch.

drewg123 6y ago

RHEL maintains a KABI to support binary device drivers. Given how frequently the linux kernel changes internal APIs, this would be much harder or even impossible if they changed kernel versions regularly. Instead, they take features and bugfixes from upstream and merge them back into their kernel version, auditing them for changes that would not be binary compatible.

heavyset_go 6y ago

Ubuntu does the same thing. I'm only pointing it out because if you can't find a reason RH does it, you might be able to find the reason Canonical does it, though.