5 ms·

Does that include Red Hat Enterprise Linux? I can't believe how old of a kernel they use still. I have to use it for work, but I run Arch at home which is runni

Does that include Red Hat Enterprise Linux? I can't believe how old of a kernel they use still. I have to use it for work, but I run Arch at home which is running 4.16.13. I'm honestly surprised that Red Hat can't manage to keep their distros current with stable package builds.

snuxoll 8y ago
Red Hat backports security fixes, hardware support and certain new features to their kernel packages, staying on a stable kernel version throughout the product lifecycle is what allows them to guarantee a consistent kernel ABI that virtually every other distribution throws to the wayside. Personally I prefer not having third-party kernel modules on my servers break every time I run `yum upgrade` like I do with Fedora (oh VMWare, how I hate you so).
- readams 8y ago
  except they still break because Red Hat backports bleeding-edge features but doesn't change the version number.
- vbezhenar 8y ago
  I'm using HP B120i fakeraid controller with proprietary driver and it broke after 7.4 upgrade, so while they probably doing a good job about binary compatibility (7.5 didn't broke it), it's not ideal.
  - snuxoll 8y ago
    Unfortunately the kABI does not encompass every symbol exported by the kernel, there is a whitelist maintained in the kernel-abi-whitelists package and scripts to check conformance of a module to the whitelist. Symbols are only ever added throughout the lifecycle of a RHEL release, so anything that conforms to the 7.3 kABI will also work on 7.4 - but if the module uses a symbol NOT whitelisted in 7.3 there's no guarantee the 7.4 update won't break it. Not that it's much solace when your proprietary blob breaks, but Red Hat does make considerable effort to give vendors a stable target to build against - unfortunately not everyone fully validates conformance which results in the same old problems cropping up from time to time.
    - silly-silly 8y ago
      That and Spectre fixes forced a BUNCH of third party code that relied on non exported code to break.
  - xnyanta 8y ago
    I really hate HP for this stupid fakeraid shit. The regular SATA AHCI controller is unusable on the HP servers I have because it does not pass the temperature info to the iLO and causes the fans to skyrocket to 70% permanently. Obviously, using the B120i was out of question due to their crappy driver only targeting RedHat's ancient kernel so my only viable solution was to have to go out and purchase a 200$ real HP P420 smart array RAID card to connect my drives to.
gicmo 8y ago
RHEL kernels are heavily patched, i.e. stuff gets backported from new kernels, sometimes even from -rc if needed. The version number is therefore not that useful. Some call it a Frankenkernel. :)
- kbenson 8y ago
  The nice thing is that it's all fully documented. If you download the kernel src.rpm[1] and install it, you'll get the kernel RPM spec file in your default RPM build environment paths (or use rpm2cpio $KERNEL_SRPM | cpio -idmv). The way the RPM building works is to take a vanilla kernel and then apply a LOT of individual patch files to it, each with their own entry. Well, at least that's how it was a long time ago when I messed with this stuff more, but it looks like it may be a bit different now (I just did what I proposed), at least with the downstream CentOS kernel. I stopped having to care about this mostly back around the time when Oracle forked RHEL/CentOS into Oracle Linux and RedHat wasn't happy about Oracle piggybacking on their kernel testing for a paid product (as opposed to CentOS). I think maybe RedHat ships the kernel tar mostly pre-patched in-house now. I may have some of the details wrong in that, but it sounds right to me. It's easy to blame Oracle for why we can't have nice things anymore, because usually it's true. :/ Edit: Ah, I found one like what I was talking about[2], it has 5697 patches included! The most recent 5 series kernel (5.11) didn't have a lot of patches, but 5.6 did. There might be some 6.x series ones like that as well, I don't recall. 1: http://vault.centos.org/centos/7/os/Source/SPackages/kernel-3.10.0-862.el7.src.rpm http://vault.centos.org/centos/7/os/Source/SPackages/kernel-... 2: http://vault.centos.org/5.6/os/SRPMS/kernel-2.6.18-238.el5.src.rpm http://vault.centos.org/5.6/os/SRPMS/kernel-2.6.18-238.el5.s...
  - danieldk 8y ago
    I think maybe RedHat ships the kernel tar mostly pre-patched in-house now. Yes, this was in the news a couple of years ago. They do not include separate patches anymore: https://lwn.net/Articles/430098/ https://lwn.net/Articles/430098/ https://www.theregister.co.uk/2011/03/04/red_hat_twarts_oracle_and_novell_with_change_to_source_code_packaging/ https://www.theregister.co.uk/2011/03/04/red_hat_twarts_orac...
  - 8y ago
hinkley 8y ago
Contrast this with the Postgres addon community, which has a culture of pushing changes upstream until they no longer have to maintain a fork of the Postgres code. Build a business on a fork, then compartmentalize it until it can run without modifying the core so you can get off the bugfix treadmill.
- dralley 8y ago
  The changes are all upstream to begin with. It's a fork in the sense that it's heavily diverged from the source of the old kernel, but it's all still backports and not a ton of new development.
  - hinkley 8y ago
    Why does it need to be diverged if there is no new functionality in their version? Because they can?
    - vertex-four 8y ago
      Why do many distros have releases to which only bug and security fixes are applied? Same reason. The divergence is just those fixes.
    - kikoreis 8y ago
      ABI stability to support binary-only loadable modules is probably the most important reason.
pera 8y ago
> RHEL-7 will automatically default to (safe) “eager” floating point register restore on Sandy Bridge and newer Intel processors. AMD processors are not affected. https://access.redhat.com/solutions/3485131 https://access.redhat.com/solutions/3485131
- Twirrim 8y ago
  It looks like for older processors they may be vulnerable? https://access.redhat.com/security/cve/cve-2018-3665 https://access.redhat.com/security/cve/cve-2018-3665 shows there will be updated kernels coming.
4ad 8y ago
> I'm honestly surprised that Red Hat can't manage to keep their distros current with stable package builds. This is very wrong. It would be trivial for Red Hat to update the kernel shipped into RHEL. It's not a question that they "can't keep up". What they are doing is much more difficult and takes much more effort. They are backporting all security fixes, and sometimes other features from newer kernels to older kernels. Why? Because many RHEL customers want older kernels. Unlike Windows and Solaris, Linux doesn't have a stable kernel ABI (by design). Some RHEL customers want to use binary drivers, and they want their drivers to work when they update their system. Some other RHEL customers do not use or need binary drivers, but they do want to avoid possible incompatibility problems introduced by newer kernels. And no, it doesn't matter that the kernel API is very stable. Sometimes even fixing bugs can introduce incompatibility problems in badly written software. Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be. The world is full of such programs, and people who administer such systems are very grateful the behavior of their systems doesn't randomly change across the lifetime of a release. The fact that RHEL freezes the versions of shipped software is its greatest feature. It's not for everyone, but it's for some people, and for those people it is very important.
- cesarb 8y ago
  > Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be. Like changing the scheduler preferences of which process runs first on fork(), the parent or the child, exposed a bug in bash: https://yarchive.net/comp/linux/child-runs-first.html https://yarchive.net/comp/linux/child-runs-first.html
- burb_xry 8y ago
  The problem with that is that I'd argue it is "safer" to run a kernel.org stable kernel rather than RHEL, since it's sheer impossible to keep up with the amount of fixes that eventually land in stable kernels every few weeks. And it is unclear which of the stable fixes are security relevant ones even if they did not get a CVE number and the like. QAing all these is another mammoth task that often requires expertise in several areas. I'm actually glad that (upstream) Linux doesn't have a stable kernel ABI, if you've ever looked into what gross hacks are required in the RHEL kernel only to keep that intact, well, good luck.

snuxoll 8y ago

Red Hat backports security fixes, hardware support and certain new features to their kernel packages, staying on a stable kernel version throughout the product lifecycle is what allows them to guarantee a consistent kernel ABI that virtually every other distribution throws to the wayside. Personally I prefer not having third-party kernel modules on my servers break every time I run `yum upgrade` like I do with Fedora (oh VMWare, how I hate you so).

readams 8y ago
except they still break because Red Hat backports bleeding-edge features but doesn't change the version number.
vbezhenar 8y ago
I'm using HP B120i fakeraid controller with proprietary driver and it broke after 7.4 upgrade, so while they probably doing a good job about binary compatibility (7.5 didn't broke it), it's not ideal.
- snuxoll 8y ago
  Unfortunately the kABI does not encompass every symbol exported by the kernel, there is a whitelist maintained in the kernel-abi-whitelists package and scripts to check conformance of a module to the whitelist. Symbols are only ever added throughout the lifecycle of a RHEL release, so anything that conforms to the 7.3 kABI will also work on 7.4 - but if the module uses a symbol NOT whitelisted in 7.3 there's no guarantee the 7.4 update won't break it. Not that it's much solace when your proprietary blob breaks, but Red Hat does make considerable effort to give vendors a stable target to build against - unfortunately not everyone fully validates conformance which results in the same old problems cropping up from time to time.
  - silly-silly 8y ago
    That and Spectre fixes forced a BUNCH of third party code that relied on non exported code to break.
- xnyanta 8y ago
  I really hate HP for this stupid fakeraid shit. The regular SATA AHCI controller is unusable on the HP servers I have because it does not pass the temperature info to the iLO and causes the fans to skyrocket to 70% permanently. Obviously, using the B120i was out of question due to their crappy driver only targeting RedHat's ancient kernel so my only viable solution was to have to go out and purchase a 200$ real HP P420 smart array RAID card to connect my drives to.

gicmo 8y ago

RHEL kernels are heavily patched, i.e. stuff gets backported from new kernels, sometimes even from -rc if needed. The version number is therefore not that useful. Some call it a Frankenkernel. :)

kbenson 8y ago
The nice thing is that it's all fully documented. If you download the kernel src.rpm[1] and install it, you'll get the kernel RPM spec file in your default RPM build environment paths (or use rpm2cpio $KERNEL_SRPM | cpio -idmv). The way the RPM building works is to take a vanilla kernel and then apply a LOT of individual patch files to it, each with their own entry. Well, at least that's how it was a long time ago when I messed with this stuff more, but it looks like it may be a bit different now (I just did what I proposed), at least with the downstream CentOS kernel. I stopped having to care about this mostly back around the time when Oracle forked RHEL/CentOS into Oracle Linux and RedHat wasn't happy about Oracle piggybacking on their kernel testing for a paid product (as opposed to CentOS). I think maybe RedHat ships the kernel tar mostly pre-patched in-house now. I may have some of the details wrong in that, but it sounds right to me. It's easy to blame Oracle for why we can't have nice things anymore, because usually it's true. :/ Edit: Ah, I found one like what I was talking about[2], it has 5697 patches included! The most recent 5 series kernel (5.11) didn't have a lot of patches, but 5.6 did. There might be some 6.x series ones like that as well, I don't recall. 1: http://vault.centos.org/centos/7/os/Source/SPackages/kernel-3.10.0-862.el7.src.rpm http://vault.centos.org/centos/7/os/Source/SPackages/kernel-... 2: http://vault.centos.org/5.6/os/SRPMS/kernel-2.6.18-238.el5.src.rpm http://vault.centos.org/5.6/os/SRPMS/kernel-2.6.18-238.el5.s...
- danieldk 8y ago
  I think maybe RedHat ships the kernel tar mostly pre-patched in-house now. Yes, this was in the news a couple of years ago. They do not include separate patches anymore: https://lwn.net/Articles/430098/ https://lwn.net/Articles/430098/ https://www.theregister.co.uk/2011/03/04/red_hat_twarts_oracle_and_novell_with_change_to_source_code_packaging/ https://www.theregister.co.uk/2011/03/04/red_hat_twarts_orac...
- 8y ago

hinkley 8y ago

Contrast this with the Postgres addon community, which has a culture of pushing changes upstream until they no longer have to maintain a fork of the Postgres code. Build a business on a fork, then compartmentalize it until it can run without modifying the core so you can get off the bugfix treadmill.

dralley 8y ago
The changes are all upstream to begin with. It's a fork in the sense that it's heavily diverged from the source of the old kernel, but it's all still backports and not a ton of new development.
- hinkley 8y ago
  Why does it need to be diverged if there is no new functionality in their version? Because they can?
  - vertex-four 8y ago
    Why do many distros have releases to which only bug and security fixes are applied? Same reason. The divergence is just those fixes.
  - kikoreis 8y ago
    ABI stability to support binary-only loadable modules is probably the most important reason.

pera 8y ago

> RHEL-7 will automatically default to (safe) “eager” floating point register restore on Sandy Bridge and newer Intel processors. AMD processors are not affected. https://access.redhat.com/solutions/3485131 https://access.redhat.com/solutions/3485131

Twirrim 8y ago
It looks like for older processors they may be vulnerable? https://access.redhat.com/security/cve/cve-2018-3665 https://access.redhat.com/security/cve/cve-2018-3665 shows there will be updated kernels coming.

4ad 8y ago

> I'm honestly surprised that Red Hat can't manage to keep their distros current with stable package builds. This is very wrong. It would be trivial for Red Hat to update the kernel shipped into RHEL. It's not a question that they "can't keep up". What they are doing is much more difficult and takes much more effort. They are backporting all security fixes, and sometimes other features from newer kernels to older kernels. Why? Because many RHEL customers want older kernels. Unlike Windows and Solaris, Linux doesn't have a stable kernel ABI (by design). Some RHEL customers want to use binary drivers, and they want their drivers to work when they update their system. Some other RHEL customers do not use or need binary drivers, but they do want to avoid possible incompatibility problems introduced by newer kernels. And no, it doesn't matter that the kernel API is very stable. Sometimes even fixing bugs can introduce incompatibility problems in badly written software. Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be. The world is full of such programs, and people who administer such systems are very grateful the behavior of their systems doesn't randomly change across the lifetime of a release. The fact that RHEL freezes the versions of shipped software is its greatest feature. It's not for everyone, but it's for some people, and for those people it is very important.

cesarb 8y ago
> Other times something like changing some trivial subtle thing about the scheduler can make a badly written program perform much differently then it used to be. Like changing the scheduler preferences of which process runs first on fork(), the parent or the child, exposed a bug in bash: https://yarchive.net/comp/linux/child-runs-first.html https://yarchive.net/comp/linux/child-runs-first.html
burb_xry 8y ago
The problem with that is that I'd argue it is "safer" to run a kernel.org stable kernel rather than RHEL, since it's sheer impossible to keep up with the amount of fixes that eventually land in stable kernels every few weeks. And it is unclear which of the stable fixes are security relevant ones even if they did not get a CVE number and the like. QAing all these is another mammoth task that often requires expertise in several areas. I'm actually glad that (upstream) Linux doesn't have a stable kernel ABI, if you've ever looked into what gross hacks are required in the RHEL kernel only to keep that intact, well, good luck.