4 ms·
Disclaimer: I'm very involved in the kernel part of this for $company. The RHEL kernels themselves do see many improvements over time, the code that you'll see
by worthless-trash 1y ago
Disclaimer: I'm very involved in the kernel part of this for $company.
The RHEL kernels themselves do see many improvements over time, the code that you'll see when the product goes end of life is considerably updated compared to the original version string that you see in the package name / uname -a. There are customer and partner feature requests, cve fixes and general bug fixes that go in almost every day.
The first problem of 'running old kernels' is exacerbated by the kernel version string not matching code reality.
The second probelm is many companies don't start moving to newer rhels when its out, they often stick to current -1, which is a bit of a problem because by the time they roll out a release, n-1 is likely entering its first stage of "maintenance" so fixes are more difficult to include. If you can think of a solution to this, I'm all ears.
The original reason behind not continually shipping newer kernel versions is to ensure stability by providing a stable whitelisted kABI that third party vendors can build on top of. This is not something that upstream and many OS vendors support, but with the "promise" of not breaking kabi, updates should happen smoothly without third party needing to update their drivers.
The kabi maintenance happens behind the scenes while ensuring that CVE fixes and new features are delivered during the relevant stage of the product lifecycle.
The kernel version is usually very close to the previous release, in the case of rhel10 its 6.13 and already with zero day fixes it has parts of newer code backported, tested, etc in the first errata release.
The security landscape is changing, maybe sometime Red Hat Business Unit may wake up and decide to ship a rolling better tested kernel (Red Hat DOES have an internal/tested https://cki-project.gitlab.io/kernel-ark/ https://cki-project.gitlab.io/kernel-ark/ which is functionally this ). Shipping this has the downside is that the third party vendors would not have the same KABI stability guarantees that RHEL currently provides, muddy the waters of rhels value and confuse people on which kernel they should be running.
I believe there are two customer types, ones who would love to see this, and get the newest features for their full lifecycle, and ones who would hate it, because the churn and change would be too much introducing risk and problems for them down the line.
Its hard, and likely impossible to keep everyone happy.
- jabl 1y agoAs I mentioned in another comment on this thread: > As an aside, that kABI guarantee only goes so far. I work in HPC/AI, and the out-of-tree drivers we use like MOFED and Lustre drivers would break with EVERY SINGLE RHEL minor update (like RHEL X.Y -> X.(Y+1) ). Using past form here because I haven't been using RHEL for this purpose for the past ~5 years, so maybe it has changed since although I doubt it. I'm not sure what the underlying problem here is, is the kABI guarantee worthless generally or is it just that MOFED and Lustre drivers need to use features not covered by some kind of "kABI stability guarantee"?
- lustre-fan 1y agoI work on Lustre development. Lustre uses a lot of kernel symbols not covered by the kABI stability guarantee and we already need to maintain configure checks for all of the other kernels (SuSe, Ubuntu, mainline, etc) that don't offer kABI anyway. So in my opinion, it's not worth the effort to adhere to kABI just for RHEL. Especially when RHEL derivatives might not offer the same guarantees. DKMS works well enough, especially for something open source like Lustre. Honestly, I'm not sure who kABI is even designed for. All of the drivers I've interacted with the HPC space (NVIDIA, Lustre, vendor network drivers, etc.) don't seem to adhere to kABI. DKMS is far more standard. I'd be interested to know which vendors are making heavy use of it.
- worthless-trash 1y agoI believe kABI is becoming less and less valuable over time. > Especially when RHEL derivatives might not offer the same guarantees. They do not, as you likely have experienced. > Honestly, I'm not sure who kABI is even designed for. You make it work once, using the accepted kabi's and then not have to worry about updating your drivers (You likely know this). Some customers and systems are very change adverse, almost any change is too much for them.
- worthless-trash 1y agoThe problem is that some vendors don't participate or care about the KABI program. They have their reasons, maybe the cost is too high to maintain RHEL compat and upstream compatability, so they simply choose the one that is the least pain to adhere to when a customer requests a fix. If companies talked to partner engineering about their kABI requirements, I think there would be a lot less breaking however I'm sure that i'm oversimplifying the reason that they cant or wont do this. I completely understand that the work is non-trivial and that they have many environmental pressures that affect their choices. The KABI is the olive leaf, they can take it or not.
- pabs3 1y agoHow much of the RHEL kernels is stuff that isn't in Linux mainline or LTS?
- Conan_Kudo 1y agoPretty much all of it is in mainline modulo the secure boot lockdown patches, which are downstream for all distributions because Linus fundamentally believes those patches do not make sense. Linux longterm often is missing stuff the RHEL kernel has, because RHEL backports subsystems from mainline with features and hardware support.
- znpy 1y agoWhat if, more than a rolling kernel, we get a new kernel every two years or so? Or maybe one in the middle of the (expected) lifetime of the major release ? Just thinking out loud, but I acknowledge that maintaining a kernel version is no small task (probably takes a lot of engineering time)
- worthless-trash 1y ago> Or maybe one in the middle of the (expected) lifetime of the major release ? This idea has been floated internally, thank you voicing this idea from the customers perspective. It seems like the business unit believes the better idea is to just release another version. I'm not sold on more frequent releases, but I'm not in a decision maker in that area.
- znpy 1y agoThank you for listening! I want to reiterate that this is not a super strong pain point for me. Overall i still like RHEL very much!