16 ms·

RHEL has a major ecosystem advantage related to drivers that they may not be aware of. They risk ruining this, as I will try to explain. At the time industry s

RHEL has a major ecosystem advantage related to drivers that they may not be aware of. They risk ruining this, as I will try to explain. At the time industry started to take Linux seriously, RHEL was the dominant distro. As a result, and by accident, RHEL+derivs became the primary target for commercial hardware drivers. As an example - it's easier to get obscure low-latency network and packet-capture cards working on RHEL+derivs than on other systems. RHEL+derivs is the assumed standard when you talk to firms that make that kind of stuff. As a result of driver support, redhat derivs became the standard for commercial deployment. Redhat did not get this volume. The deployed volume is with all the Centos/Rocky/Alma that is deployed in hedge funds, prop firms, oil search grids, etc. As I see it, this driver situation is the real value in the Redhat ecosystem. The userland is a bit of a mess, the package manager does not impress me. But I deploy RHEL-derivs because drivers work with no hassle. If Redhat were able to cancel the derivatives, they would trigger the tipping point where industry ceases to treat RHEL+derivs as the standard driver target. Firms using centos on five thousand servers are not suddenly going to start paying USD 350 / year / server where previously they paid nothing. They will move to Debian, and eat the pain of the transition. Commercial driver culture would swiftly follow. That culture change would not take years. It would take days. I think Redhat's per-server sales model is naive. USD 350 / year gets you a license with no support. This probably gets some commerce from small-business, and a bit more from firms with traditional service expectations. But is that really where the opportunity is? When I set up a data-centre presence, I want to PXE-boot each server. This way I can modify the system image for the grid by creating a new PXE image and rebooting hosts. Getting to this setup is fiddly. I wish there was an off-the-shelf solution from Redhat that did a good job of it. USD 2000 per year per site per 100 servers, including support for the PXE device itself. Redhat could coexist with the derivs if it took this path. If they built this as an appliance, that product could become ubiquitous like firewalls and network switches.

nine_k 3y ago
If you don't mind, what makes more niche hardware drivers work / build better against RHEL? I suppose RHEL uses basically the same kernel, with patches that don't alter its interface too substantially. So I presume that a driver in source form, or even partly in binary blob form, should build and work approximately equally well with any stock kernel. Beside the driver developers apparently using RHEL / CentOS (so on them the drivers are known to work), what am I missing?
- pak9rabid 3y ago
  In my past experience as a sysadmin, some vendors will only release drivers (as binary blobs) for RHEL. EMC PowerPath comes to mind, granted that's kind of a moot point now that the FOSS kernel MPIO drivers are good enough, but I'm sure there's other such examples that exist to this day.
- cturner 3y ago
  It is both drivers and the userland tools that you get in the tarball. This stuff is often developed by people with a focus on hardware. They will build something to work well on their machine but may never have used a different linux distro. The userland tools might assume compile options in libraries will be as they are in redhat. There will be hard coded paths where there shouldn’t be. The source may not be available, it might be static binaries. That kind of stuff.
- yjftsjthsd-h 3y ago
  > I suppose RHEL uses basically the same kernel, with patches that don't alter its interface too substantially. So I presume that a driver in source form, or even partly in binary blob form, should build and work approximately equally well with any stock kernel. No, I think that's exactly the difference; Linux (in)famously has no stable ABI for drivers, and regularly makes changes that break things if you don't recompile against the updated kernel. Part of the value in RHEL was that they froze significantly more ABI surface, which made it possible to write proprietary drivers that would work across at least minor kernel updates without needing to be changed. That won't work on an upstream kernel because upstream doesn't go out of their way to freeze that ABI surface.
  - adrian_b 3y ago
    Yes, exactly this. Maintaining any kernel modules that are not in the Linux kernel tree is extremely painful. Almost every new kernel version breaks the older kernel modules, e.g. device drivers, by moving definitions between kernel headers, by adding or deleting function parameters, or by adding or deleting structure members. Most of these changes are very poorly documented, so anyone who does not follow daily the kernel development is clueless for instance about which values should be put in the new function parameters or new structure members in order to obtain the same behavior as in the previous kernel version. The kernel developers who make these breaking changes do not bother to write upgrading instructions for the benefit of those who must maintain an out-of-tree device driver, so those may have to waste a lot of time with reading the kernel sources, to discover what must be done to make the old device drivers compatible with a new kernel.

nine_k 3y ago

If you don't mind, what makes more niche hardware drivers work / build better against RHEL? I suppose RHEL uses basically the same kernel, with patches that don't alter its interface too substantially. So I presume that a driver in source form, or even partly in binary blob form, should build and work approximately equally well with any stock kernel. Beside the driver developers apparently using RHEL / CentOS (so on them the drivers are known to work), what am I missing?

pak9rabid 3y ago
In my past experience as a sysadmin, some vendors will only release drivers (as binary blobs) for RHEL. EMC PowerPath comes to mind, granted that's kind of a moot point now that the FOSS kernel MPIO drivers are good enough, but I'm sure there's other such examples that exist to this day.
cturner 3y ago
It is both drivers and the userland tools that you get in the tarball. This stuff is often developed by people with a focus on hardware. They will build something to work well on their machine but may never have used a different linux distro. The userland tools might assume compile options in libraries will be as they are in redhat. There will be hard coded paths where there shouldn’t be. The source may not be available, it might be static binaries. That kind of stuff.
yjftsjthsd-h 3y ago
> I suppose RHEL uses basically the same kernel, with patches that don't alter its interface too substantially. So I presume that a driver in source form, or even partly in binary blob form, should build and work approximately equally well with any stock kernel. No, I think that's exactly the difference; Linux (in)famously has no stable ABI for drivers, and regularly makes changes that break things if you don't recompile against the updated kernel. Part of the value in RHEL was that they froze significantly more ABI surface, which made it possible to write proprietary drivers that would work across at least minor kernel updates without needing to be changed. That won't work on an upstream kernel because upstream doesn't go out of their way to freeze that ABI surface.
- adrian_b 3y ago
  Yes, exactly this. Maintaining any kernel modules that are not in the Linux kernel tree is extremely painful. Almost every new kernel version breaks the older kernel modules, e.g. device drivers, by moving definitions between kernel headers, by adding or deleting function parameters, or by adding or deleting structure members. Most of these changes are very poorly documented, so anyone who does not follow daily the kernel development is clueless for instance about which values should be put in the new function parameters or new structure members in order to obtain the same behavior as in the previous kernel version. The kernel developers who make these breaking changes do not bother to write upgrading instructions for the benefit of those who must maintain an out-of-tree device driver, so those may have to waste a lot of time with reading the kernel sources, to discover what must be done to make the old device drivers compatible with a new kernel.