4 ms·

Hi! I'm sorry this has been your experience. I'm one of the Red Hatters who's been working behind the scenes to get this over the finish line. I do say my genu

Hi! I'm sorry this has been your experience. I'm one of the Red Hatters who's been working behind the scenes to get this over the finish line. I do say my genuine thanks for your earnest expression. The version and ABI guarantee is not for everyone. At the same time some folks around these parts know that I'm "not an apologist for running an out of date kernel". I can assure you that everything shipped in the forthcoming P550 image is fresh. GCC 15. LLVM 19, etc. It's intended for development to get more software over the finish line for RISC-V. Conflict of interest statement: I work for Red Hat (Formerly CoreOS), and I'm also the working group lead for the distro integration group within RISE (RISC-V Software Ecosystem).

jabl 1y ago
> The version and ABI guarantee is not for everyone. As an aside, that kABI guarantee only goes so far. I work in HPC/AI, and the out-of-tree drivers we use like MOFED and Lustre drivers would break with EVERY SINGLE RHEL minor update (like RHEL X.Y -> X.(Y+1) ). Using past form here because I haven't been using RHEL for this purpose for the past ~5 years, so maybe it has changed since although I doubt it.
- pabs3 1y ago
  Is anyone trying to get those drivers upstreamed?
  - globalc 1y ago
    That, and if not possible one can try to get the used kABI symbols graylisted at Red Hat, to get informed when they change.
  - jabl 1y ago
    Yes, and yes. The kernel parts of MOFED are largely backports of the latest and greatest upstream kernel drivers to the various distro kernels their customers actually run. (The non-kernel parts of MOFED is mostly open source but does contain some proprietary special sauce on top, like IIRC SHARP support isn't available in FOSS.). The HPC community does tend to want to use the latest RDMA drivers as those are critical for at scale performance. For Lustre, the client driver was upstreamed into staging, where it sat AFAIU largely unused for a few years until it was ripped out again. The problem was that Lustre developers didn't adopt an upstream-first development approach, and thus the in-kernel driver was basically a throw over the fence fork that nobody cared about. I think there is an effort to try again and hopefully adopt an upstream-first approach, remains to be seen whether it'll succeed.
    - pabs3 1y ago
      For MOFED, why not just wholesale use a newer Linux kernel version?
      - jabl 1y ago
        Perhaps the cure is worse than the disease? There are several reasons to stay with a distro kernel: - Lustre releases target distro kernels, upstream would likely break. - Distro stays on top of CVE's etc. and provide updates when needed. - HW likely certified for a few supported distros only, use anything else and you're on your own.
  - aragilar 1y ago
    Lustre got dropped from the kernel, so it seems unlikely?
  - lustre-fan 1y ago
    [dead]
- kj4ips 1y ago
  In that boat now, weak-modules means you sometimes get lucky, and can reuse. However, since it's more effort to determine if a rebuild is needed than just slap the "build all the vendor kmods and slurm" button, we tend to build for each kernel. IIRC el8 added kernel symbol signature hashes as Provide/Requires, with automation to extract them at build time, so kmods got a lot easier to deal with.
  - jabl 1y ago
    Sorry, I was being imprecise. Rebuilds per se are no problem, as both MOFED and Lustre provide sources, and DKMS nicely automates rebuilding when installing a new kernel image. The actual problem is that RHEL minor releases would also break kernel internal API's, and thus building the kernel modules would fail.

jabl 1y ago

> The version and ABI guarantee is not for everyone. As an aside, that kABI guarantee only goes so far. I work in HPC/AI, and the out-of-tree drivers we use like MOFED and Lustre drivers would break with EVERY SINGLE RHEL minor update (like RHEL X.Y -> X.(Y+1) ). Using past form here because I haven't been using RHEL for this purpose for the past ~5 years, so maybe it has changed since although I doubt it.

pabs3 1y ago
Is anyone trying to get those drivers upstreamed?
- globalc 1y ago
  That, and if not possible one can try to get the used kABI symbols graylisted at Red Hat, to get informed when they change.
- jabl 1y ago
  Yes, and yes. The kernel parts of MOFED are largely backports of the latest and greatest upstream kernel drivers to the various distro kernels their customers actually run. (The non-kernel parts of MOFED is mostly open source but does contain some proprietary special sauce on top, like IIRC SHARP support isn't available in FOSS.). The HPC community does tend to want to use the latest RDMA drivers as those are critical for at scale performance. For Lustre, the client driver was upstreamed into staging, where it sat AFAIU largely unused for a few years until it was ripped out again. The problem was that Lustre developers didn't adopt an upstream-first development approach, and thus the in-kernel driver was basically a throw over the fence fork that nobody cared about. I think there is an effort to try again and hopefully adopt an upstream-first approach, remains to be seen whether it'll succeed.
  - pabs3 1y ago
    For MOFED, why not just wholesale use a newer Linux kernel version?
    - jabl 1y ago
      Perhaps the cure is worse than the disease? There are several reasons to stay with a distro kernel: - Lustre releases target distro kernels, upstream would likely break. - Distro stays on top of CVE's etc. and provide updates when needed. - HW likely certified for a few supported distros only, use anything else and you're on your own.
- aragilar 1y ago
  Lustre got dropped from the kernel, so it seems unlikely?
- lustre-fan 1y ago
  [dead]
kj4ips 1y ago
In that boat now, weak-modules means you sometimes get lucky, and can reuse. However, since it's more effort to determine if a rebuild is needed than just slap the "build all the vendor kmods and slurm" button, we tend to build for each kernel. IIRC el8 added kernel symbol signature hashes as Provide/Requires, with automation to extract them at build time, so kmods got a lot easier to deal with.
- jabl 1y ago
  Sorry, I was being imprecise. Rebuilds per se are no problem, as both MOFED and Lustre provide sources, and DKMS nicely automates rebuilding when installing a new kernel image. The actual problem is that RHEL minor releases would also break kernel internal API's, and thus building the kernel modules would fail.