5 ms·

Six Great Features with the Upcoming Linux 6.6 Kernel

deleted 3y ago
[deleted]
ahmedfromtunis 3y ago
Off topic, but I had this question for years now: why does the Unix kernel (try to) contain the drivers to every and each device? Wouldn't it be cleaner, easier and simpler to keep those separate and add them later as "modules"? Isn't this how we're supposed to write other types of software? Why is this case different?
- woodruffw 3y ago
  > Wouldn't it be cleaner, easier and simpler to keep those separate and add them later as "modules"? That is exactly what Linux does. Part of building a Linux distribution is determining which modules to ship by default to your users. To the best of my knowledge, the overwhelming majority of drivers are not installed by default, and the average user will never need them.
  - MaxBarraclough 3y ago
    I think that by 'module' they were thinking of user-space drivers rather than kernel modules.
    - IshKebab 3y ago
      No, I think they were imagining kernel drivers but distributed separately and interacting with the kernel through a stable ABI.
- chownie 3y ago
  There's upsides and downsides to both approaches. I think that Linux kernel maintainers continue this approach because this forces driver maintainers to meet a minimum level of effort to get their work merged.
  - joecool1029 3y ago
    Theoretically the monolithic approach is supposed to be more performant. Abstracting away everything into modules can be more stable, but also usually incurs some overhead. But this is holy war territory and I'm trying to ELI5 best as I can.
  - rwmj 3y ago
    One serious downside (for end users) of having a stable driver ABI would be that manufacturers would release a binary blob driver for their hardware which just about works, but no bug could ever be fixed. Manufacturers wouldn't have any incentive to fix bugs and kernel developers couldn't do it either. The situation of GPL'd drivers in a kernel with an unstable ABI wasn't really planned, but for users it's probably the best outcome.
    - nuancebydefault 3y ago
      I always thought that ABI was the interface between CPU and executable code, ie the spec of the assembly language for a CPU architecture: aarch64, x86 etc. I now would guess it is also a term used for the interface between libraries (kernel/driver) as long as they are binary (without the source code or headers) ?
      - gdprrrr 3y ago
        It's also used for programming languages like Rust and Haskell. The Rust compiler not having a stable ABI means the way the compiler stores information about function signatures, debug symbols etc in libraries may change between compiler versions, so you have to recompile all libraries.
      - rwmj 3y ago
        ABI is just the name for any binary interface between parts of a program, eg in C there is a well-defined and very long term stable ABI for calling and returning from functions: https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/master/raw/x86-64-ABI/abi.pdf?job=build https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/ma... https://www.agner.org/optimize/calling_conventions.pdf https://www.agner.org/optimize/calling_conventions.pdf If you combine that with a stable set of C function names and parameters, then you could define an ABI for kernel device drivers if you (and the Linux developers) wanted. While it sounds like a good idea, the outcome probably wouldn't be great for users. Stable ABIs work in other places. For nbdkit we defined a stable ABI for plugins and have maintained it for over 10 years, so a binary plugin written for 10 years old nbdkit can be loaded by modern nbdkit just fine (and we test this too). This was done to encourage proprietary modules, because that helps with our wider goal to make all block devices and formats visible through an open, interoperable protocol (ie. NBD). Implementing this wasn't very easy, since we also want to allow the plugin interface to evolve. It involves a bunch of C macros and careful expansion of C structs so earlier fields remain compatible while later fields add the new features. https://libguestfs.org/nbdkit-plugin.3.html#API-and-ABI-guarantee-for-C-plugins https://libguestfs.org/nbdkit-plugin.3.html#API-and-ABI-guar... https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0acb2ef8887709c47d3cc/include/nbdkit-plugin.h#L44-L153 https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0...
- joecool1029 3y ago
  You're literally describing something like macOS/iOS which is based around a microkernel (XNU, based on Mach). See also QNX for a non-unix OS that does this.
  - speed_spread 3y ago
    Err.. loading drivers dynamically according to a stable ABI doesn't make an OS a microkernel. If the drivers all share the same address space that's a pretty conventional OS. For a microkernel you need to have separate processes for each driver. It's more robust but there's more IPC overhead.
- danielklnstein 3y ago
  As another comment explains - the vast majority of drivers are not compiled as part of a standard Linux distribution. If you're asking why so many drivers are included in Linux's mainline repository and not maintained off-tree - the major advantage of being upstreamed is that changes to the kernel have to take your driver into account as well. e.g. anyone who changes any APIs will have to update your driver as well. Moreover, being upstreamed certifies that your driver has gone through a non-trivial review process to be merged into the mainline and is an indication of quality/stability.
- adrian_b 3y ago
  Because the Linux kernel does not have a stable API for modules. The modules included in the kernel are updated whenever the API changes. For the external modules, every time when a new kernel version is released, they may become broken and there is a lot of work to identify how to update the modules, unless one watches the kernel mail lists every day, because there is also no documentation about how to migrate the old modules to the new kernel. The reasons for module breakage may be just the movement of some definitions from one header to another, which break compiling, but most frequently some structures gain new members or lose old members, or some functions gain new arguments or lose old arguments. When there are new members or arguments it is hard to discover what values should be put in them, while when members or arguments are deleted it is hard to discover whether their absence must be compensated somehow, e.g. by inserting invocations to other functions. In the worst case everything can be solved by reading the kernel source, but that takes a lot of time and those who maintain out-of-kernel modules usually do not do this as a full-time job, so they do not have time to scan every day the kernel mail lists, to see if anyone has plans to make changes that will break their modules.
  - seanwilson 3y ago
    How does this manage to scale? With even a modest number of drivers, you'd think the maintenance burden would be unreasonably high in terms of needing to have someone available who understands how a set of drivers work, knows how to test them, and keeps them up-to-date.
    - adrian_b 3y ago
      That is why everybody attempts to push their modules into the kernel source. Those who cannot do that because their modules are not open source, like NVIDIA, have a lot of work to do for module maintenance. Nevertheless, there are also open-source modules that cannot be pushed into the kernel because they are for uncommon hardware with few users, and those also require a lot of work for maintenance, so they are frequently abandoned and no longer brought up-to-date, to be compatible with the latest kernels.
      - 3y ago
- maccam94 3y ago
  The Linux kernel doesn't have a stable ABI. Thus, if a kernel function signature changes, or a subsystem gets refactored, etc, drivers get updated as part of the process. If the drivers lived outside the kernel tree, they would have to be updated separately by their own maintainers. That's less efficient and prone to breakage, so generally driver modules are merged into the kernel tree. Often they can even share code with other hardware devices!
  - adrian_b 3y ago
    Not only there is no stable ABI, but there is no stable API. When only the ABI changes, a recompilation of all sources is enough. When the API changes, then programmers must manually edit the sources and change function invocations and data structures, to match the new API.
- phendrenad2 3y ago
  Real answer: Because Linux pretty much follows the Unix architecture, and Unix comes from an era where operating system releases were not created for a class of machines, but a specific machine. This design has worked reasonably well on the PC architecture (a class of machines) too, and the pain of adding a stable driver API outweighs the pain of having broken drivers for longer on average.
Jeff_Brown 3y ago
They need to put more ads on that page. I managed to see some of the text.
mnd999 3y ago
I’m guessing they broke zfs again? Just because they can.

deleted 3y ago

[deleted]

Off topic, but I had this question for years now: why does the Unix kernel (try to) contain the drivers to every and each device? Wouldn't it be cleaner, easier and simpler to keep those separate and add them later as "modules"? Isn't this how we're supposed to write other types of software? Why is this case different?

woodruffw 3y ago
> Wouldn't it be cleaner, easier and simpler to keep those separate and add them later as "modules"? That is exactly what Linux does. Part of building a Linux distribution is determining which modules to ship by default to your users. To the best of my knowledge, the overwhelming majority of drivers are not installed by default, and the average user will never need them.
- MaxBarraclough 3y ago
  I think that by 'module' they were thinking of user-space drivers rather than kernel modules.
  - IshKebab 3y ago
    No, I think they were imagining kernel drivers but distributed separately and interacting with the kernel through a stable ABI.
chownie 3y ago
There's upsides and downsides to both approaches. I think that Linux kernel maintainers continue this approach because this forces driver maintainers to meet a minimum level of effort to get their work merged.
- joecool1029 3y ago
  Theoretically the monolithic approach is supposed to be more performant. Abstracting away everything into modules can be more stable, but also usually incurs some overhead. But this is holy war territory and I'm trying to ELI5 best as I can.
- rwmj 3y ago
  One serious downside (for end users) of having a stable driver ABI would be that manufacturers would release a binary blob driver for their hardware which just about works, but no bug could ever be fixed. Manufacturers wouldn't have any incentive to fix bugs and kernel developers couldn't do it either. The situation of GPL'd drivers in a kernel with an unstable ABI wasn't really planned, but for users it's probably the best outcome.
  - nuancebydefault 3y ago
    I always thought that ABI was the interface between CPU and executable code, ie the spec of the assembly language for a CPU architecture: aarch64, x86 etc. I now would guess it is also a term used for the interface between libraries (kernel/driver) as long as they are binary (without the source code or headers) ?
    - gdprrrr 3y ago
      It's also used for programming languages like Rust and Haskell. The Rust compiler not having a stable ABI means the way the compiler stores information about function signatures, debug symbols etc in libraries may change between compiler versions, so you have to recompile all libraries.
    - rwmj 3y ago
      ABI is just the name for any binary interface between parts of a program, eg in C there is a well-defined and very long term stable ABI for calling and returning from functions: https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/master/raw/x86-64-ABI/abi.pdf?job=build https://gitlab.com/x86-psABIs/x86-64-ABI/-/jobs/artifacts/ma... https://www.agner.org/optimize/calling_conventions.pdf https://www.agner.org/optimize/calling_conventions.pdf If you combine that with a stable set of C function names and parameters, then you could define an ABI for kernel device drivers if you (and the Linux developers) wanted. While it sounds like a good idea, the outcome probably wouldn't be great for users. Stable ABIs work in other places. For nbdkit we defined a stable ABI for plugins and have maintained it for over 10 years, so a binary plugin written for 10 years old nbdkit can be loaded by modern nbdkit just fine (and we test this too). This was done to encourage proprietary modules, because that helps with our wider goal to make all block devices and formats visible through an open, interoperable protocol (ie. NBD). Implementing this wasn't very easy, since we also want to allow the plugin interface to evolve. It involves a bunch of C macros and careful expansion of C structs so earlier fields remain compatible while later fields add the new features. https://libguestfs.org/nbdkit-plugin.3.html#API-and-ABI-guarantee-for-C-plugins https://libguestfs.org/nbdkit-plugin.3.html#API-and-ABI-guar... https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0acb2ef8887709c47d3cc/include/nbdkit-plugin.h#L44-L153 https://gitlab.com/nbdkit/nbdkit/-/blob/9f96d53dbbbf67ae5ed0...
joecool1029 3y ago
You're literally describing something like macOS/iOS which is based around a microkernel (XNU, based on Mach). See also QNX for a non-unix OS that does this.
- speed_spread 3y ago
  Err.. loading drivers dynamically according to a stable ABI doesn't make an OS a microkernel. If the drivers all share the same address space that's a pretty conventional OS. For a microkernel you need to have separate processes for each driver. It's more robust but there's more IPC overhead.
danielklnstein 3y ago
As another comment explains - the vast majority of drivers are not compiled as part of a standard Linux distribution. If you're asking why so many drivers are included in Linux's mainline repository and not maintained off-tree - the major advantage of being upstreamed is that changes to the kernel have to take your driver into account as well. e.g. anyone who changes any APIs will have to update your driver as well. Moreover, being upstreamed certifies that your driver has gone through a non-trivial review process to be merged into the mainline and is an indication of quality/stability.
adrian_b 3y ago
Because the Linux kernel does not have a stable API for modules. The modules included in the kernel are updated whenever the API changes. For the external modules, every time when a new kernel version is released, they may become broken and there is a lot of work to identify how to update the modules, unless one watches the kernel mail lists every day, because there is also no documentation about how to migrate the old modules to the new kernel. The reasons for module breakage may be just the movement of some definitions from one header to another, which break compiling, but most frequently some structures gain new members or lose old members, or some functions gain new arguments or lose old arguments. When there are new members or arguments it is hard to discover what values should be put in them, while when members or arguments are deleted it is hard to discover whether their absence must be compensated somehow, e.g. by inserting invocations to other functions. In the worst case everything can be solved by reading the kernel source, but that takes a lot of time and those who maintain out-of-kernel modules usually do not do this as a full-time job, so they do not have time to scan every day the kernel mail lists, to see if anyone has plans to make changes that will break their modules.
- seanwilson 3y ago
  How does this manage to scale? With even a modest number of drivers, you'd think the maintenance burden would be unreasonably high in terms of needing to have someone available who understands how a set of drivers work, knows how to test them, and keeps them up-to-date.
  - adrian_b 3y ago
    That is why everybody attempts to push their modules into the kernel source. Those who cannot do that because their modules are not open source, like NVIDIA, have a lot of work to do for module maintenance. Nevertheless, there are also open-source modules that cannot be pushed into the kernel because they are for uncommon hardware with few users, and those also require a lot of work for maintenance, so they are frequently abandoned and no longer brought up-to-date, to be compatible with the latest kernels.
    - 3y ago
maccam94 3y ago
The Linux kernel doesn't have a stable ABI. Thus, if a kernel function signature changes, or a subsystem gets refactored, etc, drivers get updated as part of the process. If the drivers lived outside the kernel tree, they would have to be updated separately by their own maintainers. That's less efficient and prone to breakage, so generally driver modules are merged into the kernel tree. Often they can even share code with other hardware devices!
- adrian_b 3y ago
  Not only there is no stable ABI, but there is no stable API. When only the ABI changes, a recompilation of all sources is enough. When the API changes, then programmers must manually edit the sources and change function invocations and data structures, to match the new API.
phendrenad2 3y ago
Real answer: Because Linux pretty much follows the Unix architecture, and Unix comes from an era where operating system releases were not created for a class of machines, but a specific machine. This design has worked reasonably well on the PC architecture (a class of machines) too, and the pain of adding a stable driver API outweighs the pain of having broken drivers for longer on average.

Jeff_Brown 3y ago

They need to put more ads on that page. I managed to see some of the text.

mnd999 3y ago

I’m guessing they broke zfs again? Just because they can.