14 ms·

I have mixed feelings. The modus operandi should be to get upgrades to the current stable kernel frequently enough, not to backport bugfixes to 6 year old kerne

by buster 9y ago

I have mixed feelings. The modus operandi should be to get upgrades to the current stable kernel frequently enough, not to backport bugfixes to 6 year old kernels.

ausjke 9y ago
that is not going to happen in the industry, unfortunately. I worked there for years. the data in the article is probably correct, it takes about two years from the kernel->cpu_vendor->OEM on the development cycle for a stable product even in the fast-paced consumer electronic market, you just can not upgrade the kernel to chase the new releases which could cause instability chaos in the development team and put off the product delivery dates forever.
kuschku 9y ago
As long as Linus refuses to provide a stable driver API, this won’t happen. OEMs will only pay to write custom drivers for a SoC once, and that means you’ll be stuck with one kernel (or any binary-compatible version). Every device requires entirely custom drivers for this, because ARM has no device enumeration. Until Torvalds finally stops being so stubborn on this, Android and IoT devices will have such a short lifetime, and people will have to continue backporting fixes to ancient kernels.
- wbl 9y ago
  Or they could put drivers in that are upstreamable.
  - kuschku 9y ago
    That’s not easily possible – most ARM SoCs have no device enumeration, so you need to hardcode the memory addresses and physical ports of everything. These are different for every single chip. EDIT: yes, some problems are now solved with Device Tree, but from my experience building Android ROMs, that doesn't solve half of the problems (as many SoCs have special custom hardware, too)
    - kelnos 9y ago
      Isn't that what devie-tree and those dts/dtb/dto files starting to appear under the arch/arm/ tree in the kernel source are meant to address?
    - theatrus2 9y ago
      This is what DeviceTree solves. Not using DeviceTree now for SoC drivers is just pure laziness. https://elinux.org/Device_Tree_Reference https://elinux.org/Device_Tree_Reference
      - kuschku 9y ago
        Well, Device Tree solves some of the issues, but sadly not all of them. A more stable ABI would certainly help a lot.
- cmurf 9y ago
  Is it all that unreasonable? I don't see that Google's Zircon kernel has API guarantees either, but as the driver model doesn't require mainlining, it suggests the code can remain under wraps and ostensibly kernel specific.
  - throwawaysml 9y ago
    If Zircon is a proper microkernel system, then it ought to have such guarantees. But it's too early to tell anything about Zircon and at what end of the microkernel spectrum it will land.
- greenhouse_gas 9y ago
  Does FreeBSD or NetBSD have a stable ABI?
  - throwawaysml 9y ago
    FreeBSD has a stable kernel ABI for each release and is allowed to break it in a new version. This allows nVidia to have an easier time providing binary drivers on FreeBSD.
  - artificial 9y ago
    Yes, especially for the y part of X.y. [1] Although since version 7 multiple symbols are supported. [2] If you're interested in what the release engineering team has to say about it this page [3] contains more details. [1] https://wiki.freebsd.org/Releng/ABI https://wiki.freebsd.org/Releng/ABI [2] https://wiki.freebsd.org/BinaryCompatibility https://wiki.freebsd.org/BinaryCompatibility [3] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/article.html https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd...
  - pjmlp 9y ago
    Yes, the *BSDs, all proprietary UNIX variants, Windows, mainframes and pretty most everyone else with exception of Linux.
  - kev009 9y ago
    The kernel ABI is stable between major version numbers. It is never intentionally broken between majors. More importantly, the kernel APIs are very stable, which seems to be intentionally broken on Linux regularly. Userland ABI is much more stringent, pretty much any commercially used OS will maintain compatibility for old applications indefinitely, Linux and Free and Net BSD included.
    - tosstossy 9y ago
      > which seems to be intentionally broken on Linux regularly Please provide evidence to back the claim of intent. This comes across as if you're implying malice, when it's far more likely interfaces change in the natural course of development where there are no restraints like abi/api stability commitments.
- dankohn1 9y ago
  I wrote this explanation for the intentional lack of a stable device driver API a decade ago, and it’s still relevant today: https://www.linuxfoundation.org/publications/the-linux-driver-model-a-better-way-to-support-devices/ https://www.linuxfoundation.org/publications/the-linux-drive...
  - kuschku 9y ago
    That’s an explanation, but doesn’t fix the very real issues. We have a situation today where nvidia works better on Linux than AMD, even though AMD open sourced their entire base driver, and only wants to provide a few modules as proprietary additions (such as an implementation of 3D screenshots as Ansel competitor), because the kernel maintainers refuse to allow any stable ABI in the kernel. And this is just stubborn, and doesn’t help at all. If this mindset continues, we continue to be stuck with drivers being proprietary DKMS modules. Already today I have to build dozens of proprietary DKMS modules into my kernel all the time, and recompilation times are horrible.
    - colemannugent 9y ago
      I think you're confusing performance with ease of use. Top of the line Nvidia hardware beats AMD hardware. As much as I hate to admit it, nvidia's cards perform better across all platforms leading me to believe that they have a real hardware advantage. Nvidia's cards perform better on Linux than AMD, which I would argue is because Nvidia has more money to throw at the problem. If the issue was the non-stable kernel ABI, then Nvidia surely would be affected in exactly the same way. AMD cards have more of the "Just Works" thing going for them that Dan mentioned in his article. In terms of ease of use I can't imagine it getting much easier than the driver you need being mainlined. What I really liked about Dan's article was this bit: In Linux, hardware support only gets better; it never gets worse.
      - kuschku 9y ago
        But that's not the case. AMDGPU with mesa on any distro I've tried leads to crashes in mesa, it leads to OpenGL not working, and nothing running. The mainline driver is useless, the average result is "the color of a television tuned to a dead channel". AMDs AMDGPU-PRO proprietary solution at least works sometimes. In contrast, NVIDIAs proprietary driver always simply works.
nitrogen 9y ago
Integrating a new kernel into an embedded device can be quite time consuming, and new kernels may introduce new bugs. For example, back when I was working with Sheevaplugs, I found that, say, kernel whatever.13 worked really well, but whatever.14 and later would provide significant glitching in USB audio. Each new kernel can require a lot of time configuring for the use case at hand, building, testing, etc. And then if a significant new subsystem is added (systemd-related features, for example) or internal kernel architecture changes, it would be incredibly time consuming to update the rest of the toolchain and on-device software to match the new kernel, and then validate the device performance again. So it is better to have LTS versions with minimal changes. This makes security updates more likely on critical embedded devices, since manufacturers can updare the kernel without being forced to retool a ton of unrelated software.
- ploxiln 9y ago
  This side-steps the question of why these smartphones, with more computing power than desktop computers 15 years ago, can't use new mainline kernels, while desktop computers 15 years ago, and desktop computers today, can. It's few different factors, like: Smartphones are harder to reverse-engineer, because of the smaller parts and greater integration and fewer common interconnects. Smartphones are more diverse, and each SoC "architecture" is significantly changed or replaced after just a year or two. There's never enough time and mindshare for third-parties or the vendor itself to write high-quality base system drivers for these things. There's just crappy vendor "BSP" code that's never good enough to maintain support for multiple generations in the same code tree. It's always "fork and add messy hacks until it works". There's never enough time and motivation to make any of this support code good enough to be reused. Some kernel maintainers have tried for some time to convince "enterprise" linux distros to keep up with newer kernels, and apply more of their testing and validation effort to newer kernels, instead of just maintaining one old kernel on their own for most of a decade. This enables more sharing of this effort between multiple distros and other users of the linux kernel. To some extent they have succeeded. But where they have failed, at least the "enterprise" distros did maintain the old kernels they distribute. But these smartphone SoC vendors don't even do that. They really give zero fucks about software quality. It seems to work, ship it, move on to the next iteration because it has to make it to market in X months to keep up with the competition. At that rate, third-parties can't even come close to keeping up and justifying the investment of effort in reverse-engineering (which has been done for a lot of desktop hardware).
cmurf 9y ago
I agree. But maybe this is what's needed to get the companies that depend on such long term support to actually contribute their on-going patches publicly rather than only internally? Ostensibly any patches found in an LTS kernel, must first be proven in a mainline release. But it'll be interesting to see just exactly how old and crusty these kernels end up becoming.
- jlgaddis 9y ago
  You'd hope so. In reality, though, if they haven't been contributing by this point, they probably aren't going to bother doing it now either.
joeyo 9y ago
> Today, it's on Google to maintain dead kernels that it is > still supporting. The 2016 Google Pixel shipped with > Linux kernel 3.18, which was first released in 2014 and > hit end-of-life in January 2017. This is good. If it is going to take so long to go from kernel to shipped phone, that kernel should be supported for a reasonable amount of time.
- deleted 9y ago
  [deleted]
throwawaysml 9y ago
The constant churn in some of the driver stacks like graphics leaves us with long stretches of serious regressions. So while 4.14 can be perfectly stable in all regards and you might appreciate the new features in general code, it's not unusual that something in your graphics drivers gets broken and stays that way for a year or longer. For example Intel's DRM stack wasn't flawless but didn't have as many regressions before they introduced atomic modesetting in 4.2. From there on it's been hit and miss. 4.9 is fine, 4.12 is fine and 4.13 is pretty bad. I have one machine where I'm forced to stay at 4.1-lts. It's atomic modesetting, fences, etc. that caused many gpu regressions. What I'm getting at is that there's no stable driver API or ABI and we're forced to suffer through regressions caused by development to support new graphics stack features or new GPUs, while one's "old" GPU was fully supported and stable in earlier kernels. With more and of the desktop and applications using the GPU for compositing, it's very easy to hang your Wayland or Xorg session for seconds until the driver recovers. This is on top of the many DMAR issues. I don't like using an LTS kernel but I do need stable drivers because without those I cannot use the computer and neither can industrial embedded machines. I admit that it's highly likely that the high churn in the graphics drivers is the cause and it can be singled out, because I don't remember any regression in other drivers. I say it's time to rethink the development model of Linux kernel drm drivers.
mtgx 9y ago
Tell that to Android chip makers and OEMs. It almost never happens. A chip and device dies with the first and only kernel that were supported on them.
- pjmlp 9y ago
  On the video of the keynote he mentions that even Treble is pretty much WIP, where Google expects cooperation from them, regarding the goals of moving everyone forward.
philips 9y ago
100% agree with this. I started my career backporting fixes to old Kernel's and it is error prone, time intensive, and certainly not catching every bug with security impact. The focus of the industry should move to shipping up to date software in all devices, particularly in this infrastructure level software.
lbenes 9y ago
> The modus operandi should be to get upgrades to the current stable kernel frequently enough How can you expect embedded manufactures to do this when Intel can't even keep it's CPU line stable on recent kernels. I have an Bay Trail tablet that I put Linux on, but was forced to move it back on to the ancient Android kernel that shipped with it because of this bug[1]. And if you think, these bugs only happen to Atom/Celron CPUs, think again. My Intel i3 laptop just started hard freeing with the latest 4.13 Kernel in Arch.[2] As a hobbyist, i can deal, but there is no way you can expect small manufactures to deal with the constant influx of bugs introduced by kernel updates. [1] https://bugzilla.kernel.org/show_bug.cgi?id=109051 https://bugzilla.kernel.org/show_bug.cgi?id=109051 [2] https://bbs.archlinux.org/viewtopic.php?id=230676 https://bbs.archlinux.org/viewtopic.php?id=230676
- ams6110 9y ago
  > there is no way you can expect small manufactures to deal with the constant influx of bugs introduced by kernel updates Well then maybe they should not be in the business they are in. As people so like to point out here, their inability to support the devices they make becomes an externality that all the users end up paying for, when they have to deal with security vulnerabilities that cannot be resolved.

by buster 9y ago

I have mixed feelings. The modus operandi should be to get upgrades to the current stable kernel frequently enough, not to backport bugfixes to 6 year old kernels.

ausjke 9y ago

that is not going to happen in the industry, unfortunately. I worked there for years. the data in the article is probably correct, it takes about two years from the kernel->cpu_vendor->OEM on the development cycle for a stable product even in the fast-paced consumer electronic market, you just can not upgrade the kernel to chase the new releases which could cause instability chaos in the development team and put off the product delivery dates forever.

kuschku 9y ago

As long as Linus refuses to provide a stable driver API, this won’t happen. OEMs will only pay to write custom drivers for a SoC once, and that means you’ll be stuck with one kernel (or any binary-compatible version). Every device requires entirely custom drivers for this, because ARM has no device enumeration. Until Torvalds finally stops being so stubborn on this, Android and IoT devices will have such a short lifetime, and people will have to continue backporting fixes to ancient kernels.

wbl 9y ago
Or they could put drivers in that are upstreamable.
- kuschku 9y ago
  That’s not easily possible – most ARM SoCs have no device enumeration, so you need to hardcode the memory addresses and physical ports of everything. These are different for every single chip. EDIT: yes, some problems are now solved with Device Tree, but from my experience building Android ROMs, that doesn't solve half of the problems (as many SoCs have special custom hardware, too)
  - kelnos 9y ago
    Isn't that what devie-tree and those dts/dtb/dto files starting to appear under the arch/arm/ tree in the kernel source are meant to address?
  - theatrus2 9y ago
    This is what DeviceTree solves. Not using DeviceTree now for SoC drivers is just pure laziness. https://elinux.org/Device_Tree_Reference https://elinux.org/Device_Tree_Reference
    - kuschku 9y ago
      Well, Device Tree solves some of the issues, but sadly not all of them. A more stable ABI would certainly help a lot.
cmurf 9y ago
Is it all that unreasonable? I don't see that Google's Zircon kernel has API guarantees either, but as the driver model doesn't require mainlining, it suggests the code can remain under wraps and ostensibly kernel specific.
- throwawaysml 9y ago
  If Zircon is a proper microkernel system, then it ought to have such guarantees. But it's too early to tell anything about Zircon and at what end of the microkernel spectrum it will land.
greenhouse_gas 9y ago
Does FreeBSD or NetBSD have a stable ABI?
- throwawaysml 9y ago
  FreeBSD has a stable kernel ABI for each release and is allowed to break it in a new version. This allows nVidia to have an easier time providing binary drivers on FreeBSD.
- artificial 9y ago
  Yes, especially for the y part of X.y. [1] Although since version 7 multiple symbols are supported. [2] If you're interested in what the release engineering team has to say about it this page [3] contains more details. [1] https://wiki.freebsd.org/Releng/ABI https://wiki.freebsd.org/Releng/ABI [2] https://wiki.freebsd.org/BinaryCompatibility https://wiki.freebsd.org/BinaryCompatibility [3] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-releng/article.html https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd...
- pjmlp 9y ago
  Yes, the *BSDs, all proprietary UNIX variants, Windows, mainframes and pretty most everyone else with exception of Linux.
- kev009 9y ago
  The kernel ABI is stable between major version numbers. It is never intentionally broken between majors. More importantly, the kernel APIs are very stable, which seems to be intentionally broken on Linux regularly. Userland ABI is much more stringent, pretty much any commercially used OS will maintain compatibility for old applications indefinitely, Linux and Free and Net BSD included.
  - tosstossy 9y ago
    > which seems to be intentionally broken on Linux regularly Please provide evidence to back the claim of intent. This comes across as if you're implying malice, when it's far more likely interfaces change in the natural course of development where there are no restraints like abi/api stability commitments.
dankohn1 9y ago
I wrote this explanation for the intentional lack of a stable device driver API a decade ago, and it’s still relevant today: https://www.linuxfoundation.org/publications/the-linux-driver-model-a-better-way-to-support-devices/ https://www.linuxfoundation.org/publications/the-linux-drive...
- kuschku 9y ago
  That’s an explanation, but doesn’t fix the very real issues. We have a situation today where nvidia works better on Linux than AMD, even though AMD open sourced their entire base driver, and only wants to provide a few modules as proprietary additions (such as an implementation of 3D screenshots as Ansel competitor), because the kernel maintainers refuse to allow any stable ABI in the kernel. And this is just stubborn, and doesn’t help at all. If this mindset continues, we continue to be stuck with drivers being proprietary DKMS modules. Already today I have to build dozens of proprietary DKMS modules into my kernel all the time, and recompilation times are horrible.
  - colemannugent 9y ago
    I think you're confusing performance with ease of use. Top of the line Nvidia hardware beats AMD hardware. As much as I hate to admit it, nvidia's cards perform better across all platforms leading me to believe that they have a real hardware advantage. Nvidia's cards perform better on Linux than AMD, which I would argue is because Nvidia has more money to throw at the problem. If the issue was the non-stable kernel ABI, then Nvidia surely would be affected in exactly the same way. AMD cards have more of the "Just Works" thing going for them that Dan mentioned in his article. In terms of ease of use I can't imagine it getting much easier than the driver you need being mainlined. What I really liked about Dan's article was this bit: In Linux, hardware support only gets better; it never gets worse.
    - kuschku 9y ago
      But that's not the case. AMDGPU with mesa on any distro I've tried leads to crashes in mesa, it leads to OpenGL not working, and nothing running. The mainline driver is useless, the average result is "the color of a television tuned to a dead channel". AMDs AMDGPU-PRO proprietary solution at least works sometimes. In contrast, NVIDIAs proprietary driver always simply works.

nitrogen 9y ago

Integrating a new kernel into an embedded device can be quite time consuming, and new kernels may introduce new bugs. For example, back when I was working with Sheevaplugs, I found that, say, kernel whatever.13 worked really well, but whatever.14 and later would provide significant glitching in USB audio. Each new kernel can require a lot of time configuring for the use case at hand, building, testing, etc. And then if a significant new subsystem is added (systemd-related features, for example) or internal kernel architecture changes, it would be incredibly time consuming to update the rest of the toolchain and on-device software to match the new kernel, and then validate the device performance again. So it is better to have LTS versions with minimal changes. This makes security updates more likely on critical embedded devices, since manufacturers can updare the kernel without being forced to retool a ton of unrelated software.

ploxiln 9y ago
This side-steps the question of why these smartphones, with more computing power than desktop computers 15 years ago, can't use new mainline kernels, while desktop computers 15 years ago, and desktop computers today, can. It's few different factors, like: Smartphones are harder to reverse-engineer, because of the smaller parts and greater integration and fewer common interconnects. Smartphones are more diverse, and each SoC "architecture" is significantly changed or replaced after just a year or two. There's never enough time and mindshare for third-parties or the vendor itself to write high-quality base system drivers for these things. There's just crappy vendor "BSP" code that's never good enough to maintain support for multiple generations in the same code tree. It's always "fork and add messy hacks until it works". There's never enough time and motivation to make any of this support code good enough to be reused. Some kernel maintainers have tried for some time to convince "enterprise" linux distros to keep up with newer kernels, and apply more of their testing and validation effort to newer kernels, instead of just maintaining one old kernel on their own for most of a decade. This enables more sharing of this effort between multiple distros and other users of the linux kernel. To some extent they have succeeded. But where they have failed, at least the "enterprise" distros did maintain the old kernels they distribute. But these smartphone SoC vendors don't even do that. They really give zero fucks about software quality. It seems to work, ship it, move on to the next iteration because it has to make it to market in X months to keep up with the competition. At that rate, third-parties can't even come close to keeping up and justifying the investment of effort in reverse-engineering (which has been done for a lot of desktop hardware).

cmurf 9y ago

I agree. But maybe this is what's needed to get the companies that depend on such long term support to actually contribute their on-going patches publicly rather than only internally? Ostensibly any patches found in an LTS kernel, must first be proven in a mainline release. But it'll be interesting to see just exactly how old and crusty these kernels end up becoming.

jlgaddis 9y ago
You'd hope so. In reality, though, if they haven't been contributing by this point, they probably aren't going to bother doing it now either.

joeyo 9y ago

> Today, it's on Google to maintain dead kernels that it is > still supporting. The 2016 Google Pixel shipped with > Linux kernel 3.18, which was first released in 2014 and > hit end-of-life in January 2017. This is good. If it is going to take so long to go from kernel to shipped phone, that kernel should be supported for a reasonable amount of time.

deleted 9y ago
[deleted]

throwawaysml 9y ago

The constant churn in some of the driver stacks like graphics leaves us with long stretches of serious regressions. So while 4.14 can be perfectly stable in all regards and you might appreciate the new features in general code, it's not unusual that something in your graphics drivers gets broken and stays that way for a year or longer. For example Intel's DRM stack wasn't flawless but didn't have as many regressions before they introduced atomic modesetting in 4.2. From there on it's been hit and miss. 4.9 is fine, 4.12 is fine and 4.13 is pretty bad. I have one machine where I'm forced to stay at 4.1-lts. It's atomic modesetting, fences, etc. that caused many gpu regressions. What I'm getting at is that there's no stable driver API or ABI and we're forced to suffer through regressions caused by development to support new graphics stack features or new GPUs, while one's "old" GPU was fully supported and stable in earlier kernels. With more and of the desktop and applications using the GPU for compositing, it's very easy to hang your Wayland or Xorg session for seconds until the driver recovers. This is on top of the many DMAR issues. I don't like using an LTS kernel but I do need stable drivers because without those I cannot use the computer and neither can industrial embedded machines. I admit that it's highly likely that the high churn in the graphics drivers is the cause and it can be singled out, because I don't remember any regression in other drivers. I say it's time to rethink the development model of Linux kernel drm drivers.

mtgx 9y ago

Tell that to Android chip makers and OEMs. It almost never happens. A chip and device dies with the first and only kernel that were supported on them.

pjmlp 9y ago
On the video of the keynote he mentions that even Treble is pretty much WIP, where Google expects cooperation from them, regarding the goals of moving everyone forward.

philips 9y ago

100% agree with this. I started my career backporting fixes to old Kernel's and it is error prone, time intensive, and certainly not catching every bug with security impact. The focus of the industry should move to shipping up to date software in all devices, particularly in this infrastructure level software.

lbenes 9y ago

> The modus operandi should be to get upgrades to the current stable kernel frequently enough How can you expect embedded manufactures to do this when Intel can't even keep it's CPU line stable on recent kernels. I have an Bay Trail tablet that I put Linux on, but was forced to move it back on to the ancient Android kernel that shipped with it because of this bug[1]. And if you think, these bugs only happen to Atom/Celron CPUs, think again. My Intel i3 laptop just started hard freeing with the latest 4.13 Kernel in Arch.[2] As a hobbyist, i can deal, but there is no way you can expect small manufactures to deal with the constant influx of bugs introduced by kernel updates. [1] https://bugzilla.kernel.org/show_bug.cgi?id=109051 https://bugzilla.kernel.org/show_bug.cgi?id=109051 [2] https://bbs.archlinux.org/viewtopic.php?id=230676 https://bbs.archlinux.org/viewtopic.php?id=230676

ams6110 9y ago
> there is no way you can expect small manufactures to deal with the constant influx of bugs introduced by kernel updates Well then maybe they should not be in the business they are in. As people so like to point out here, their inability to support the devices they make becomes an externality that all the users end up paying for, when they have to deal with security vulnerabilities that cannot be resolved.