7 ms·

Warning: Debian stable kernel upgrade breaks most ARM SBC

132 points by geppetto 8y ago

sm4rk0 8y ago
This should be made "sticky", if there was this feature @HN. Strange that ci.debian.net didn't catch this.
- deleted 8y ago
  [deleted]
- kfwhp 8y ago
  What % of HN users use an ARM SBC with Debian?
  - blihp 8y ago
    I'm sure it's small, but at least a few of us are here. Also worth noting is that (most) Raspberry Pi (an ARM SBC) users are indirectly Debian users too. (and I'm a Pi user as well)
    - amacbride 8y ago
      My BeagleBone Black runs a Debian-based distro (Angstrom), and there are spins based on Debian proper, Ubuntu, etc. So it’s definitely good to know.
      - cozzyd 8y ago
        The BBB doesn't use an upstream kernel though as far as I can tell.
      - pabs3 8y ago
        Ångström appears to be based on OpenEmbedded rather than Debian: https://en.wikipedia.org/wiki/%C3%85ngstr%C3%B6m_distribution https://en.wikipedia.org/wiki/%C3%85ngstr%C3%B6m_distributio...
    - walrus01 8y ago
      There's just about zero chance that this bug would make it into raspbian stable, though, since they are always at least several months behind debian-stable on AMD64.
      - kingosticks 8y ago
        And you'd really hope they'd actually test it on real hardware. Although I don't know if they have anything automated, that'd be interesting to know.
  - coconut_crab 8y ago
    Not Debian directly but one of their derivative: Armbian, they have great support for Allwinner SoC and it's easy to build your own image.
  - markvdb 8y ago
    We do at https://fosdem.org https://fosdem.org on our custom made video gear. That's 56 boxes in a crucial spot in our operation. We went out of our way to make sure we had plain vanilla mainline Debian stable packaged linux running on the Allwinner A20 based boards that power them, instead of some binary arm kernel with all kinds of issues.
    - AceJohnny2 8y ago
      Was your fleet impacted by this? Edit: oh, FOSDEM was 2 weeks ago, not in the near future, so impact is less
    - BiggusDijkus 8y ago
      OffTopic: At FOSDEM, I was quite curious about those media boxes. Is there any docs to read about them?
      - markvdb 8y ago
        Here's some info at least: * a rough diagram: https://github.com/FOSDEM/video/tree/master/hardware https://github.com/FOSDEM/video/tree/master/hardware * the software running on them: https://github.com/FOSDEM/infrastructure/tree/master/ansible/playbooks/roles/video-box https://github.com/FOSDEM/infrastructure/tree/master/ansible... Your message also served as a reminder that I should really publish the laser cut box cutting and assembly info. I'll get to that eventually.
    - Insanity 8y ago
      Off-topic, but Fosdem is great! Started going about 6ish years ago and always enjoy it! Thanks for the hard work and for providing such a great event for free. And for the cool t-shirts :D
      - markvdb 8y ago
        Thank you for the kind words. Much appreciated!
  - cmcginty 8y ago
    I have a Rock64 NAS box running an RK3328 chip.
  - tlavoie 8y ago
    I do, at least for some personal projects. They're more than capable enough for many tasks, and cheap enough to be willing to just grab another if you have a quick task.
  - rarecoil 8y ago
    I don't run Debian, but I have multiple ARM SBCs that I work with as well.
  - FerretFred 8y ago
    I have 10 Raspberry Pi's with Debian Stretch and Jessie - the Raspbian variant. I updated a "sacrificial" Pi and rebooted without error.. this time! Package: linux-libc-dev Version: 4.9.82-1+deb9u3+rpi1 Priority: optional Section: devel Source: linux-4.9 Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org> Installed-Size: 4,466 kB Provides: linux-kernel-headers Homepage: https://www.kernel.org/ Download-Size: 1,300 kB APT-Manual-Installed: no APT-Sources: http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages Description: Linux support headers for userspace development This package provides userspaces headers from the Linux kernel. These headers are used by the installed headers for GNU libc and other system libraries. edit: formatting
    - deleted 8y ago
      [deleted]
    - kingosticks 8y ago
      As others have mentioned, Raspbian kernel updates are delayed behind the Debian ones. The linked bug report states it was introduced between 4.9.135-1 and 4.9.144-1 (of linux-image-4.9.0).
  - wvh 8y ago
    I do, but I'm OK with being a statistical fluke. I imagine there could be a number of IOT, media or other devices unknowingly running Debian on ARM though; more than we realise.
  - holri 8y ago
    I use 2 Olimex A20 Micro with Debian stable as personal servers (Web, Email, Backup). One of them had already installed the toxic update via unattended-upgrades. Fortunately I did not configure automatic reboot. So I had the chance to read about the issue here on Hacker News and reinstall the old version: $ wget https://snapshot.debian.org/archive/debian/20181028T150508Z/pool/main/l/linux/linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb https://snapshot.debian.org/archive/debian/20181028T150508Z/... $ dpkg -i linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb and set the version on hold: $ apt-mark hold linux-image-4.9.0-8-armmp-lpae
  - geppetto 8y ago
    I'm glad I can. It's a known, trusted distribution I've been using since 2001 on my main PC. It runs my personal web server since 2013 on a Cuibeboard and it works well. I've got a stable system which I'm familiar with, plus automatic updates, no binary blobs, vanilla kernel and all. I can't think of a better solution if the SBC is supported. What would you recommend as an alternative?
- pabs3 8y ago
  As far as I can tell, ci.debian.net doesn't support any of the ARM architectures and does not test suites other than testing and unstable: https://ci.debian.net/data/ https://ci.debian.net/data/
  - pabs3 8y ago
    Also, I think something like LAVA is more suited to kernel/bootloader testing than debci: https://wiki.debian.org/LAVA https://wiki.debian.org/LAVA
  - uasm 8y ago
    Why do they call it "stable" then?
    - zaarn 8y ago
      Because when it breaks, it is broken in a stable way.
      - yebyen 8y ago
        This is actually 100% true. Stable is the culmination of the Debian Release Engineering / Testing Freeze process. In Unstable, packages are promoted from Experimental as I understand it which is a comparatively easy threshold to get past. In Testing, packages are only promoted from Unstable if they 1) don't have any bugs open against them, and 2) there is not a freeze on. 50% of the time (1 year) there is a freeze on. In Freeze, only bug fixes are promoted to Testing. New features have to take a number. When the freeze is over, Testing is cut and becomes the new Stable. I might have described some part of this subtly wrong, but this is the Debian release management process in a nutshell. To read more: https://release.debian.org/ https://release.debian.org/ In short, "Stable" should be read like stable compound, not like we usually interpret it to mean "doesn't crash" in IT. It's a side effect of not changing often, that it probably doesn't crash too much. The crashing bugs wouldn't have made it into testing, or through the testing/freeze process. Hopefully, at least. I personally prefer to run Unstable on my developer machines, because it gets fixes more frequently: > Security updates are made by the maintainer; they may not be effective on all architectures, and may be delayed. Packages uploaded may not meet release standards, but any breakage is expected to be fixed promptly. Updates are made by maintainers.
voltagex_ 8y ago
Oops: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478#72 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478#72 The problem patch is in http://deb.debian.org/debian/pool/main/l/linux/linux_4.9.144-3.debian.tar.xz http://deb.debian.org/debian/pool/main/l/linux/linux_4.9.144... (I think) "Commit 901e325f772f "ARM: bugs: add support for per-processor bug checking" added a new member to struct processor. This structure is always instantiated in built-in code and the new member is only used in built-in code. Therefore we can safely add the new member at the end instead. Move it to the end and hide it from genksyms. Also hide it when building modules, to make sure they really don't use it."
- goodwork 8y ago
  If only Debian had someone like Linus Torvalds, he'd have insulted the fuck out of the developer who introduced this bug. Diabian will never be as good as Linux without Torvalds.
moocowtruck 8y ago
then it's not a debian stable kernel now is it
vectorEQ 8y ago
love this note on it: "Sure, maybe. I've suggested kernelci as a useful thing to help here, but we really need to be testing kernels complete with all the Debian patches to..." so does that mean they test the kernel but not their own patches to it? seems silly :D

132 points by geppetto 8y ago

sm4rk0 8y ago

This should be made "sticky", if there was this feature @HN. Strange that ci.debian.net didn't catch this.

deleted 8y ago
[deleted]
kfwhp 8y ago
What % of HN users use an ARM SBC with Debian?
- blihp 8y ago
  I'm sure it's small, but at least a few of us are here. Also worth noting is that (most) Raspberry Pi (an ARM SBC) users are indirectly Debian users too. (and I'm a Pi user as well)
  - amacbride 8y ago
    My BeagleBone Black runs a Debian-based distro (Angstrom), and there are spins based on Debian proper, Ubuntu, etc. So it’s definitely good to know.
    - cozzyd 8y ago
      The BBB doesn't use an upstream kernel though as far as I can tell.
    - pabs3 8y ago
      Ångström appears to be based on OpenEmbedded rather than Debian: https://en.wikipedia.org/wiki/%C3%85ngstr%C3%B6m_distribution https://en.wikipedia.org/wiki/%C3%85ngstr%C3%B6m_distributio...
  - walrus01 8y ago
    There's just about zero chance that this bug would make it into raspbian stable, though, since they are always at least several months behind debian-stable on AMD64.
    - kingosticks 8y ago
      And you'd really hope they'd actually test it on real hardware. Although I don't know if they have anything automated, that'd be interesting to know.
- coconut_crab 8y ago
  Not Debian directly but one of their derivative: Armbian, they have great support for Allwinner SoC and it's easy to build your own image.
- markvdb 8y ago
  We do at https://fosdem.org https://fosdem.org on our custom made video gear. That's 56 boxes in a crucial spot in our operation. We went out of our way to make sure we had plain vanilla mainline Debian stable packaged linux running on the Allwinner A20 based boards that power them, instead of some binary arm kernel with all kinds of issues.
  - AceJohnny2 8y ago
    Was your fleet impacted by this? Edit: oh, FOSDEM was 2 weeks ago, not in the near future, so impact is less
  - BiggusDijkus 8y ago
    OffTopic: At FOSDEM, I was quite curious about those media boxes. Is there any docs to read about them?
    - markvdb 8y ago
      Here's some info at least: * a rough diagram: https://github.com/FOSDEM/video/tree/master/hardware https://github.com/FOSDEM/video/tree/master/hardware * the software running on them: https://github.com/FOSDEM/infrastructure/tree/master/ansible/playbooks/roles/video-box https://github.com/FOSDEM/infrastructure/tree/master/ansible... Your message also served as a reminder that I should really publish the laser cut box cutting and assembly info. I'll get to that eventually.
  - Insanity 8y ago
    Off-topic, but Fosdem is great! Started going about 6ish years ago and always enjoy it! Thanks for the hard work and for providing such a great event for free. And for the cool t-shirts :D
    - markvdb 8y ago
      Thank you for the kind words. Much appreciated!
- cmcginty 8y ago
  I have a Rock64 NAS box running an RK3328 chip.
- tlavoie 8y ago
  I do, at least for some personal projects. They're more than capable enough for many tasks, and cheap enough to be willing to just grab another if you have a quick task.
- rarecoil 8y ago
  I don't run Debian, but I have multiple ARM SBCs that I work with as well.
- FerretFred 8y ago
  I have 10 Raspberry Pi's with Debian Stretch and Jessie - the Raspbian variant. I updated a "sacrificial" Pi and rebooted without error.. this time! Package: linux-libc-dev Version: 4.9.82-1+deb9u3+rpi1 Priority: optional Section: devel Source: linux-4.9 Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org> Installed-Size: 4,466 kB Provides: linux-kernel-headers Homepage: https://www.kernel.org/ Download-Size: 1,300 kB APT-Manual-Installed: no APT-Sources: http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages Description: Linux support headers for userspace development This package provides userspaces headers from the Linux kernel. These headers are used by the installed headers for GNU libc and other system libraries. edit: formatting
  - deleted 8y ago
    [deleted]
  - kingosticks 8y ago
    As others have mentioned, Raspbian kernel updates are delayed behind the Debian ones. The linked bug report states it was introduced between 4.9.135-1 and 4.9.144-1 (of linux-image-4.9.0).
- wvh 8y ago
  I do, but I'm OK with being a statistical fluke. I imagine there could be a number of IOT, media or other devices unknowingly running Debian on ARM though; more than we realise.
- holri 8y ago
  I use 2 Olimex A20 Micro with Debian stable as personal servers (Web, Email, Backup). One of them had already installed the toxic update via unattended-upgrades. Fortunately I did not configure automatic reboot. So I had the chance to read about the issue here on Hacker News and reinstall the old version: $ wget https://snapshot.debian.org/archive/debian/20181028T150508Z/pool/main/l/linux/linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb https://snapshot.debian.org/archive/debian/20181028T150508Z/... $ dpkg -i linux-image-4.9.0-8-armmp-lpae_4.9.130-2_armhf.deb and set the version on hold: $ apt-mark hold linux-image-4.9.0-8-armmp-lpae
- geppetto 8y ago
  I'm glad I can. It's a known, trusted distribution I've been using since 2001 on my main PC. It runs my personal web server since 2013 on a Cuibeboard and it works well. I've got a stable system which I'm familiar with, plus automatic updates, no binary blobs, vanilla kernel and all. I can't think of a better solution if the SBC is supported. What would you recommend as an alternative?
pabs3 8y ago
As far as I can tell, ci.debian.net doesn't support any of the ARM architectures and does not test suites other than testing and unstable: https://ci.debian.net/data/ https://ci.debian.net/data/
- pabs3 8y ago
  Also, I think something like LAVA is more suited to kernel/bootloader testing than debci: https://wiki.debian.org/LAVA https://wiki.debian.org/LAVA
- uasm 8y ago
  Why do they call it "stable" then?
  - zaarn 8y ago
    Because when it breaks, it is broken in a stable way.
    - yebyen 8y ago
      This is actually 100% true. Stable is the culmination of the Debian Release Engineering / Testing Freeze process. In Unstable, packages are promoted from Experimental as I understand it which is a comparatively easy threshold to get past. In Testing, packages are only promoted from Unstable if they 1) don't have any bugs open against them, and 2) there is not a freeze on. 50% of the time (1 year) there is a freeze on. In Freeze, only bug fixes are promoted to Testing. New features have to take a number. When the freeze is over, Testing is cut and becomes the new Stable. I might have described some part of this subtly wrong, but this is the Debian release management process in a nutshell. To read more: https://release.debian.org/ https://release.debian.org/ In short, "Stable" should be read like stable compound, not like we usually interpret it to mean "doesn't crash" in IT. It's a side effect of not changing often, that it probably doesn't crash too much. The crashing bugs wouldn't have made it into testing, or through the testing/freeze process. Hopefully, at least. I personally prefer to run Unstable on my developer machines, because it gets fixes more frequently: > Security updates are made by the maintainer; they may not be effective on all architectures, and may be delayed. Packages uploaded may not meet release standards, but any breakage is expected to be fixed promptly. Updates are made by maintainers.

voltagex_ 8y ago

Oops: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478#72 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922478#72 The problem patch is in http://deb.debian.org/debian/pool/main/l/linux/linux_4.9.144-3.debian.tar.xz http://deb.debian.org/debian/pool/main/l/linux/linux_4.9.144... (I think) "Commit 901e325f772f "ARM: bugs: add support for per-processor bug checking" added a new member to struct processor. This structure is always instantiated in built-in code and the new member is only used in built-in code. Therefore we can safely add the new member at the end instead. Move it to the end and hide it from genksyms. Also hide it when building modules, to make sure they really don't use it."

goodwork 8y ago
If only Debian had someone like Linus Torvalds, he'd have insulted the fuck out of the developer who introduced this bug. Diabian will never be as good as Linux without Torvalds.

moocowtruck 8y ago

then it's not a debian stable kernel now is it

vectorEQ 8y ago

love this note on it: "Sure, maybe. I've suggested kernelci as a useful thing to help here, but we really need to be testing kernels complete with all the Debian patches to..." so does that mean they test the kernel but not their own patches to it? seems silly :D