6 ms·

Linux 6.1 on NanoPi R4S – On fixing SD-card support, Heisenbugs and Rabbit Holes

174 points by kohlschuetter 4y ago

rbanffy 4y ago
One of the most amazing things in Free Software is its multiplying effect. One person had a problem, dug into it, figured it out and got it fixed. The effects will ripple through time and space and improve the support of a lot of other devices now and in the future. This is FOSS at its best. Thank you, @kohlschuetter.
- squarefoot 4y ago
  Agreed. That is one of the most overlooked benefits of using Open Source by users dealing only with proprietary software. There's a lot more than just being free as in beer.
  - rbanffy 4y ago
    Remember FLOSS started out of RMS's frustration with a Xerox printer driver.
- Taniwha 4y ago
  Yeah, I think I was chasing this exact same bug elsewhere just the other day - modulated by the bulk caps, and variations on SD cards (some worked some didn't)
MrBuddyCasino 4y ago
OT, but that guy's SoundCloud doesn't disappoint if acoustic jaw harp techno is your thing: https://soundcloud.com/user-734845980/popular-tracks https://soundcloud.com/user-734845980/popular-tracks
uniqueuid 4y ago
Wow, if the name sounds familiar, it may be because you've read the author's seminal paper [1] on web page segmentation. Or used the boilerpipe library [2] to extract text yourself. Both are still great references. Thanks a lot! [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=ThQGwioAAAAJ&sortby=pubdate&citation_for_view=ThQGwioAAAAJ:u-x6o8ySG0sC https://scholar.google.com/citations?view_op=view_citation&h... [2] https://github.com/kohlschutter/boilerpipe https://github.com/kohlschutter/boilerpipe
_joel 4y ago
Ah cool, didn't even realise 6.x had been released.
bjt2n3904 4y ago
This is an incredible article. If you've never tried to bring up a circuit board before with embedded Linux, this is exactly what it is like. And for the record, device tree has the most abominable syntax ever. status = "okay";
- megous 4y ago
  Actually I like DT, especially the ability to name nodes and point to them from node properties, so I made similar config format for myself and happily use it to describe arbitrary graphs in configuration files. It's pretty powerful if the config doesn't have to be represented by a usual tree structure and there's a standard way to link between configuration objects.
  - bjt2n3904 4y ago
    Yeah, I'd just be happier if it was a more sane format like JSON-LD or YAML.
nfriedly 4y ago
Some of the newer NanoPi models have 2.5G ethernet ports - I may end up replacing my current router with one of those if they eventually receive official OpenWRT support.
- simcop2387 4y ago
  Given the fixes here for the boot up sequence, I'd imagine that it'll happen when OpenWRT starts it's upgrades to the kernel for the next major release. I believe 6.1 is supposed to be the next LTS series which means that OpenWRT and other distros will pick it up as their next major kernel version to support. Even if they don't pick it, these patches sound like excellent candidates for back porting since they fix some key issues for some boards and look to be fairly isolated and so less likely to cause other bugs.
- outworlder 4y ago
  Are they able to route packets at 2.5G speeds? They don't have dedicated routing or switching chips, so I'm wondering if the CPU can handle that.
  - nfriedly 4y ago
    I don't have either of these, but they claim the R5S can handle 2.1Gbps, and that the R6S can do 2.35Gbps. So, not quite 2.5, but close. NanoPi R5S: https://www.friendlyelec.com/index.php?route=product/product&product_id=287 https://www.friendlyelec.com/index.php?route=product/product... NanoPi R6S: https://www.friendlyelec.com/index.php?route=product/product&path=69&product_id=289 https://www.friendlyelec.com/index.php?route=product/product...
deleted 4y ago
[deleted]
jacob019 4y ago
The R4S is a great little device. Love the solid metal case, tiny size, and low power usage. I used one to route my gigabit Ethernet for several months. Compiled OpenWRT from source with LibreELEC's patches. No hardware problems for me--I did apply that voltage patch for the MicroSD. It handled the connection well, but couldn't quite push full gigabit PPPoE with VLAN. Gets close if you enable acceleration, but then no AQM. CenturyLink requires you to tunnel IPv6, so that on top of the PPPoE was a bit much for it, I could only IPv6 at about 300mbps. No eMMC, you have to provide a MicroSD, but OpenWRT doesn't write to the flash at all in normal operation, so that was fine. Now I have upgraded to the R5S. This thing has HDMI out, dual 2.5Gbps + 1x Gigabit Ethernet, and eMMC. It was a bit difficult to acquire, but I'm very happy with it. I can max out the Gigabit WAN now, with AQM which really helps. IPv6 reaches over 750mbps, and the R5S has cycles to spare, so I think that's the full speed. I can saturate the link over wireguard tunnels too. The R4S is up on eBay now.
- zokier 4y ago
  Do you have good source for OpenWRT for R5S? I have one sitting on my desk, but trying to get nice clean version of OpenWRT has been taking more time than I was hoping.
  - jacob019 4y ago
    I'm ashamed to admit that I was lazy this time, just used their FriendlyWRT build and then debloated. It's easy enough to disable all the packages, but a proper OpenWRT build would be preferable and you could be more confident that it is secure. You should be able to apply the patches from their github repo and build your own OpenWRT image. It was easy when I did it for the R4S (now I think it's supported upstream). The OpenWRT build system is very nice and easy.
    - btgeekboy 4y ago
      Yeah, I’ve built FriendlyWRT for the R5S. It’s reasonably doable in an afternoon, depending on skill level. FriendlyWRT, or at least the sources for it, is just a few minor configuration tweaks for performance + default packages. http://wiki.friendlyelec.com/wiki/index.php/How_to_Build_FriendlyWrt http://wiki.friendlyelec.com/wiki/index.php/How_to_Build_Fri...
Havoc 4y ago
> The R4S isn’t fully supported yet by a release-quality version of OpenWRT, but at least a so-called “snapshot” build (read: unstable) is available for tinkering This part isn't right...the latest release has r4s in it, not just the nightlies Pretty recent development though so not surprised author missed it
technion 4y ago
This whole write up is what gives me nightmares, kudos for anyone who works in this field. You cannot write a test suite to catch regressions. How would you build a ci that detects sd boot failures on commit? A static analyser couldn't help. Rewriting in rust isn't an advantage. You probably wont get help on stack overflow. Given all the above you might try to approach it as formally as possible, from the top down. That works in some spaces. Except half the writers problem was specs not being clearly laid out.
osigurdson 4y ago
Note that the Heisenberg uncertainty principle is about fundamental limits on accuracy of position and momentum. It doesn't state that the act of observing something might change it - that is the (completely classical) observer effect.
LAC-Tech 4y ago
There’s quite the market for ARM-powered single board computers with Rockchip, Amlogic, Allwinner, Broadcom, and other system-on-chip (SoC) brands at its heart... Depending on what device you buy, you may or may not get excellent support from the manufacturer, an online community, or maybe even books and magazines with step-by-step instructions. In many cases, the device isn’t fully supported by the “vanilla” Linux kernel, so you’re either stuck with some old kernel build or, by using a newer mainline kernel, you miss out on some features and performance optimizations that haven’t been upstreamed yet. Could anyone recommend something in this class that does have good linux support with the vanilla kernel?

174 points by kohlschuetter 4y ago

One of the most amazing things in Free Software is its multiplying effect. One person had a problem, dug into it, figured it out and got it fixed. The effects will ripple through time and space and improve the support of a lot of other devices now and in the future. This is FOSS at its best. Thank you, @kohlschuetter.

squarefoot 4y ago
Agreed. That is one of the most overlooked benefits of using Open Source by users dealing only with proprietary software. There's a lot more than just being free as in beer.
- rbanffy 4y ago
  Remember FLOSS started out of RMS's frustration with a Xerox printer driver.
Taniwha 4y ago
Yeah, I think I was chasing this exact same bug elsewhere just the other day - modulated by the bulk caps, and variations on SD cards (some worked some didn't)

MrBuddyCasino 4y ago

OT, but that guy's SoundCloud doesn't disappoint if acoustic jaw harp techno is your thing: https://soundcloud.com/user-734845980/popular-tracks https://soundcloud.com/user-734845980/popular-tracks

uniqueuid 4y ago

Wow, if the name sounds familiar, it may be because you've read the author's seminal paper [1] on web page segmentation. Or used the boilerpipe library [2] to extract text yourself. Both are still great references. Thanks a lot! [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=ThQGwioAAAAJ&sortby=pubdate&citation_for_view=ThQGwioAAAAJ:u-x6o8ySG0sC https://scholar.google.com/citations?view_op=view_citation&h... [2] https://github.com/kohlschutter/boilerpipe https://github.com/kohlschutter/boilerpipe

_joel 4y ago

Ah cool, didn't even realise 6.x had been released.

bjt2n3904 4y ago

This is an incredible article. If you've never tried to bring up a circuit board before with embedded Linux, this is exactly what it is like. And for the record, device tree has the most abominable syntax ever. status = "okay";

megous 4y ago
Actually I like DT, especially the ability to name nodes and point to them from node properties, so I made similar config format for myself and happily use it to describe arbitrary graphs in configuration files. It's pretty powerful if the config doesn't have to be represented by a usual tree structure and there's a standard way to link between configuration objects.
- bjt2n3904 4y ago
  Yeah, I'd just be happier if it was a more sane format like JSON-LD or YAML.

nfriedly 4y ago

Some of the newer NanoPi models have 2.5G ethernet ports - I may end up replacing my current router with one of those if they eventually receive official OpenWRT support.

simcop2387 4y ago
Given the fixes here for the boot up sequence, I'd imagine that it'll happen when OpenWRT starts it's upgrades to the kernel for the next major release. I believe 6.1 is supposed to be the next LTS series which means that OpenWRT and other distros will pick it up as their next major kernel version to support. Even if they don't pick it, these patches sound like excellent candidates for back porting since they fix some key issues for some boards and look to be fairly isolated and so less likely to cause other bugs.
outworlder 4y ago
Are they able to route packets at 2.5G speeds? They don't have dedicated routing or switching chips, so I'm wondering if the CPU can handle that.
- nfriedly 4y ago
  I don't have either of these, but they claim the R5S can handle 2.1Gbps, and that the R6S can do 2.35Gbps. So, not quite 2.5, but close. NanoPi R5S: https://www.friendlyelec.com/index.php?route=product/product&product_id=287 https://www.friendlyelec.com/index.php?route=product/product... NanoPi R6S: https://www.friendlyelec.com/index.php?route=product/product&path=69&product_id=289 https://www.friendlyelec.com/index.php?route=product/product...

deleted 4y ago

[deleted]

jacob019 4y ago

The R4S is a great little device. Love the solid metal case, tiny size, and low power usage. I used one to route my gigabit Ethernet for several months. Compiled OpenWRT from source with LibreELEC's patches. No hardware problems for me--I did apply that voltage patch for the MicroSD. It handled the connection well, but couldn't quite push full gigabit PPPoE with VLAN. Gets close if you enable acceleration, but then no AQM. CenturyLink requires you to tunnel IPv6, so that on top of the PPPoE was a bit much for it, I could only IPv6 at about 300mbps. No eMMC, you have to provide a MicroSD, but OpenWRT doesn't write to the flash at all in normal operation, so that was fine. Now I have upgraded to the R5S. This thing has HDMI out, dual 2.5Gbps + 1x Gigabit Ethernet, and eMMC. It was a bit difficult to acquire, but I'm very happy with it. I can max out the Gigabit WAN now, with AQM which really helps. IPv6 reaches over 750mbps, and the R5S has cycles to spare, so I think that's the full speed. I can saturate the link over wireguard tunnels too. The R4S is up on eBay now.

zokier 4y ago
Do you have good source for OpenWRT for R5S? I have one sitting on my desk, but trying to get nice clean version of OpenWRT has been taking more time than I was hoping.
- jacob019 4y ago
  I'm ashamed to admit that I was lazy this time, just used their FriendlyWRT build and then debloated. It's easy enough to disable all the packages, but a proper OpenWRT build would be preferable and you could be more confident that it is secure. You should be able to apply the patches from their github repo and build your own OpenWRT image. It was easy when I did it for the R4S (now I think it's supported upstream). The OpenWRT build system is very nice and easy.
  - btgeekboy 4y ago
    Yeah, I’ve built FriendlyWRT for the R5S. It’s reasonably doable in an afternoon, depending on skill level. FriendlyWRT, or at least the sources for it, is just a few minor configuration tweaks for performance + default packages. http://wiki.friendlyelec.com/wiki/index.php/How_to_Build_FriendlyWrt http://wiki.friendlyelec.com/wiki/index.php/How_to_Build_Fri...

Havoc 4y ago

> The R4S isn’t fully supported yet by a release-quality version of OpenWRT, but at least a so-called “snapshot” build (read: unstable) is available for tinkering This part isn't right...the latest release has r4s in it, not just the nightlies Pretty recent development though so not surprised author missed it

technion 4y ago

This whole write up is what gives me nightmares, kudos for anyone who works in this field. You cannot write a test suite to catch regressions. How would you build a ci that detects sd boot failures on commit? A static analyser couldn't help. Rewriting in rust isn't an advantage. You probably wont get help on stack overflow. Given all the above you might try to approach it as formally as possible, from the top down. That works in some spaces. Except half the writers problem was specs not being clearly laid out.

osigurdson 4y ago

Note that the Heisenberg uncertainty principle is about fundamental limits on accuracy of position and momentum. It doesn't state that the act of observing something might change it - that is the (completely classical) observer effect.

LAC-Tech 4y ago

There’s quite the market for ARM-powered single board computers with Rockchip, Amlogic, Allwinner, Broadcom, and other system-on-chip (SoC) brands at its heart... Depending on what device you buy, you may or may not get excellent support from the manufacturer, an online community, or maybe even books and magazines with step-by-step instructions. In many cases, the device isn’t fully supported by the “vanilla” Linux kernel, so you’re either stuck with some old kernel build or, by using a newer mainline kernel, you miss out on some features and performance optimizations that haven’t been upstreamed yet. Could anyone recommend something in this class that does have good linux support with the vanilla kernel?