8 ms·

The state of netbooting Raspberry Pis

138 points by alexellisuk 9y ago

geerlingguy 9y ago
Not to mention the onboard LAN is over the USB connection, but is topped off at 100 Mbps, so no high-throughput uses will work over the network. Alternative SBCs have started adding onboard eMMC, USB 3.0, and GigE; while they’re still not desktop PCs, having faster IO makes tinkering so much less annoying (for many applications). I hope if anything, the next Pi has Gigabit.
- mschuster91 9y ago
  Given that many people use RPi as base for home entertainment systems... I'd prefer GigE and one or two SATA ports.
  - api 9y ago
    There are boards with all that. The fastest low-cost networking board right now (that I can find) is the espressobin, built around the Marvell Armada chipset. It's got three full-duplex GigE ports that can actually run as such as well as pci-E and SATA. Unfortunately it's oriented toward routers and stuff so it lacks HDMI, so no good for home entertainment.
    - silasb 9y ago
      Never knew about this board. Might make for a pretty nice home router.
      - eltoozero 9y ago
        Soekris[0] and RouterBoard[1] are specifically for Routers too but at least Soekris is a more general purpose SBC. Also Pine64[2] is supposed to be suitable for HTPC (Kodi) usage but I haven’t tried it and YMMV. I <3 RouterBoards + MikroTik. [0]: http://www.soekris.dk http://www.soekris.dk [1]: https://mikrotik.com/products/group/routerboard https://mikrotik.com/products/group/routerboard [2]: https://www.pine64.org https://www.pine64.org
        giancarlostoro 9y ago
        I tried to get a Pinebook from those Pine64 guys but never heard back from them. Not sure I'd recommend them since they didn't even send a confirmation email or anything of the sort.
        ac29 9y ago
        Sokris effectively went out of business about 9 months ago [0]. The danish company you linked to doesn't seem to sell any of their SBCs. [0] http://www.soekris.com/ http://www.soekris.com/
    - mschuster91 9y ago
      > There are boards with all that. Problem with these: I can never be sure what level of acceleration for video playback is supported - and in what quality. The Pi is pretty rock solid in that department. Also, the Pi has by far the biggest community and extensive documentation, and I can rely on there still being Pis available in years, compared to the Chinese outfits where no one knows how long they'll stay in market.
      - heegemcgee 9y ago
        Concur. I've played with the ODroid C2, Beaglebone Black, and the three "full flavor" Raspberry Pi models. The RasPis have, far and away, the best supportability. The ODroids are great when they work - love that GigE, but they aren't maintaining an OS distribution with the same care and man-power that Raspbian is getting.
  - gerdesj 9y ago
    Me too. However, the board is cheap and properly supported and conforms to a lot of standards. If you stick to branded stuff then for less than £100 you get a thin client capable of 1080p in a decent case that can be bungeed to the back of a monitor with a power supply that wont blow up and a decent HDMI cable. The wifi and ethernet are good enough for 1080p streaming and good enough for good quality RDP. Modern low cost SD cards are pretty quick as well. I have rather a lot of them, at home and work. For me, quality and standards count. Nowadays Pis are made in Wales (by Sony). They are designed by a British foundation. To all intents and purposes they are a British computer that fulfils a particular niche rather well. No, the Pi3 does not have GBE or SATA but it is (in my opinion) a piece of kit that you can trust to not set fire to your cat.
  - xellisx 9y ago
    I want two 25 Gb interfaces. =)
  - tlrobinson 9y ago
    I agree, though maybe as a “Raspberry Pi Pro” or “Plus” model. (Aside from that, I’d really like if all models had a serial console over USB, and perhaps a small amount of built-in flash memory to avoid the need for a MicroSD card in some cases)
revelation 9y ago
Netbooting is one of these problems that turns out to be much more difficult than it's inherent complexity warrants. Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too. No libext, the only userland tool you get is a bunch of 10000 LoC C monsters that an autoconf behemoth ensures can only be built on a modern Linux glibc system. When at the end of the day we are talking about an abstraction that should work perfectly fine with just read(), write() and seek(). My ideal "netbooting" setup is a faux SD card that translates block IO to UDP packets on a GBit link and has a little software on my computer reading/writing from a binary blob. I can still mount it if theres something mountable in the blob. Being a SD card it solves the other big problem with netbooting, which is that it's not transparent. The bootloader needs to know it's netbooting. The kernel needs to have support for NFS. init needs to know you are netbooting to mount the differently-named FS. And then when a service decides on startup to reset the network connection (hello Android), you're still fucked.
- mschuster91 9y ago
  > Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. Linux kernel + initramfs, there you are. Why reimplement stuff when it can be done cheaper in kernelspace? > One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too That's true of any modern general-purpose filesystem, for all OSes - there's a reason why embedded devices often enough speak only FAT32, and some even only FAT16.
- kardianos 9y ago
  If you haven't seen it, try out the all-in-one tool: https://github.com/google/netboot https://github.com/google/netboot go get go.universe.tf/netboot/cmd/pixiecore It is completly self contained. The user just provides it with a linux kernel image and the initrd image.
  - 9y ago
mirceal 9y ago
this is a bit misleading in that it's not necessarily about netbooting (which per linked article just works). As someone who's played with raspberry pi's on "real" networks I can tell you that netbooting one is not as reliable as you think. It may work on a small home network, but once you have 50+ devices tftp-ing something will show it's limits. The firmware that netboots is also not as reliable as one would think. If it fails it just hangs. A proper netboot/PXE client will retry forever. The approach I followed was to build a minimum ramdisk image which I've placed on the SD card. When that starts, it creates a in-memory file system and downloads the actual image to this file system. The SD card is in read-only mode after that.
- exhilaration 9y ago
  It seems like another benefit of your method is that it will allow the subsequent image to be downloaded over WiFi, is that right? Or are you also limited to ethernet?
  - mirceal 9y ago
    it can work on whatever. the biggest advantage is that it's not using the SD card and it's always using the latest image.
- ChuckMcM 9y ago
  I don't disagree, it reads a bit like submarine marketing for the clustering hardware. But the difficulty in net booting is for the Pi1 or Pi2. On the Pi 3 it "just works." That said, net booting used to be a thing. When I joined Sun Microsystems in 1986 it was all about the 'diskless workstation.' At the time was the brand new Sun 3/50.
  - mirceal 9y ago
    point was that it does not just work on a pi 3 if you put it in a real-life environment. also, net booting is still a thing, but not for the diskless scenario :)
  - deleted 9y ago
znpy 9y ago
The real issue with raspberry pis is that ethernet is actually on the usb2 controller, so it's capped at 80mbps afaik, and it's also shared with other usb storage. I hope raspberry pi 4 has gigabit ethernet and netbooting becomes a first class function.
- blacksmith_tb 9y ago
  And/or a usb3 controller to tie networking to...
Improvotter 9y ago
I'm pretty curious about using RPis for a Kubernetes Cluster. Is there a benefit to using a RPi cluster compared to a simple server or is this purely for research purposes?
- bendews 9y ago
  Depending on the amount of nodes you’re wanting, it usually is better to just grab an Intel NUC and virtualise. Much smaller than a RPi cluster and much more powerful.

138 points by alexellisuk 9y ago

Not to mention the onboard LAN is over the USB connection, but is topped off at 100 Mbps, so no high-throughput uses will work over the network. Alternative SBCs have started adding onboard eMMC, USB 3.0, and GigE; while they’re still not desktop PCs, having faster IO makes tinkering so much less annoying (for many applications). I hope if anything, the next Pi has Gigabit.

mschuster91 9y ago
Given that many people use RPi as base for home entertainment systems... I'd prefer GigE and one or two SATA ports.
- api 9y ago
  There are boards with all that. The fastest low-cost networking board right now (that I can find) is the espressobin, built around the Marvell Armada chipset. It's got three full-duplex GigE ports that can actually run as such as well as pci-E and SATA. Unfortunately it's oriented toward routers and stuff so it lacks HDMI, so no good for home entertainment.
  - silasb 9y ago
    Never knew about this board. Might make for a pretty nice home router.
    - eltoozero 9y ago
      Soekris[0] and RouterBoard[1] are specifically for Routers too but at least Soekris is a more general purpose SBC. Also Pine64[2] is supposed to be suitable for HTPC (Kodi) usage but I haven’t tried it and YMMV. I <3 RouterBoards + MikroTik. [0]: http://www.soekris.dk http://www.soekris.dk [1]: https://mikrotik.com/products/group/routerboard https://mikrotik.com/products/group/routerboard [2]: https://www.pine64.org https://www.pine64.org
      - giancarlostoro 9y ago
        I tried to get a Pinebook from those Pine64 guys but never heard back from them. Not sure I'd recommend them since they didn't even send a confirmation email or anything of the sort.
      - ac29 9y ago
        Sokris effectively went out of business about 9 months ago [0]. The danish company you linked to doesn't seem to sell any of their SBCs. [0] http://www.soekris.com/ http://www.soekris.com/
  - mschuster91 9y ago
    > There are boards with all that. Problem with these: I can never be sure what level of acceleration for video playback is supported - and in what quality. The Pi is pretty rock solid in that department. Also, the Pi has by far the biggest community and extensive documentation, and I can rely on there still being Pis available in years, compared to the Chinese outfits where no one knows how long they'll stay in market.
    - heegemcgee 9y ago
      Concur. I've played with the ODroid C2, Beaglebone Black, and the three "full flavor" Raspberry Pi models. The RasPis have, far and away, the best supportability. The ODroids are great when they work - love that GigE, but they aren't maintaining an OS distribution with the same care and man-power that Raspbian is getting.
- gerdesj 9y ago
  Me too. However, the board is cheap and properly supported and conforms to a lot of standards. If you stick to branded stuff then for less than £100 you get a thin client capable of 1080p in a decent case that can be bungeed to the back of a monitor with a power supply that wont blow up and a decent HDMI cable. The wifi and ethernet are good enough for 1080p streaming and good enough for good quality RDP. Modern low cost SD cards are pretty quick as well. I have rather a lot of them, at home and work. For me, quality and standards count. Nowadays Pis are made in Wales (by Sony). They are designed by a British foundation. To all intents and purposes they are a British computer that fulfils a particular niche rather well. No, the Pi3 does not have GBE or SATA but it is (in my opinion) a piece of kit that you can trust to not set fire to your cat.
- xellisx 9y ago
  I want two 25 Gb interfaces. =)
- tlrobinson 9y ago
  I agree, though maybe as a “Raspberry Pi Pro” or “Plus” model. (Aside from that, I’d really like if all models had a serial console over USB, and perhaps a small amount of built-in flash memory to avoid the need for a MicroSD card in some cases)

revelation 9y ago

Netbooting is one of these problems that turns out to be much more difficult than it's inherent complexity warrants. Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too. No libext, the only userland tool you get is a bunch of 10000 LoC C monsters that an autoconf behemoth ensures can only be built on a modern Linux glibc system. When at the end of the day we are talking about an abstraction that should work perfectly fine with just read(), write() and seek(). My ideal "netbooting" setup is a faux SD card that translates block IO to UDP packets on a GBit link and has a little software on my computer reading/writing from a binary blob. I can still mount it if theres something mountable in the blob. Being a SD card it solves the other big problem with netbooting, which is that it's not transparent. The bootloader needs to know it's netbooting. The kernel needs to have support for NFS. init needs to know you are netbooting to mount the differently-named FS. And then when a service decides on startup to reset the network connection (hello Android), you're still fucked.

mschuster91 9y ago
> Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. Linux kernel + initramfs, there you are. Why reimplement stuff when it can be done cheaper in kernelspace? > One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too That's true of any modern general-purpose filesystem, for all OSes - there's a reason why embedded devices often enough speak only FAT32, and some even only FAT16.
kardianos 9y ago
If you haven't seen it, try out the all-in-one tool: https://github.com/google/netboot https://github.com/google/netboot go get go.universe.tf/netboot/cmd/pixiecore It is completly self contained. The user just provides it with a linux kernel image and the initrd image.
- 9y ago

mirceal 9y ago

this is a bit misleading in that it's not necessarily about netbooting (which per linked article just works). As someone who's played with raspberry pi's on "real" networks I can tell you that netbooting one is not as reliable as you think. It may work on a small home network, but once you have 50+ devices tftp-ing something will show it's limits. The firmware that netboots is also not as reliable as one would think. If it fails it just hangs. A proper netboot/PXE client will retry forever. The approach I followed was to build a minimum ramdisk image which I've placed on the SD card. When that starts, it creates a in-memory file system and downloads the actual image to this file system. The SD card is in read-only mode after that.

exhilaration 9y ago
It seems like another benefit of your method is that it will allow the subsequent image to be downloaded over WiFi, is that right? Or are you also limited to ethernet?
- mirceal 9y ago
  it can work on whatever. the biggest advantage is that it's not using the SD card and it's always using the latest image.
ChuckMcM 9y ago
I don't disagree, it reads a bit like submarine marketing for the clustering hardware. But the difficulty in net booting is for the Pi1 or Pi2. On the Pi 3 it "just works." That said, net booting used to be a thing. When I joined Sun Microsystems in 1986 it was all about the 'diskless workstation.' At the time was the brand new Sun 3/50.
- mirceal 9y ago
  point was that it does not just work on a pi 3 if you put it in a real-life environment. also, net booting is still a thing, but not for the diskless scenario :)
- deleted 9y ago

znpy 9y ago

The real issue with raspberry pis is that ethernet is actually on the usb2 controller, so it's capped at 80mbps afaik, and it's also shared with other usb storage. I hope raspberry pi 4 has gigabit ethernet and netbooting becomes a first class function.

blacksmith_tb 9y ago
And/or a usb3 controller to tie networking to...

Improvotter 9y ago

I'm pretty curious about using RPis for a Kubernetes Cluster. Is there a benefit to using a RPi cluster compared to a simple server or is this purely for research purposes?

bendews 9y ago
Depending on the amount of nodes you’re wanting, it usually is better to just grab an Intel NUC and virtualise. Much smaller than a RPi cluster and much more powerful.