22 ms·
Best decision I ever made was setting up a Pi netboot server on my LAN. Now I have a whole bunch of Pi 3's and 4's scattered around my home with no local storag
by LeoPanthera 6y ago
Best decision I ever made was setting up a Pi netboot server on my LAN. Now I have a whole bunch of Pi 3's and 4's scattered around my home with no local storage at all doing cool things, and I can make them boot into a completely different OS just by renaming a symlink on the server.
- mmastrac 6y agoWhat are you using for a filesystem? NFS? My worry would be saturating the older 100Mbit connections when downloading things like docker images.
- mkj 6y ago100mbit is similar to a slow sd card or usb stick anyway?
- takeda 6y agoThat's more a problem of docker than NFS though. You probably would be better if you just use it traditionally or use something like NixOS. There was a project that I read about a while ago: https://matthewbauer.us/blog/nixiosk.html https://matthewbauer.us/blog/nixiosk.html
- threentaway 6y agoThe Pi4 can do 300-350mbps, which is faster than most people's internet connection.
- vetinari 6y agoNetbooting and mounting NFS root would be done over local lan, the speed of internet connection has no relevance there.
- threentaway 6y agoRight, I was referring to my parent comment that talked about pulling Docker images which come from the internet unless you set up a proxy.
- vetinari 6y agoOk, that depends on the docker images; things like pihole are relatively small (300 MB, including all layers); I would be more afraid of running containers over NFS slowed down by rpi ethernet, than downloading the images themselves.
- threentaway 6y agoAgreed. I'd have it set up so that there is some local storage via a USB3 thumb drive to hold the container images, which anecdotally pretty easily hits 80MB/s for reads, often hovering in the 100-110MB/s range.
- zbrozek 6y agoTrue, but if you're netbooting you're probably not going to the internet.
- bscphil 6y agoActually, just tested on my local network, got ~940 Mbps sending and receiving. Pi4 has real Gbit now, not limited by the USB bandwidth any more.
- ehonda 6y agoI can download large files from my pi at 900mb/s and upload around 800mb/s. The hard drives attached to the pie are formatted ext3. If you use an NTFS drive though, you will get speeds around what you are describing.
- LeoPanthera 6y agoYes, TFTP for the boot rom, and then NFS for the root fs. I don't use docker or any other kind of container system because it's so easy to just reboot a Pi into a new OS on demand. Everything runs natively.
- Gibbon1 6y agoThat's what we did when developing our original Embedded Linux stuff. Once you get it working it's really nice. You have your standard tools, version control, etc running locally on your system.
- Dylan16807 6y ago> My worry would be saturating the older 100Mbit connections when downloading things like docker images. Is that too slow? I feel like 30 seconds to fill up half the ram is a perfectly acceptable boot time.
- umvi 6y ago> Now I have a whole bunch of Pi 3's and 4's scattered around my home with no local storage at all doing cool things Elaborate? I have a single pi3 as a Plex server, looking for more ideas.
- threentaway 6y agohttps://github.com/badaix/snapcast https://github.com/badaix/snapcast this seems really cool, although I haven't got around to doing it myself.
- geekuillaume 6y agoSnapcast is really cool, Johannes did an awesome job! I used it for a long time at home on multiple RaspberryPis. I've been working for some time on a more user-friendly and cross-OS alternative. It's available here: https://github.com/geekuillaume/soundsync https://github.com/geekuillaume/soundsync I think I'm going to try to integrate net-boot in Soundsync this weekend. It would be awesome to be able to install Soundsync on any computer on the network, turn on an option and then just start the rPi to get it to play music.
- LeoPanthera 6y agoTwo play music, one into my hifi, one into my home audio distribution system. One is setup for RetroPie. One has LibreELEC installed, showing TV from my HDHomeRun. One has a GPS hat and is a local NTP server. And one boots into an Amiga emulator because I'm very old and nostalgic.
- joshspankit 6y ago(If you’re able to play to both simultaneously) Do the music playback devices stay in sync?
- LeoPanthera 6y agoNo they're independent. But if I want multiroom audio I just use my very low-tech home audio system. Most rooms have ceiling speakers that are all wired back to the garage where one of the Pi's lives.
- abhishekjha 6y ago> just by renaming a symlink on the server. Wow. How?
- LeoPanthera 6y agoNetboot instructions are here: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md https://www.raspberrypi.org/documentation/hardware/raspberry... The important bit is that it will look for its boot files in a subdirectory named the serial number of the Pi. Only if that fails will it look in the root. So I have symlinks for the serial numbers of all my Pis that point to whatever OS I want to boot.
- therealx 6y agoVery interesting it looks for the serial number and not the MAC address!
- yjftsjthsd-h 6y agoPretty sure that's because the pi doesn't actually have a hardcoded MAC; I think I've read that it's just set in software and persisted by the OS, which is weird but workable right up until there is no local storage to persist on.
- JamesSwift 6y agoThats awesome. I asked about people netbooting for retropie in reddit and people were really confused why I would want to do that. I think I'm going to go ahead and set it up now.
- zbrozek 6y agoI did this, but using a more traditional server. It was... OK. The pi3b+ bootloader was flaky and I ended up needing an SD card with just the kernel on it, but I wanted it ro. This made updating kind of touchy. I can imagine that if they've made the bootloader more robust on the pi4 that this is truly excellent now.
- StillBored 6y agoIf you already have the SD card and USB, your probably better off using https://github.com/pftf/RPi3/releases/tag/v1.30 https://github.com/pftf/RPi3/releases/tag/v1.30 which is a open source UEFI implementation. Its far more capable of dealing with underlying os updates using your OS's preferred update method. You put the UEFI firmware on the SD, update the config.txt to point at it, and then pretend the rpi is a normal x86/uefi system and use apt/zypper/dnf/whatever for updates (including the kernel). Everything then boots/runs/updates off the USB disk. The arm ecosystem seems to be slowly moving to some form of UEFI+grub boot for all the linux/etc distributions and away from all these platform specific boot methods.
- jes5199 6y agowhat hardware does the netboot server run on?
- LeoPanthera 6y agoIt's a Dell Poweredge running FreeNAS, which I was already using as, well, a NAS. The required DHCP responses come from my router running pfSense, which is some kind of cheap Supermicro thing.
- giancarlostoro 6y agoCan you host the netboot server from a workstation by chance? This is something I had never considered and it would really change things up for me, how does that even work? Is it just an installed distro or just a network image on the device it's booting from? I'm somewhat familiar with netbooting an installer but I've never setup a server just used the tech to install an OS from the network to hardware. I did bump into these docs: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md https://www.raspberrypi.org/documentation/hardware/raspberry... Are these the gist of it, or did you do a little more?
- LeoPanthera 6y agoWell, that's instructions on how to setup a Pi itself to be the netboot server. That will work fine, but your DHCP server still needs to be configurable to emit the correct responses to find the TFTP server, if you don't want to use a Pi itself as your DHCP server, which I would imagine most people don't. My DHCP server is pfSense, and I'm using a FreeNAS server as the TFTP/NFS server. The Pi netboot procedure is documented in more general terms, here: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md https://www.raspberrypi.org/documentation/hardware/raspberry...
- jcims 6y agoPi notwithstanding, it feels like this process hasn’t changed in 30 years.
- tinus_hn 6y agoI know for a start Microsoft made some ‘updates to the standards’ when they found out tftp isn’t really suitable for fetching gigabyte size images.
- MayeulC 6y ago> That will work fine, but your DHCP server still needs to be configurable to emit the correct responses to find the TFTP server, if you don't want to use a Pi itself as your DHCP server, which I would imagine most people don't. Why wouldn'y you? That's what I do: I use the DHCP server (dnsmasq) that comes with pi-hole, with a few additional config files, and it works pretty well. It even grants leases faster than my router was (which has been an issue for radv packets & DNS).
- fulafel 6y agoWhaty the best way to do this without trusting the untrusted LAN?
- dikei 6y agoIf you just want to prevent other users from snooping you, use VLAN. There's no protection against the network administrators though.
- r22z 6y agoPantera, é você? :-O
- deleted 6y ago[deleted]
- mikorym 6y agoWhat happens when you get a power cut? Or, assuming you are from a well functioning country, what would happen and how would you restart?
- stedaniels 6y agoI'm not sure I see the problem. Most servers will come up before a netboot/PXE boot timeout. At least in my experience they do.
- sgt 6y agoThat server should be on a UPS. Some people like to use an old laptop as a server at home, and that's even more ideal as it has a built in battery, often allowing the machine to stay alive for an hour or so before dying.
- mikorym 6y agoThanks, this answers my question. In some developing countries you could have power cuts of over 10 hours.
- chaz6 6y agoHow do you handle persistent storage? I find it is often non-trivial to adapt an appliation-specific operating system to run via pxe boot.
- rwmj 6y agoNo one in the thread has mentioned it yet, but NBD is also an option, either NBD root or NBD for particular filesystems. I've used it for managing Pis and a couple of SiFive boards. It has different trade-offs from NFS (not necessarily faster/slower/better/worse, depends on what you're doing). For what I'm doing which is software builds I was able to write a specific NBD driver[1] for this task which has quite large performance gains, although there are trade-offs that only make sense for temporary build directories. I don't believe this is possible with NFS at all, or at least not without far more effort. [1] https://rwmj.wordpress.com/2020/03/21/new-nbdkit-remote-tmpfs-tmpdisk-plugin/ https://rwmj.wordpress.com/2020/03/21/new-nbdkit-remote-tmpf...
- sippingjippers 6y agoProbably the most important tradeoff between nbd/nfs is that NFS by virtue of supporting shared mutation, means client-side caching is either non-existent or barely useful With NBD you're giving a filesystem driver exclusive access to a block device, the kernel knows the underlying isn't expected to change, so for example no round-trips are required to read or revalidate some cached data, and something like "find /" while slow to run the first time around, will likely be served from cache on a subsequent pass, and thus lightning fast
- rwmj 6y agoThis is very true, but I think NFSv4 makes things a little better?
- terrywang 6y agoThat requires your network attached storage (NFS/iSCSI or something else) to be HA ;-) I started booting from USB SSD a couple of months ago since I got my Pi 8GB, intended to use it as my low power consumption workstation replacing the Optiplex SFF Running Manjaro + KDE Plasma. Turn out to be, Manjaro ARM + KDE Plasma on Pi 4 runs pretty well (even all KWin eye candies work as well as the Intel HD integrated GPU on the x86_64 variant). However, USB 3.0 to SATA III SSD is clearly a bottleneck still. I am yet to check if NFS over 1Gbps ethernet can be faster (anyway, my btrfs backed consumer grade NFS may not be production grade to support 10+ devices).