3 ms·

Netbooting is one of these problems that turns out to be much more difficult than it's inherent complexity warrants. Setting up DHCP and TFTP doesn't cause a fu

Netbooting is one of these problems that turns out to be much more difficult than it's inherent complexity warrants. Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too. No libext, the only userland tool you get is a bunch of 10000 LoC C monsters that an autoconf behemoth ensures can only be built on a modern Linux glibc system. When at the end of the day we are talking about an abstraction that should work perfectly fine with just read(), write() and seek(). My ideal "netbooting" setup is a faux SD card that translates block IO to UDP packets on a GBit link and has a little software on my computer reading/writing from a binary blob. I can still mount it if theres something mountable in the blob. Being a SD card it solves the other big problem with netbooting, which is that it's not transparent. The bootloader needs to know it's netbooting. The kernel needs to have support for NFS. init needs to know you are netbooting to mount the differently-named FS. And then when a service decides on startup to reset the network connection (hello Android), you're still fucked.

mschuster91 9y ago
> Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. Linux kernel + initramfs, there you are. Why reimplement stuff when it can be done cheaper in kernelspace? > One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too That's true of any modern general-purpose filesystem, for all OSes - there's a reason why embedded devices often enough speak only FAT32, and some even only FAT16.
kardianos 9y ago
If you haven't seen it, try out the all-in-one tool: https://github.com/google/netboot https://github.com/google/netboot go get go.universe.tf/netboot/cmd/pixiecore It is completly self contained. The user just provides it with a linux kernel image and the initrd image.
- revelation 9y ago
  That's a very good start, if you install a DHCP server on Linux they usually assume it's because you trying to run a federated thousand machine network, not just to stub some initial config values, and so the complexity matches that expectation.. But the biggest pain point remains the whole NFS mess. I'm not sure you can even do something like SELinux over NFS, and even if you could, you still need to configure the right host computer permissions and attributes only for stuff to match on the netbooting machine.
  - kardianos 9y ago
    Netboot is most effective when the system is a minimal system and is configuration driven, see tinycore linux or CoreOS container linux with ignition. NFS is a mess. So design your solutions to not need it. I see no reason why the pixieboot solution isn't something capable of scaling to thousands of machines on a network.
    - revelation 9y ago
      I mean, we are in agreement. There are just systems like mobile SoC running platforms like Android where you desperately want some sort of network booting scheme for development because you are constantly rebuilding parts of the software, where the storage options for the part are highly constrained, slow and unreliable (like the RPi) and the performance and architecture necessitate cross-building.
  - gerdesj 9y ago
    "But the biggest pain point remains the whole NFS mess" That's a little unfair. NFS has been around for a very long time and yet it is still modern and being developed. You do have to do things the Unix way to get the most out of it, so you need to get Unix uid/gid standardised across your network for example. Samba's winbind and the rid backend to idmap is a rather easy way to turn AD users into full Unix users with consistent uid/gid. If you get your time, DNS and user IDs right then NFS can be pretty easy and very, very reliable. On modern Unix systems, you also have Samba to play with and other things. Lots of choice and with a bit of thought and research, you will find a decent solution.

mschuster91 9y ago

> Setting up DHCP and TFTP doesn't cause a full psychosis yet, but then we enter the "no win zone" of NFS and filesystems. Linux kernel + initramfs, there you are. Why reimplement stuff when it can be done cheaper in kernelspace? > One of the persistent frustrations with any sort of Linux filesystem is that their makers feel you would only ever want to use them with a full-blown Linux kernel, too That's true of any modern general-purpose filesystem, for all OSes - there's a reason why embedded devices often enough speak only FAT32, and some even only FAT16.

kardianos 9y ago

If you haven't seen it, try out the all-in-one tool: https://github.com/google/netboot https://github.com/google/netboot go get go.universe.tf/netboot/cmd/pixiecore It is completly self contained. The user just provides it with a linux kernel image and the initrd image.

revelation 9y ago
That's a very good start, if you install a DHCP server on Linux they usually assume it's because you trying to run a federated thousand machine network, not just to stub some initial config values, and so the complexity matches that expectation.. But the biggest pain point remains the whole NFS mess. I'm not sure you can even do something like SELinux over NFS, and even if you could, you still need to configure the right host computer permissions and attributes only for stuff to match on the netbooting machine.
- kardianos 9y ago
  Netboot is most effective when the system is a minimal system and is configuration driven, see tinycore linux or CoreOS container linux with ignition. NFS is a mess. So design your solutions to not need it. I see no reason why the pixieboot solution isn't something capable of scaling to thousands of machines on a network.
  - revelation 9y ago
    I mean, we are in agreement. There are just systems like mobile SoC running platforms like Android where you desperately want some sort of network booting scheme for development because you are constantly rebuilding parts of the software, where the storage options for the part are highly constrained, slow and unreliable (like the RPi) and the performance and architecture necessitate cross-building.
- gerdesj 9y ago
  "But the biggest pain point remains the whole NFS mess" That's a little unfair. NFS has been around for a very long time and yet it is still modern and being developed. You do have to do things the Unix way to get the most out of it, so you need to get Unix uid/gid standardised across your network for example. Samba's winbind and the rid backend to idmap is a rather easy way to turn AD users into full Unix users with consistent uid/gid. If you get your time, DNS and user IDs right then NFS can be pretty easy and very, very reliable. On modern Unix systems, you also have Samba to play with and other things. Lots of choice and with a bit of thought and research, you will find a decent solution.