5 ms·

Every time there’s a conversation about ISO’s and booting someone mentions netboot, I still don’t understand at a granular level how it works and the native doc

by vxNsr 5y ago

Every time there’s a conversation about ISO’s and booting someone mentions netboot, I still don’t understand at a granular level how it works and the native documentation, at least to me, appears to be written in Greek. I can’t make heads nor tails of it. Do you have a guide you followed that explained how it worked and what to do?

CMCDragonkai 5y ago
It basically bundles iPXE which allows one to boot over the network including ethernet and I think WiFi although WiFi depends on drivers. It downloads an bootable image and chain boots into it. We used in our ML data center to iterate on our NixOS images meaning we could push out updates to the underlying OS and then just trigger a restart via IPMI and make it download the new images. Note that we didn't end up using netboot at the end, we just use iPXE directly.
toast0 5y ago
PC Network booting these days is usually PXE booting. You can do it with BIOS or UEFI styles, but I've only had success with the BIOS version. I setup my PXE boot setup over a long time, so I don't have a guide, but here's my idea of the steps the boot process takes. a) (if enabled and selected for booting) the system firmware jumps to the Option ROM on your NIC to boot. These days, almost all cards have a PXE rom, but if you have an ancient card, it probably has a socket fot a boot rom. Getting that setup is possible but outside my scope. b) the boot rom includes a driver for the NIC and usually Intel's PXE software; either way it will do a DHCP (or sometimes BOOTP) request and ask for a 'next server' and indicate the platform (x86 bios, x86 uefi, amd64 bios, amd64 uefi, sparc variations, etc, etc). The DHCP server will assign an IP and all that junk, but also provide a pointer to the next server (and a path), used to fetch more code to run. c) next server is traditionally a TFTP server, and the path may be client or platform specific. More modern pxe stacks can do http(s), usually at higher performance d) I find it easiest to have next server point at pxelinux, part of syslinux that offers a nice menu system and http(s) support that's usually a lot faster than tftp e) from the menu, you can often selection options that will load a kernel and modules and boot with a command line; in the boot process, usually all the pxe stuff is terminated or discarded. f) You can also load disk images (floppy, hard drive, iso) and boot from them using memdisk from syslinux. This comes with some caveats; MEMDISK is BIOS only, it hooks the (defacto) standard BIOS disk calls to provide access and doesn't have a UEFI equivalent (AFAIK, corrections welcome), but also if you use an OS that doesn't use BIOS disk access (most protected mode OSes), it needs to support MEMDISK or you can't access the images. This means only some Linux images work, although recent images often do. I've made a patch for FreeBSD, but it's been in review for a long time. Older FreeBSD setup images worked because the setup filesystem was a disk image loaded from the disc by the bootloader, you'd need enough memory to have the cd disc image loaded by memdisk and the setup filesystem loaded by the bootloader and enough left to run the installer, but it's not too bad on today's machines. So, TLDR, you need to fiddle with your dhcp server, setup a tftp server, copy some files for a bootmenu and/or something to boot. And finally, tweak the boot settings on your netbooting device. Useful things I've done with this include booting Debian installers, using weird 3rd party software to install windows XP, booting FreeBSD installers with my patch, running disk firmware updates packaged as linux isos (intel), and most fun, running my hobby OS without having to setup a bootdisk (it's multiboot compatible, so pxelinux can boot it easily)
- kstenerud 5y ago
  So does this mean that pwning the PXE image server basically pwns the entire organization?
  - toast0 5y ago
    Depends on your setup and timing. If your machines only PXE boot when doing system setup, pwning the image only pwns on next system setup. Which maybe is never for existing systems; when I ran prod systems, most of my prod servers never got resetup unless their root drive gave out, but that was rare, too. So only new machines would get pwnd in a setup like that. If your machines always PXE boot, pwning the image pwns machines on next boot. Most of my prod machines got rebooted once or twice after they were setup, but if I had an always PXE boot setup, maybe I would run things differently and rebooting might happen more often. Then again, if everything is PXE booted, maybe we're running with a network filesystem for the root drive, and then you'd want to pwn that filesystem/file server to take over. Up to you to figure out if you can update the binary for something that's already running as root to take over, or if you have to wait for something new to spawn as root. On FreeBSD, you can edit a binary on disk while it's running and if you do it wrong (or maybe right), also edit the binary in memory, but if it's a networked filesystem, I dunno if that's still possible. Other systems may not be so capable. A rogue DHCP server that's faster than the real one can also do nasty things to a PXE client. There's some mentions of SecureBoot and PXE, so there's probably a way to get things aligned in a secure way, but somehow you've got to enroll your keys (and probably disenroll the default keys). Anyway, that's way outside my expertise.
  - xxpor 5y ago
    It could, but you have to consider your threat model. Which is more of a risk: a PXE box with logging and other auditing mechanisms, and isn't exposed to the internet, or random data center ops people running around with USB drives doing who knows what? There's also secure boot etc like the parallel comment said. But yes, if you can pwn the bootstrap process of anything, you have a lot of power. That's been true since computers were invented.
- p_l 5y ago

by vxNsr 5y ago

CMCDragonkai 5y ago

It basically bundles iPXE which allows one to boot over the network including ethernet and I think WiFi although WiFi depends on drivers. It downloads an bootable image and chain boots into it. We used in our ML data center to iterate on our NixOS images meaning we could push out updates to the underlying OS and then just trigger a restart via IPMI and make it download the new images. Note that we didn't end up using netboot at the end, we just use iPXE directly.

toast0 5y ago

PC Network booting these days is usually PXE booting. You can do it with BIOS or UEFI styles, but I've only had success with the BIOS version. I setup my PXE boot setup over a long time, so I don't have a guide, but here's my idea of the steps the boot process takes. a) (if enabled and selected for booting) the system firmware jumps to the Option ROM on your NIC to boot. These days, almost all cards have a PXE rom, but if you have an ancient card, it probably has a socket fot a boot rom. Getting that setup is possible but outside my scope. b) the boot rom includes a driver for the NIC and usually Intel's PXE software; either way it will do a DHCP (or sometimes BOOTP) request and ask for a 'next server' and indicate the platform (x86 bios, x86 uefi, amd64 bios, amd64 uefi, sparc variations, etc, etc). The DHCP server will assign an IP and all that junk, but also provide a pointer to the next server (and a path), used to fetch more code to run. c) next server is traditionally a TFTP server, and the path may be client or platform specific. More modern pxe stacks can do http(s), usually at higher performance d) I find it easiest to have next server point at pxelinux, part of syslinux that offers a nice menu system and http(s) support that's usually a lot faster than tftp e) from the menu, you can often selection options that will load a kernel and modules and boot with a command line; in the boot process, usually all the pxe stuff is terminated or discarded. f) You can also load disk images (floppy, hard drive, iso) and boot from them using memdisk from syslinux. This comes with some caveats; MEMDISK is BIOS only, it hooks the (defacto) standard BIOS disk calls to provide access and doesn't have a UEFI equivalent (AFAIK, corrections welcome), but also if you use an OS that doesn't use BIOS disk access (most protected mode OSes), it needs to support MEMDISK or you can't access the images. This means only some Linux images work, although recent images often do. I've made a patch for FreeBSD, but it's been in review for a long time. Older FreeBSD setup images worked because the setup filesystem was a disk image loaded from the disc by the bootloader, you'd need enough memory to have the cd disc image loaded by memdisk and the setup filesystem loaded by the bootloader and enough left to run the installer, but it's not too bad on today's machines. So, TLDR, you need to fiddle with your dhcp server, setup a tftp server, copy some files for a bootmenu and/or something to boot. And finally, tweak the boot settings on your netbooting device. Useful things I've done with this include booting Debian installers, using weird 3rd party software to install windows XP, booting FreeBSD installers with my patch, running disk firmware updates packaged as linux isos (intel), and most fun, running my hobby OS without having to setup a bootdisk (it's multiboot compatible, so pxelinux can boot it easily)

kstenerud 5y ago
So does this mean that pwning the PXE image server basically pwns the entire organization?
- toast0 5y ago
  Depends on your setup and timing. If your machines only PXE boot when doing system setup, pwning the image only pwns on next system setup. Which maybe is never for existing systems; when I ran prod systems, most of my prod servers never got resetup unless their root drive gave out, but that was rare, too. So only new machines would get pwnd in a setup like that. If your machines always PXE boot, pwning the image pwns machines on next boot. Most of my prod machines got rebooted once or twice after they were setup, but if I had an always PXE boot setup, maybe I would run things differently and rebooting might happen more often. Then again, if everything is PXE booted, maybe we're running with a network filesystem for the root drive, and then you'd want to pwn that filesystem/file server to take over. Up to you to figure out if you can update the binary for something that's already running as root to take over, or if you have to wait for something new to spawn as root. On FreeBSD, you can edit a binary on disk while it's running and if you do it wrong (or maybe right), also edit the binary in memory, but if it's a networked filesystem, I dunno if that's still possible. Other systems may not be so capable. A rogue DHCP server that's faster than the real one can also do nasty things to a PXE client. There's some mentions of SecureBoot and PXE, so there's probably a way to get things aligned in a secure way, but somehow you've got to enroll your keys (and probably disenroll the default keys). Anyway, that's way outside my expertise.
- xxpor 5y ago
  It could, but you have to consider your threat model. Which is more of a risk: a PXE box with logging and other auditing mechanisms, and isn't exposed to the internet, or random data center ops people running around with USB drives doing who knows what? There's also secure boot etc like the parallel comment said. But yes, if you can pwn the bootstrap process of anything, you have a lot of power. That's been true since computers were invented.
p_l 5y ago