4 ms·

Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds

Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).

bretpiatt 13y ago
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
- stephengillie 13y ago
  I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo. Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
  - baruch 13y ago
    TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it. The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.
stephengillie 13y ago
Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.
devicenull 13y ago
So, a 'PXE server' is nothing more then a DHCP server and some method of fetching files (historically, TFTP, but this is not a requirement). ISC DHCPD is pretty bulletproof. Many large carriers/ISPs use it, so scaling this end should be pretty straightforward. You'd probably want to statically configure leases for all your servers though. Pretty much all PXE roms built into NICs only support TFTP. I would use TFTP to load a iPXE, then use iPXE to load the operating system over something else (probably HTTP). Scaling HTTP is well understood (this is about the easiest HTTP scaling you can do, downloading a few static files). Nothing about this is hard to scale, nor does it require any special network setup.
- zdw 13y ago
  iPXE is awesome. My standard install is to embed a script into iPXE that chainloads a config file over TFTP (this works properly with non-ISC DHCPD servers, like OpenBSD's fork and dnsmasq), then loads various OS's and diagnostic tools over HTTP or NFS. So much easier than slinging around optical media or hard disks for installation/diagnostic/booting diskless clients.
fintler 13y ago
We commonly reboot entire clusters at once (around 10,000 servers in larger clusters -- each running a full Linux OS) over PXE without a problem. We have a configuration management machine that creates an image, then we push that down to a small cluster of TFTP servers that serve it out. The strain on NFS (we keep parts of the OS in RAM, and load other parts on demand over NFS) after we kexec from the PXE kernel into the production kernel causes more problems than the initial TFTP traffic (but it usually works fine as well). Btw, after booting, we use PanFS (DirectFlow) or Lustre for computing stuff, not NFS. Although it's not what we use, here's a program that does a similar type of management: http://warewulf.lbl.gov/trac http://warewulf.lbl.gov/trac If you take the time to combine Warewulf with something like Puppet or Chef, you'll have a nice system for managing 100s of thousands of machines (I could easily see this scaling to over a million servers if you have the cash to build something like that). If you're wondering about dynamic libraries in an environment like this, take a look at https://github.com/hpc/Spindle https://github.com/hpc/Spindle And yes, I still get giddy when I type one command to reboot 10,000 servers.
- ajdecon 13y ago
  Since you mention kexec and TFTP+NFS, are you currently using Perceus? Or is there another system out there with that combo?
  - fintler 13y ago
    We're using a modified Perceus tied into cfengine.
ook 13y ago
Use iPxe (http://ipxe.org/ http://ipxe.org/) to boot from a http image server.
jlgaddis 13y ago
> In reality it's a pointless SPOF. Like pretty much anything else, it's not a problem when implemented correctly. Even if you have hundreds or thousands of machines, it's not that difficult to configure your network switches (assuming you don't buy your network gear at Office Depot) to distribute the DHCP and TFTP across multiple servers -- if you even need to, that is; DHCP and TFTP are two of the most lightweight networking protocols in use.
pkj 13y ago
If you are ok with CentOS/RHEL, there is a scalable way to boot multiple machines simulataneously using ........ bittorrent ! http://www.rocksclusters.org/rocks-doc/papers/two-pager/paper.pdf http://www.rocksclusters.org/rocks-doc/papers/two-pager/pape... Rocks is used at production cluster installations with thousands of servers. http://www.rocksclusters.org/rocks-register/index.php?sortby=CPUs&sortorder=down http://www.rocksclusters.org/rocks-register/index.php?sortby...

bretpiatt 13y ago

I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.

stephengillie 13y ago
I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo. Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
- baruch 13y ago
  TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it. The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.

stephengillie 13y ago

Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.

devicenull 13y ago

So, a 'PXE server' is nothing more then a DHCP server and some method of fetching files (historically, TFTP, but this is not a requirement). ISC DHCPD is pretty bulletproof. Many large carriers/ISPs use it, so scaling this end should be pretty straightforward. You'd probably want to statically configure leases for all your servers though. Pretty much all PXE roms built into NICs only support TFTP. I would use TFTP to load a iPXE, then use iPXE to load the operating system over something else (probably HTTP). Scaling HTTP is well understood (this is about the easiest HTTP scaling you can do, downloading a few static files). Nothing about this is hard to scale, nor does it require any special network setup.

zdw 13y ago
iPXE is awesome. My standard install is to embed a script into iPXE that chainloads a config file over TFTP (this works properly with non-ISC DHCPD servers, like OpenBSD's fork and dnsmasq), then loads various OS's and diagnostic tools over HTTP or NFS. So much easier than slinging around optical media or hard disks for installation/diagnostic/booting diskless clients.

fintler 13y ago

We commonly reboot entire clusters at once (around 10,000 servers in larger clusters -- each running a full Linux OS) over PXE without a problem. We have a configuration management machine that creates an image, then we push that down to a small cluster of TFTP servers that serve it out. The strain on NFS (we keep parts of the OS in RAM, and load other parts on demand over NFS) after we kexec from the PXE kernel into the production kernel causes more problems than the initial TFTP traffic (but it usually works fine as well). Btw, after booting, we use PanFS (DirectFlow) or Lustre for computing stuff, not NFS. Although it's not what we use, here's a program that does a similar type of management: http://warewulf.lbl.gov/trac http://warewulf.lbl.gov/trac If you take the time to combine Warewulf with something like Puppet or Chef, you'll have a nice system for managing 100s of thousands of machines (I could easily see this scaling to over a million servers if you have the cash to build something like that). If you're wondering about dynamic libraries in an environment like this, take a look at https://github.com/hpc/Spindle https://github.com/hpc/Spindle And yes, I still get giddy when I type one command to reboot 10,000 servers.

ajdecon 13y ago
Since you mention kexec and TFTP+NFS, are you currently using Perceus? Or is there another system out there with that combo?
- fintler 13y ago
  We're using a modified Perceus tied into cfengine.

ook 13y ago

Use iPxe (http://ipxe.org/ http://ipxe.org/) to boot from a http image server.

jlgaddis 13y ago

> In reality it's a pointless SPOF. Like pretty much anything else, it's not a problem when implemented correctly. Even if you have hundreds or thousands of machines, it's not that difficult to configure your network switches (assuming you don't buy your network gear at Office Depot) to distribute the DHCP and TFTP across multiple servers -- if you even need to, that is; DHCP and TFTP are two of the most lightweight networking protocols in use.

pkj 13y ago

If you are ok with CentOS/RHEL, there is a scalable way to boot multiple machines simulataneously using ........ bittorrent ! http://www.rocksclusters.org/rocks-doc/papers/two-pager/paper.pdf http://www.rocksclusters.org/rocks-doc/papers/two-pager/pape... Rocks is used at production cluster installations with thousands of servers. http://www.rocksclusters.org/rocks-register/index.php?sortby=CPUs&sortorder=down http://www.rocksclusters.org/rocks-register/index.php?sortby...