8 ms·

CoreOS: Boot on Bare Metal with PXE

145 points by philips 13y ago

wmf 13y ago
Since I asked for this in the original CoreOS thread, let me be the first to say thanks. I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. I recommend experimenting with PXE in Vagrant or on a separate physical network to avoid breaking your production DHCP.
- 286c8cb04bda 13y ago
  > BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. You don't even need a config file. Here's a command line snippet I have saved for testing PXE installs: $ sudo dnsmasq -hdq -i en0 -p0 --enable-tftp --tftp-root=`pwd` -Mpxelinux.0 -F10.10.10.100,10.10.10.199
  - samstave 13y ago
    So this snippet makes the machine you run it from into the PXE Server?
    - wmf 13y ago
      Yes.
- jlgaddis 13y ago
  > I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. Heh, we were doing this 20-25 years ago. It's funny sometimes how I.T. seems to circle back around.
  - nl 13y ago
    >> stateless immutable servers [emphasis added] >Heh, we were doing this 20-25 years ago. It's possible, but I'd be surprised (unless you mean X servers in the crazy X-windows server-is-really-what-any-sane-person-would-call-a-client sense). Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk.
    - derekp7 13y ago
      The NCD X terminals I used to work on did this (they actually ran a BSD Unix kernel under the hood, I believe). Also was popular for some Sun diskless workstations -- although the protocol was bootp instead of pxe.
      - bodyfour 13y ago
        > although the protocol was bootp instead of pxe Well, they used bootp followed by tftp, and so did most other UNIX workstations of that time period. DHCP is effectively a superset of bootp -- it uses the same UDP port. PXE is DHCP+tftp along with some specification of tftp paths, where the image is loaded in RAM, etc. Really the overall netboot mechanism is nearly identical as it was 25 years ago, just some protocol tweaks and a new acronym.
    - lsc 13y ago
      maybe not 25 years ago... but it was common in the early '00s. This is not new tech. I participated in a project to do this with FreeBSD and nfs, I think around '03 or so. It was a smallish cluster (under 100 nodes) built to handle a simple Freebsd Apache Mysql PHP stack for a growing realestate ASP. Even then, it was not a new thing; there was plenty of documentation from other folks doing it. I mean, it didn't go as smoothly as pxe booting does today; we ended up needing to reflash all our network cards to make it work, something I have not had to do in the last 5 years.
    - primitur 13y ago
      >Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk. Not really, no. Servers all over the IT world have been using PXE and tftpboot for decades - in fact it was a very common method of booting a server even in the 80's. I know of quite a few Rail/Transportation companies whose track-side computers are booted over the network using these techniques, and there are about 250,000 of those things just in Europe alone .. What's new is that another generation are discovering this technique.
- noelherrick 13y ago
  What are the upsides/downsides of this? It's such a paradigm shift, I'm having a hard time thinking of any advantages.
  - wmf 13y ago
    It's based on the concept of immutable servers http://martinfowler.com/bliki/ImmutableServer.html http://martinfowler.com/bliki/ImmutableServer.html which helps eliminate errors due to drift. Containers also have lower overhead than VMs.
hosay123 13y ago
Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).
- bretpiatt 13y ago
  I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
  - stephengillie 13y ago
    I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo. Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
    - baruch 13y ago
      TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it. The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.
- stephengillie 13y ago
  Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.
- 13y ago
vezzy-fnord 13y ago
So is this basically LFS with built-in PXE boot support and orientation towards servers?
WestCoastJustin 13y ago
For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network. A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and servers), can make a DHCP request (outside of an operating system), download via TFTP, then boot a kernel/initial ram disk, say for example an OS installer, a LIVE CD, or CoreOS. +-----> #1 PXE makes DHCP request, | redirects to TFTP server, | loads kernel/initial ram disk via TFTP | | +--+ #2 PXE boots kernel/initial ram disk | | + v +------------------+ | PXE Network Card | |------------------| | server | | hardware/os | +------------------+ So, with a little infrastructure (DHCP, TFTP, and elbow grease), you can boot many things over the network without even having a hard drive or cd-rom in a machine. I use this often for installing new desktops and servers. Just boot into the PXE menu and then select the installer that I want. Then you use something like puppet to configure the machine as needed. A typical RHEL install can take ~5 minutes. In summary, it looks like CoreOS provides you with their kernel and initial ram disk [2], then you can just boot the machine over the network without actually installing anything. Basically running everything out of RAM, just like a LIVE CD, most likely with the option to install to disk for persistent storage of your images, etc. ps. It is common to use NFS to mount persistent storage in these environments too, or if you are using a HPC environment, then Luster or something like that. [1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment http://en.wikipedia.org/wiki/Preboot_Execution_Environment [2] http://coreos.com/docs/pxe/ http://coreos.com/docs/pxe/
- ewams 13y ago
  It is possible to PXE boot a full blown OS, not just OS installer, Live CD or CoreOS. The product usually used to do this is: http://support.citrix.com/proddocs/topic/technologies/pvs-provisioning.html http://support.citrix.com/proddocs/topic/technologies/pvs-pr... It is just a nice GUI that ties into PXE and TFTP and just streams an image from a disk file. On a 1Gb network I have booted over 300 physical machines in under 2 minutes simultaneously. For those curious, unfortunately PVS is packaged with XenDesktop so you have to buy XenDesktop to get PVS. Citrix does not seem willing to separate the two as different products even though I have deployed PVS for several customers without even download the XenDestkop bits.
volokoumphetico 13y ago
from the title I thought somebody had discovered a way to run an operating system on just a thin sheet of metal
- johnpmayer 13y ago
  Bare metal is a common term used to describe running directly on the hardware. Such as a non-virtualized operating system, or a program running without an operating system. Or yeah, we could just downmod everybody.
  - volokoumphetico 13y ago
    wow tough crowd.
songgao 13y ago
Awesome! My favorite part is this: http://coreos.com/docs/pxe/#state-only-installation http://coreos.com/docs/pxe/#state-only-installation which would enable you to run CoreOS in memory only (loaded from PXE), but still store all of your containers on filesystem. It liberates me from having to install OS on a cluster, but still lets me use persistent storage.
sturadnidge 13y ago
I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure.
- lsc 13y ago
  >I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure. I find comments like these amusing. Sysadmins have been using PXE to boot servers... for quite some time now. Long before joyent existed, and long before VMware was something you'd seriously run a server under. (vmware came about around the time of the 2.1 version of the PXE standard, which was when I was first getting my feet wet; I didn't seriously start using pxe until the early oughts.) Hell, /I/ built a nfs/pxe diskless cluster before joyent existed. As part of that, I demo'd a 'initrd only' system like this coreos thing, only we were using FreeBSD. We ended up going with / on nfs; it was way easier to update. That said, I'm not knocking CoreOS; I might even use this. Maintaining your own bootable initrd with root filesystem is work. I currently use distro 'rescue images' (for centos, at least, you append 'rescue' to the installer, and it downloads a small initrd /... but it's less than optimal.) I mean, I'm not shitting on joyent, either; I think most of the value they bring is managing this shit ongoing, which is not a trivial amount of work. I mean, the whole idea behind companies like Joyent is to make is so you don't need a me screwing with your dhcp server, and there's value in that. I'm grouchy and charge a lot of money.
voltagex_ 13y ago
PXE support on some consumer boards is a mess - often I have to use iPXE [1] just to get them loading from TFTP reliably. Now I've played with UEFI PXE boot, and it seems to be even worse - instead of requesting an "x64 bootloader", the NIC seems to request a "UEFI bytecode bootloader" which I haven't been able to supply. [1]: http://ipxe.org/ http://ipxe.org/
jimmcslim 13y ago
Running CoreOS on Intel NUC's via PXE boot from a HP Microserver; a poor man's blade chassis/datacenter?
slynux 13y ago
I had written a Wifi based LTSP for Linux sometime ago. http://www.sarathlakshman.com/2010/03/14/wireless-ltsp/ http://www.sarathlakshman.com/2010/03/14/wireless-ltsp/
wiradikusuma 13y ago
Is this some 'lightweight' OS that you can use for Raspberry Pi?

145 points by philips 13y ago

wmf 13y ago

Since I asked for this in the original CoreOS thread, let me be the first to say thanks. I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. I recommend experimenting with PXE in Vagrant or on a separate physical network to avoid breaking your production DHCP.

286c8cb04bda 13y ago
> BTW, if people haven't tried PXE booting before, it's pretty easy with dnsmasq. You can basically read the sample config file and uncomment a few lines. You don't even need a config file. Here's a command line snippet I have saved for testing PXE installs: $ sudo dnsmasq -hdq -i en0 -p0 --enable-tftp --tftp-root=`pwd` -Mpxelinux.0 -F10.10.10.100,10.10.10.199
- samstave 13y ago
  So this snippet makes the machine you run it from into the PXE Server?
  - wmf 13y ago
    Yes.
jlgaddis 13y ago
> I think stateless immutable servers that boot from the network and run from RAM are going to be a great base layer to build on. Heh, we were doing this 20-25 years ago. It's funny sometimes how I.T. seems to circle back around.
- nl 13y ago
  >> stateless immutable servers [emphasis added] >Heh, we were doing this 20-25 years ago. It's possible, but I'd be surprised (unless you mean X servers in the crazy X-windows server-is-really-what-any-sane-person-would-call-a-client sense). Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk.
  - derekp7 13y ago
    The NCD X terminals I used to work on did this (they actually ran a BSD Unix kernel under the hood, I believe). Also was popular for some Sun diskless workstations -- although the protocol was bootp instead of pxe.
    - bodyfour 13y ago
      > although the protocol was bootp instead of pxe Well, they used bootp followed by tftp, and so did most other UNIX workstations of that time period. DHCP is effectively a superset of bootp -- it uses the same UDP port. PXE is DHCP+tftp along with some specification of tftp paths, where the image is loaded in RAM, etc. Really the overall netboot mechanism is nearly identical as it was 25 years ago, just some protocol tweaks and a new acronym.
  - lsc 13y ago
    maybe not 25 years ago... but it was common in the early '00s. This is not new tech. I participated in a project to do this with FreeBSD and nfs, I think around '03 or so. It was a smallish cluster (under 100 nodes) built to handle a simple Freebsd Apache Mysql PHP stack for a growing realestate ASP. Even then, it was not a new thing; there was plenty of documentation from other folks doing it. I mean, it didn't go as smoothly as pxe booting does today; we ended up needing to reflash all our network cards to make it work, something I have not had to do in the last 5 years.
  - primitur 13y ago
    >Doing this kind of thing is pretty new for servers. Even massive HPC clusters tend to boot from persistent disk. Not really, no. Servers all over the IT world have been using PXE and tftpboot for decades - in fact it was a very common method of booting a server even in the 80's. I know of quite a few Rail/Transportation companies whose track-side computers are booted over the network using these techniques, and there are about 250,000 of those things just in Europe alone .. What's new is that another generation are discovering this technique.
noelherrick 13y ago
What are the upsides/downsides of this? It's such a paradigm shift, I'm having a hard time thinking of any advantages.
- wmf 13y ago
  It's based on the concept of immutable servers http://martinfowler.com/bliki/ImmutableServer.html http://martinfowler.com/bliki/ImmutableServer.html which helps eliminate errors due to drift. Containers also have lower overhead than VMs.

hosay123 13y ago

Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF. Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).

bretpiatt 13y ago
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
- stephengillie 13y ago
  I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo. Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.
  - baruch 13y ago
    TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it. The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.
stephengillie 13y ago
Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.
13y ago

vezzy-fnord 13y ago

So is this basically LFS with built-in PXE boot support and orientation towards servers?

WestCoastJustin 13y ago

For anyone who does not know what PXE (pixie) boot is, it is an environment to boot computers using a network interface independently of data storage devices (like hard disks) or installed operating systems. [1] In the BIOS, rather than booting from CD, or Hard disk, you would select Network. A very simplified explanation is that the PXE enabled network card (almost all modern network cards support PXE, desktop and servers), can make a DHCP request (outside of an operating system), download via TFTP, then boot a kernel/initial ram disk, say for example an OS installer, a LIVE CD, or CoreOS. +-----> #1 PXE makes DHCP request, | redirects to TFTP server, | loads kernel/initial ram disk via TFTP | | +--+ #2 PXE boots kernel/initial ram disk | | + v +------------------+ | PXE Network Card | |------------------| | server | | hardware/os | +------------------+ So, with a little infrastructure (DHCP, TFTP, and elbow grease), you can boot many things over the network without even having a hard drive or cd-rom in a machine. I use this often for installing new desktops and servers. Just boot into the PXE menu and then select the installer that I want. Then you use something like puppet to configure the machine as needed. A typical RHEL install can take ~5 minutes. In summary, it looks like CoreOS provides you with their kernel and initial ram disk [2], then you can just boot the machine over the network without actually installing anything. Basically running everything out of RAM, just like a LIVE CD, most likely with the option to install to disk for persistent storage of your images, etc. ps. It is common to use NFS to mount persistent storage in these environments too, or if you are using a HPC environment, then Luster or something like that. [1] http://en.wikipedia.org/wiki/Preboot_Execution_Environment http://en.wikipedia.org/wiki/Preboot_Execution_Environment [2] http://coreos.com/docs/pxe/ http://coreos.com/docs/pxe/

ewams 13y ago
It is possible to PXE boot a full blown OS, not just OS installer, Live CD or CoreOS. The product usually used to do this is: http://support.citrix.com/proddocs/topic/technologies/pvs-provisioning.html http://support.citrix.com/proddocs/topic/technologies/pvs-pr... It is just a nice GUI that ties into PXE and TFTP and just streams an image from a disk file. On a 1Gb network I have booted over 300 physical machines in under 2 minutes simultaneously. For those curious, unfortunately PVS is packaged with XenDesktop so you have to buy XenDesktop to get PVS. Citrix does not seem willing to separate the two as different products even though I have deployed PVS for several customers without even download the XenDestkop bits.

volokoumphetico 13y ago

from the title I thought somebody had discovered a way to run an operating system on just a thin sheet of metal

johnpmayer 13y ago
Bare metal is a common term used to describe running directly on the hardware. Such as a non-virtualized operating system, or a program running without an operating system. Or yeah, we could just downmod everybody.
- volokoumphetico 13y ago
  wow tough crowd.

songgao 13y ago

Awesome! My favorite part is this: http://coreos.com/docs/pxe/#state-only-installation http://coreos.com/docs/pxe/#state-only-installation which would enable you to run CoreOS in memory only (loaded from PXE), but still store all of your containers on filesystem. It liberates me from having to install OS on a cluster, but still lets me use persistent storage.

sturadnidge 13y ago

I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure.

lsc 13y ago
>I'm really glad to see CoreOS taking this path, forged by the likes of VMware's ESXi and Joyent's SmartOS. It truly is the only way to run scalable infrastructure. I find comments like these amusing. Sysadmins have been using PXE to boot servers... for quite some time now. Long before joyent existed, and long before VMware was something you'd seriously run a server under. (vmware came about around the time of the 2.1 version of the PXE standard, which was when I was first getting my feet wet; I didn't seriously start using pxe until the early oughts.) Hell, /I/ built a nfs/pxe diskless cluster before joyent existed. As part of that, I demo'd a 'initrd only' system like this coreos thing, only we were using FreeBSD. We ended up going with / on nfs; it was way easier to update. That said, I'm not knocking CoreOS; I might even use this. Maintaining your own bootable initrd with root filesystem is work. I currently use distro 'rescue images' (for centos, at least, you append 'rescue' to the installer, and it downloads a small initrd /... but it's less than optimal.) I mean, I'm not shitting on joyent, either; I think most of the value they bring is managing this shit ongoing, which is not a trivial amount of work. I mean, the whole idea behind companies like Joyent is to make is so you don't need a me screwing with your dhcp server, and there's value in that. I'm grouchy and charge a lot of money.

voltagex_ 13y ago

PXE support on some consumer boards is a mess - often I have to use iPXE [1] just to get them loading from TFTP reliably. Now I've played with UEFI PXE boot, and it seems to be even worse - instead of requesting an "x64 bootloader", the NIC seems to request a "UEFI bytecode bootloader" which I haven't been able to supply. [1]: http://ipxe.org/ http://ipxe.org/

jimmcslim 13y ago

Running CoreOS on Intel NUC's via PXE boot from a HP Microserver; a poor man's blade chassis/datacenter?

slynux 13y ago

I had written a Wifi based LTSP for Linux sometime ago. http://www.sarathlakshman.com/2010/03/14/wireless-ltsp/ http://www.sarathlakshman.com/2010/03/14/wireless-ltsp/

wiradikusuma 13y ago

Is this some 'lightweight' OS that you can use for Raspberry Pi?