5 ms·

Boot legacy PCs from NVMe storage

117 points by wmlive 3y ago

wmlive 3y ago
»This project allows old x86 computers using a classic BIOS to boot from modern NVMe storage attached via PCI(e). It's a heavily modified version of iPXE (which usually allows for booting from the network), but instead of the network, this code uses a port of the SeaBIOS NVMe implementation to talk to a local NVMe drive.«
icu 3y ago
You can do this with Linux easily as long as your bios allows you to boot from USB. Just put /boot on the USB and everything else on the NVMe (via PCI Express to NVMe card). This trick has allowed me to save several old PCs and use them as home servers and for my home lab.
- Havoc 3y ago
  > Just put /boot on the USB and everything else on the NVMe Hold up - that works? I’ve got a SBC device that has to boot off sd card but has ssd attached. Any pointers as to where I could read up more on how to split a fs system like that? I can copy files easily enough but presumably needs so additional magic to link/integrate it
  - jakogut 3y ago
    Believe it or not, you don't even need a physical disk to boot at all (on many systems). The first step in the boot process is loading the boot firmware. On a BIOS/UEFI PC, this gives you Preboot Execution Environment (PXE) [0]. Raspberry Pi boards have something similar since the Pi 3, though it's not exactly PXE. Network booting allows you to setup a DHCP server in a way that automatically serves the machine a kernel, initramfs, and cmdline over the network. You can run a full Linux that way, though you probably want to switch_root [1] to another filesystem that has a block device backing it with more storage, for practical purposes. This is what most initramfs images do. Once the machine can automatically retrieve those three, you're set. The initramfs can be built with whatever binaries, firmware, and kernel modules you need to initialize the network and set up an NBD or NFS share. Add `root=/dev/nbd0` or whatever, and you're running over the network. Same concept applies to booting with a split boot partition. The boot partition contains the kernel, initramfs, and bootloader, along with the boot configuration. The kernel cmdline specifies which disk contains the root filesystem. It need not be on the same disk as the kernel, or even on the same machine. [0] https://en.wikipedia.org/wiki/Preboot_Execution_Environment https://en.wikipedia.org/wiki/Preboot_Execution_Environment [1] https://landley.net/writing/rootfs-programming.html https://landley.net/writing/rootfs-programming.html
    - Havoc 3y ago
      Many thanks for the detailed response. This is probably the part that was missing in my mind >The kernel cmdline specifies which disk contains the root filesystem.
      - jakogut 3y ago
        Absolutely. It's a part of the kernel I find very interesting. Here's a related bit from the kernel docs that describes the initramfs and switch_root in more detail. https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.rst https://www.kernel.org/doc/Documentation/filesystems/ramfs-r... Interestingly, there's even a mechanism for leaving programs running even after exiting the initramfs.
  - michrassena 3y ago
    I'm doing something similar with the VisionFive 2, boot is on the SD card, the NVME does the rest.
    - snvzz 3y ago
      Note that current u-boot (latest official fw release, u-boot patches already upstreamed) can boot straight from NVME.
  - numpad0 3y ago
    Possibly you may be get by installing normally on NVMe; `cp -R /mnt/nvme0p1/boot /mnt/mmcblk0p1/boot`; `sudo reboot`. There shall be requisite kernel modules and /etc/fstab inside initramfs to find `nvme0` and chroot into it. It ultimately should not matter how Kernel + initramfs finds itself on RAM, or whether it was loaded from what it thinks it was loaded from. Kernel is just a baremetal program, and BIOS/firmware is just the least elaborate tool to make Kernel appear on RAM.
dmw_ng 3y ago
More interested in how the PCIe breakout board being used works.. is there some kind of bridge to a mini PCI slot?
- rzzzt 3y ago
  The "classic" ones have a small adapter inside the computer case (this can be ExpressCard, a mini PCIe or an M.2 plug). They'll use either HDMI or USB 3 connectors and cabling as these can maintain reasonable signal integrity; this plugs in to the regular sized PCIe slot on the outside. There's also a barrel jack and/or a PCIe PSU connector on them for power. I don't think there's any smartness hiding inside of these, the connector and number of conductors will limit you to a ×2 or ×4 connection anyways.
- snvzz 3y ago
  Explaining Computers covered them in a recent video[0]. 0. https://www.youtube.com/watch?v=Ktqa11VT1xI https://www.youtube.com/watch?v=Ktqa11VT1xI
RulerOf 3y ago
This is a really interesting approach, as embedding into an option ROM means you could potentially flash it into a modded BIOS. My preferred method to boot NVMe on a legacy system though is to boot tianocore UEFI with an NVME module in it[1]. This is a UEFI firmware that you can chainload from a legacy bios. Extremely good compatibility this way, and a great option for any system with an unused usb header. 1: https://winraid.level1techs.com/t/guide-nvme-boot-for-systems-with-legacy-bios-and-uefi-board-duet-refind/32251 https://winraid.level1techs.com/t/guide-nvme-boot-for-system...
- undersuit 3y ago
  I knew that you could boot an "EFI bios" from experience with rEFIt from my time as a Mac Admin, so when I got my rEFInd USB drive working with a NVME drive on a pci-e adapter I was ecstatic until a month later when the USB drive failed and I forgot everything I had done. Make backups of that USB drive or yeah it would be super cool if you could leverage the bios chip on a motherboard.
  - Tijdreiziger 3y ago
    Yeah, I have one of the rare old laptops (HP ZBook G2) that has a PCIe M.2 slot for storage drives, but no support for NVMe booting. This means that the only M.2 drives it’ll boot from are non-NVMe PCIe drives, which were actually made as an HP OEM part (Z Turbo Drive) for this specific model of laptop, but are otherwise nonexistent. It would be really nice if I could somehow add NVMe support to the laptop’s UEFI, but AFAIK there are no existing solutions for this, other than chainloading from a second drive. The subsequent model (ZBook G3) actually does support NVMe, but HP never backported that support to the G2/G1 (of course, according to them, if you simply use the specified OEM parts there’s no problem…). Oh well, 2.5” SATA drives work just fine for such an old beast.
    - RulerOf 3y ago
      Definitely an interesting problem. If you're interested in going down the rabbit hole, some light googling suggests that HP uses an AMI BIOS. You might be able to mod it: https://winraid.level1techs.com/t/howto-get-full-nvme-support-for-all-systems-with-an-ami-uefi-bios/30901 https://winraid.level1techs.com/t/howto-get-full-nvme-suppor...
userbinator 3y ago
Looks like an interesting project, but upon seeing that screenshot and this... It's a heavily modified version of iPXE (which usually allows for booting from the network), but instead of the network, this code uses a port of the SeaBIOS NVMe implementation to talk to a local NVMe drive. I can't help but feel like that's a very roundabout way of doing it (especially the fact that it seems to be manual?); after all, the BIOS Boot Specification exists and it's what lets PCs boot from things like SCSI and other block devices.
- toast0 3y ago
  Assuming the NVMe manufacturers neglected to support the BIOS Boot Specification themselves, you've got to put some code into your option rom... And it may as well be a modified iPXE that has NVMe support? iPXE can be scripted it you want the whole process automated, but maybe that doesn't make as nice of a demo?
neilv 3y ago
Their example of a ThinkPad T43 is an interesting choice, because it was a rare model that used a problematic SATA-PATA bridge kludge for storage. https://www.thinkwiki.org/wiki/Problem_with_non-ThinkPad_hard_disks https://www.thinkwiki.org/wiki/Problem_with_non-ThinkPad_har... It couldn't even take a proper hard drive when new, 18 years ago, but now it can boot from NVMe.
sheepscreek 3y ago
Using network boot to inject a custom loader is ingenious! Was literally faced with this today. My AMD motherboard circa 2012 did not recognize the PCIe NVMe SSD. It was rather slow on this PC for some reason anyway (yes, I did my due diligence). Ended up putting it in a laptop which was thankfully able to muster a higher throughput.
deleted 3y ago
[deleted]
lol_catz 3y ago
Cant you just use a m.2 to 2.5 inch sata dummy drive?
AshamedCaptain 3y ago
A lot of NVME drives already ship such an option ROM. This has only become a problem relatively recently now that most SSDs do not care to ship one (or ship untested ones).

117 points by wmlive 3y ago

»This project allows old x86 computers using a classic BIOS to boot from modern NVMe storage attached via PCI(e). It's a heavily modified version of iPXE (which usually allows for booting from the network), but instead of the network, this code uses a port of the SeaBIOS NVMe implementation to talk to a local NVMe drive.«

icu 3y ago

You can do this with Linux easily as long as your bios allows you to boot from USB. Just put /boot on the USB and everything else on the NVMe (via PCI Express to NVMe card). This trick has allowed me to save several old PCs and use them as home servers and for my home lab.

Havoc 3y ago
> Just put /boot on the USB and everything else on the NVMe Hold up - that works? I’ve got a SBC device that has to boot off sd card but has ssd attached. Any pointers as to where I could read up more on how to split a fs system like that? I can copy files easily enough but presumably needs so additional magic to link/integrate it
- jakogut 3y ago
  Believe it or not, you don't even need a physical disk to boot at all (on many systems). The first step in the boot process is loading the boot firmware. On a BIOS/UEFI PC, this gives you Preboot Execution Environment (PXE) [0]. Raspberry Pi boards have something similar since the Pi 3, though it's not exactly PXE. Network booting allows you to setup a DHCP server in a way that automatically serves the machine a kernel, initramfs, and cmdline over the network. You can run a full Linux that way, though you probably want to switch_root [1] to another filesystem that has a block device backing it with more storage, for practical purposes. This is what most initramfs images do. Once the machine can automatically retrieve those three, you're set. The initramfs can be built with whatever binaries, firmware, and kernel modules you need to initialize the network and set up an NBD or NFS share. Add `root=/dev/nbd0` or whatever, and you're running over the network. Same concept applies to booting with a split boot partition. The boot partition contains the kernel, initramfs, and bootloader, along with the boot configuration. The kernel cmdline specifies which disk contains the root filesystem. It need not be on the same disk as the kernel, or even on the same machine. [0] https://en.wikipedia.org/wiki/Preboot_Execution_Environment https://en.wikipedia.org/wiki/Preboot_Execution_Environment [1] https://landley.net/writing/rootfs-programming.html https://landley.net/writing/rootfs-programming.html
  - Havoc 3y ago
    Many thanks for the detailed response. This is probably the part that was missing in my mind >The kernel cmdline specifies which disk contains the root filesystem.
    - jakogut 3y ago
      Absolutely. It's a part of the kernel I find very interesting. Here's a related bit from the kernel docs that describes the initramfs and switch_root in more detail. https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.rst https://www.kernel.org/doc/Documentation/filesystems/ramfs-r... Interestingly, there's even a mechanism for leaving programs running even after exiting the initramfs.
- michrassena 3y ago
  I'm doing something similar with the VisionFive 2, boot is on the SD card, the NVME does the rest.
  - snvzz 3y ago
    Note that current u-boot (latest official fw release, u-boot patches already upstreamed) can boot straight from NVME.
- numpad0 3y ago
  Possibly you may be get by installing normally on NVMe; `cp -R /mnt/nvme0p1/boot /mnt/mmcblk0p1/boot`; `sudo reboot`. There shall be requisite kernel modules and /etc/fstab inside initramfs to find `nvme0` and chroot into it. It ultimately should not matter how Kernel + initramfs finds itself on RAM, or whether it was loaded from what it thinks it was loaded from. Kernel is just a baremetal program, and BIOS/firmware is just the least elaborate tool to make Kernel appear on RAM.

dmw_ng 3y ago

More interested in how the PCIe breakout board being used works.. is there some kind of bridge to a mini PCI slot?

rzzzt 3y ago
The "classic" ones have a small adapter inside the computer case (this can be ExpressCard, a mini PCIe or an M.2 plug). They'll use either HDMI or USB 3 connectors and cabling as these can maintain reasonable signal integrity; this plugs in to the regular sized PCIe slot on the outside. There's also a barrel jack and/or a PCIe PSU connector on them for power. I don't think there's any smartness hiding inside of these, the connector and number of conductors will limit you to a ×2 or ×4 connection anyways.
snvzz 3y ago
Explaining Computers covered them in a recent video[0]. 0. https://www.youtube.com/watch?v=Ktqa11VT1xI https://www.youtube.com/watch?v=Ktqa11VT1xI

RulerOf 3y ago

This is a really interesting approach, as embedding into an option ROM means you could potentially flash it into a modded BIOS. My preferred method to boot NVMe on a legacy system though is to boot tianocore UEFI with an NVME module in it[1]. This is a UEFI firmware that you can chainload from a legacy bios. Extremely good compatibility this way, and a great option for any system with an unused usb header. 1: https://winraid.level1techs.com/t/guide-nvme-boot-for-systems-with-legacy-bios-and-uefi-board-duet-refind/32251 https://winraid.level1techs.com/t/guide-nvme-boot-for-system...

undersuit 3y ago
I knew that you could boot an "EFI bios" from experience with rEFIt from my time as a Mac Admin, so when I got my rEFInd USB drive working with a NVME drive on a pci-e adapter I was ecstatic until a month later when the USB drive failed and I forgot everything I had done. Make backups of that USB drive or yeah it would be super cool if you could leverage the bios chip on a motherboard.
- Tijdreiziger 3y ago
  Yeah, I have one of the rare old laptops (HP ZBook G2) that has a PCIe M.2 slot for storage drives, but no support for NVMe booting. This means that the only M.2 drives it’ll boot from are non-NVMe PCIe drives, which were actually made as an HP OEM part (Z Turbo Drive) for this specific model of laptop, but are otherwise nonexistent. It would be really nice if I could somehow add NVMe support to the laptop’s UEFI, but AFAIK there are no existing solutions for this, other than chainloading from a second drive. The subsequent model (ZBook G3) actually does support NVMe, but HP never backported that support to the G2/G1 (of course, according to them, if you simply use the specified OEM parts there’s no problem…). Oh well, 2.5” SATA drives work just fine for such an old beast.
  - RulerOf 3y ago
    Definitely an interesting problem. If you're interested in going down the rabbit hole, some light googling suggests that HP uses an AMI BIOS. You might be able to mod it: https://winraid.level1techs.com/t/howto-get-full-nvme-support-for-all-systems-with-an-ami-uefi-bios/30901 https://winraid.level1techs.com/t/howto-get-full-nvme-suppor...

userbinator 3y ago

Looks like an interesting project, but upon seeing that screenshot and this... It's a heavily modified version of iPXE (which usually allows for booting from the network), but instead of the network, this code uses a port of the SeaBIOS NVMe implementation to talk to a local NVMe drive. I can't help but feel like that's a very roundabout way of doing it (especially the fact that it seems to be manual?); after all, the BIOS Boot Specification exists and it's what lets PCs boot from things like SCSI and other block devices.

toast0 3y ago
Assuming the NVMe manufacturers neglected to support the BIOS Boot Specification themselves, you've got to put some code into your option rom... And it may as well be a modified iPXE that has NVMe support? iPXE can be scripted it you want the whole process automated, but maybe that doesn't make as nice of a demo?

neilv 3y ago

Their example of a ThinkPad T43 is an interesting choice, because it was a rare model that used a problematic SATA-PATA bridge kludge for storage. https://www.thinkwiki.org/wiki/Problem_with_non-ThinkPad_hard_disks https://www.thinkwiki.org/wiki/Problem_with_non-ThinkPad_har... It couldn't even take a proper hard drive when new, 18 years ago, but now it can boot from NVMe.

sheepscreek 3y ago

Using network boot to inject a custom loader is ingenious! Was literally faced with this today. My AMD motherboard circa 2012 did not recognize the PCIe NVMe SSD. It was rather slow on this PC for some reason anyway (yes, I did my due diligence). Ended up putting it in a laptop which was thankfully able to muster a higher throughput.

deleted 3y ago

[deleted]

lol_catz 3y ago

Cant you just use a m.2 to 2.5 inch sata dummy drive?

AshamedCaptain 3y ago

A lot of NVME drives already ship such an option ROM. This has only become a problem relatively recently now that most SSDs do not care to ship one (or ship untested ones).