6 ms·

Really good talk by Bryan Cantrill (co-founder of Oxide Computer COmpany) on the problems with closed source firmware and the need to move away from BIOS and EF

Really good talk by Bryan Cantrill (co-founder of Oxide Computer COmpany) on the problems with closed source firmware and the need to move away from BIOS and EFI and how they booted their own x86 AMD custom board system all made with open source firmware and without using AMD's firmware.

convolvatron 4y ago
I've worked on this before. We had explicit help from AMD - their documentation is better than Intels for this stuff, but still. if it weren't for linux's dependence on the bios pci and memory probes, you could really cut out all that crap. you need to program the memory controllers and bring in the kernel from storage.
- Teknoman117 4y ago
  Is Linux actually dependent on the firmware's PCIe probe? You can have the kernel do probe and enumeration if you want. (Outside of the process of getting into the kernel. Most x86 users are going to be booting off of storage that's behind PCIe somehow.) Edit - for clarity, I mean calling back into firmware after boot for PCIe functionality.
  - convolvatron 4y ago
    oh no, my recollection was that at least at the time, linux used the apci pci traversal to start driver discovery instead of doing its own. obviously after that it assumes complete control of the device tree.
pclmulqdq 4y ago
I have booted a few SoCs without a BIOS or anything of the sort before (nothing nearly as big as an AMD Milan chip). Doing this with a huge, fast chip is really impressive from the folks at Oxide. DDR5 DRAM training is also a ridiculously complicated and touchy exercise, as is some of the PCIe link training that (according to the talk) Linux/Unix handles. Edit: this board uses DDR4, not DDR5.
- mlindner 4y ago
  Apparently the DRAM training is apparently one piece of AMD firmware I believe they re-used. Though I had trouble following that part. I think they said that, but they only didn't do it because they didn't need to because they had to pick and choose battles. Edit: Yes, I believe he says they used the AMD firmware for the PSP (Platform Security Processor). Edit2: This post may actually be incorrect. Please go watch the talk. I'm not sure anymore.
  - bcantrill 4y ago
    No, you're correct: the PSP does DIMM training. Also, note that this is AMD Milan, so it's DDR4, not DDR5 -- DDR5 is still forthcoming from both AMD and Intel.
    - pclmulqdq 4y ago
      DDR4 link training is a bit easier to comprehend than DDR5, but still a headache. I think almost every high-performance SoC uses an auxiliary processing core for link training at this point (including large FPGAs). I have been waiting for someone to find a security vulnerability in one of these cores.
    - mlindner 4y ago
      One part I couldn't follow on your talk is which software is Oxide designed and which is vendor supplied software? Have you really stripped out every piece of vendor software from the product? Or are there still some parts that are vendor supplied black boxes? The question after your talk implied that DIMM training was still being done by non-Oxide software.
      - bcantrill 4y ago
        On AMD, DIMM training is done by the PSP. And you have to run the PSP to run the SoC, so we have no alternative there. More generally, we have implemented and opened everything we can; there still remain opaque bits like the PSP, as well as some smaller bits scattered through the machine (e.g., SSD firmware, MCU boot ROMs, VR firmware, etc.). We have endeavored to make as open a system as possible within the constraint of not making our own silicon. And while we haven't talked about it publicly, we will also open our schematics when we ship the rack, allowing everyone to see every component in the BOM. While we do have some (necessary) proprietary blobs in the system, we want to at least be transparant about where they are!

convolvatron 4y ago

I've worked on this before. We had explicit help from AMD - their documentation is better than Intels for this stuff, but still. if it weren't for linux's dependence on the bios pci and memory probes, you could really cut out all that crap. you need to program the memory controllers and bring in the kernel from storage.

Teknoman117 4y ago
Is Linux actually dependent on the firmware's PCIe probe? You can have the kernel do probe and enumeration if you want. (Outside of the process of getting into the kernel. Most x86 users are going to be booting off of storage that's behind PCIe somehow.) Edit - for clarity, I mean calling back into firmware after boot for PCIe functionality.
- convolvatron 4y ago
  oh no, my recollection was that at least at the time, linux used the apci pci traversal to start driver discovery instead of doing its own. obviously after that it assumes complete control of the device tree.

pclmulqdq 4y ago

I have booted a few SoCs without a BIOS or anything of the sort before (nothing nearly as big as an AMD Milan chip). Doing this with a huge, fast chip is really impressive from the folks at Oxide. DDR5 DRAM training is also a ridiculously complicated and touchy exercise, as is some of the PCIe link training that (according to the talk) Linux/Unix handles. Edit: this board uses DDR4, not DDR5.

mlindner 4y ago
Apparently the DRAM training is apparently one piece of AMD firmware I believe they re-used. Though I had trouble following that part. I think they said that, but they only didn't do it because they didn't need to because they had to pick and choose battles. Edit: Yes, I believe he says they used the AMD firmware for the PSP (Platform Security Processor). Edit2: This post may actually be incorrect. Please go watch the talk. I'm not sure anymore.
- bcantrill 4y ago
  No, you're correct: the PSP does DIMM training. Also, note that this is AMD Milan, so it's DDR4, not DDR5 -- DDR5 is still forthcoming from both AMD and Intel.
  - pclmulqdq 4y ago
    DDR4 link training is a bit easier to comprehend than DDR5, but still a headache. I think almost every high-performance SoC uses an auxiliary processing core for link training at this point (including large FPGAs). I have been waiting for someone to find a security vulnerability in one of these cores.
  - mlindner 4y ago
    One part I couldn't follow on your talk is which software is Oxide designed and which is vendor supplied software? Have you really stripped out every piece of vendor software from the product? Or are there still some parts that are vendor supplied black boxes? The question after your talk implied that DIMM training was still being done by non-Oxide software.
    - bcantrill 4y ago
      On AMD, DIMM training is done by the PSP. And you have to run the PSP to run the SoC, so we have no alternative there. More generally, we have implemented and opened everything we can; there still remain opaque bits like the PSP, as well as some smaller bits scattered through the machine (e.g., SSD firmware, MCU boot ROMs, VR firmware, etc.). We have endeavored to make as open a system as possible within the constraint of not making our own silicon. And while we haven't talked about it publicly, we will also open our schematics when we ship the rack, allowing everyone to see every component in the BOM. While we do have some (necessary) proprietary blobs in the system, we want to at least be transparant about where they are!