11 ms·

I have come to bury the BIOS, not to open it [video]

287 points by pclmulqdq 4y ago

mlindner 4y ago
Really good talk by Bryan Cantrill (co-founder of Oxide Computer COmpany) on the problems with closed source firmware and the need to move away from BIOS and EFI and how they booted their own x86 AMD custom board system all made with open source firmware and without using AMD's firmware.
- convolvatron 4y ago
  I've worked on this before. We had explicit help from AMD - their documentation is better than Intels for this stuff, but still. if it weren't for linux's dependence on the bios pci and memory probes, you could really cut out all that crap. you need to program the memory controllers and bring in the kernel from storage.
  - Teknoman117 4y ago
    Is Linux actually dependent on the firmware's PCIe probe? You can have the kernel do probe and enumeration if you want. (Outside of the process of getting into the kernel. Most x86 users are going to be booting off of storage that's behind PCIe somehow.) Edit - for clarity, I mean calling back into firmware after boot for PCIe functionality.
    - convolvatron 4y ago
      oh no, my recollection was that at least at the time, linux used the apci pci traversal to start driver discovery instead of doing its own. obviously after that it assumes complete control of the device tree.
- pclmulqdq 4y ago
  I have booted a few SoCs without a BIOS or anything of the sort before (nothing nearly as big as an AMD Milan chip). Doing this with a huge, fast chip is really impressive from the folks at Oxide. DDR5 DRAM training is also a ridiculously complicated and touchy exercise, as is some of the PCIe link training that (according to the talk) Linux/Unix handles. Edit: this board uses DDR4, not DDR5.
  - mlindner 4y ago
    Apparently the DRAM training is apparently one piece of AMD firmware I believe they re-used. Though I had trouble following that part. I think they said that, but they only didn't do it because they didn't need to because they had to pick and choose battles. Edit: Yes, I believe he says they used the AMD firmware for the PSP (Platform Security Processor). Edit2: This post may actually be incorrect. Please go watch the talk. I'm not sure anymore.
    - bcantrill 4y ago
      No, you're correct: the PSP does DIMM training. Also, note that this is AMD Milan, so it's DDR4, not DDR5 -- DDR5 is still forthcoming from both AMD and Intel.
      - pclmulqdq 4y ago
        DDR4 link training is a bit easier to comprehend than DDR5, but still a headache. I think almost every high-performance SoC uses an auxiliary processing core for link training at this point (including large FPGAs). I have been waiting for someone to find a security vulnerability in one of these cores.
      - mlindner 4y ago
        One part I couldn't follow on your talk is which software is Oxide designed and which is vendor supplied software? Have you really stripped out every piece of vendor software from the product? Or are there still some parts that are vendor supplied black boxes? The question after your talk implied that DIMM training was still being done by non-Oxide software.
        bcantrill 4y ago
        On AMD, DIMM training is done by the PSP. And you have to run the PSP to run the SoC, so we have no alternative there. More generally, we have implemented and opened everything we can; there still remain opaque bits like the PSP, as well as some smaller bits scattered through the machine (e.g., SSD firmware, MCU boot ROMs, VR firmware, etc.). We have endeavored to make as open a system as possible within the constraint of not making our own silicon. And while we haven't talked about it publicly, we will also open our schematics when we ship the rack, allowing everyone to see every component in the BOM. While we do have some (necessary) proprietary blobs in the system, we want to at least be transparant about where they are!
aidenn0 4y ago
Anyone else worried by the title that Bryan is going to setup an autocratic triumvirate?
- encryptluks2 4y ago
  I'm not exactly worried but it wouldn't surprise me if he wanted to.
- rrss 4y ago
  Here was the BIOS, when comes such another?
  - aidenn0 4y ago
    My heart is in the coffin there with the BIOS, And I must pause until it returns to me.
- lifeisstillgood 4y ago
  Friends, Romans, Countrymen, lend me your binaries?
- xdmr 4y ago
  No, I am Ole Torvalds the poet!
- mlindner 4y ago
  I appear to missing some inside joke. Anyone want to explain it?
  - skybrian 4y ago
    The title alludes to the second line of a speech from one of Shakespeare's plays. https://www.poetryfoundation.org/poems/56968/speech-friends-romans-countrymen-lend-me-your-ears https://www.poetryfoundation.org/poems/56968/speech-friends-...
  - wmf 4y ago
    https://en.wikipedia.org/wiki/Friends,_Romans,_countrymen,_lend_me_your_ears https://en.wikipedia.org/wiki/Friends,_Romans,_countrymen,_l...
lifeisstillgood 4y ago
Link to the other talk he references in this talk https://youtu.be/36myc8wQhLo https://youtu.be/36myc8wQhLo And yes, this is (as ever) a call to arms on fixing a massive security problem we all have.
- pclmulqdq 4y ago
  This is an amazing talk, and I think I have seen it on HN before. It lays out the problem of operating systems on modern computers well. I'm not sure that it offers the right solution by continuing to offer a system with the same set of abstraction layers as a solution to the problem of the "operating system" abstraction. I think the folks at Oxide are probably closer to the right track by silo-busting the BIOS, OS, and hypervisor layers of a modern VM-hosting stack. Edit: I should also add that this talk lays out a huge, gaping hole in the field of OS research, which might be its most important contribution.
  - imglorp 4y ago
    But it's not an OS problem; the OS being written that way is only the symptom because it can't touch the real hardware. Cantrill and Roscoe both pointed at the SOC vendors: if there's a new problem to solve, they add more proprietary, undocumented, cores and enclaves with tightly held secret functions and OSes of their own, all of which are out of visibility of the OS. Some of them are even designed at odds with the user's interests, such as DRM goop. This gets back to the war on general purpose computing as Cory Doctrow put it. The hardware is ceasing to work for the user's interest and is starting to work for everyone else in the stack against the user.
    - pclmulqdq 4y ago
      I have a background in digital circuits and computer architecture, and I completely disagree with you that these SoC components should be run by the traditional CPU cores. Most of these cores have strict latency guarantees or security boundaries that are much harder to achieve when you are trying to share silicon with the general-purpose code running on the machine, and since the cores are so small, it does not meaningfully save silicon to use them. OSes are generally terrible at providing strict latency guarantees. If you will crash the CPU if you don't do [X] every 10 microseconds, you should not be telling the operating system to do [X]. This is the case of audio systems, radio systems, PCIe, and almost every other complicated I/O function. These could all be done with hardware state machines, but it is better (cheaper) to use a small dedicated CPU instead. When I refer to security boundaries, a lot of people think that "security through obscurity" is the idea. It is not. It is more similar to closing all the ports on a server and applying "API security" ideas to hardware. It is a lot easier to secure a function that has a simple queue interface to the main cores than to secure a function that shares the main cores - Spectre and Meltdown have showed us that it might be impossible. Yes, these secure enclaves are used for DRM crap and other nonsense, so I can see why you might not like the existence of that core, but even if you erase the DRM software from that core and make it work for the user completely, you still will want the boundary. Not to mention that every modern motherboard these days has a board management controller, which cannot be part of the CPU, and controls power and resets. From a hardware perspective, these SoCs and motherboards really need to be heterogeneous systems. It's up to the system software to work with that. Heterogeneous SoCs really have nothing to do with taking power away from the user. The user can program all of these cores, but we live with abstractions that make it very hard.
captainmuon 4y ago
Having worked with devicetrees and hated it, I don't like this idea. I don't like the modern world of ARM and embedded where you have hardcoded firmware images for every single device. I like the x86 model, where you have a one size fits all boot disk which just loads different drivers. You don't port your OS to a new computer, you just write drivers for the new hardware parts. And your hardware describes itself to the OS. Granted, UEFI is grotesquely complicated. Probably you could replace most of it with a EEPROM that has a mapping of device ID to memory address, and some instructions how to speak to the embedded controller (basically, ACPI). Unfortunately, UEFI/ACPI or simliar seems to be going nowhere in the ARM world, so it doesn't seem installing Windows/Linux is going to be as easy on ARM as on x86 anytime soon.
- mlindner 4y ago
  You realize that you're just moving the ball around on who provides what, right? Someone has to write that software that "describes itself to the OS" and rather than having the hardware describe itself to the OS, why can't the OS itself determine what hardware it has? That "describes itself to the OS" part is actually often being done by a complete OS as well. > Unfortunately, UEFI/ACPI or simliar seems to be going nowhere in the ARM world, so it doesn't seem installing Windows/Linux is going to be as easy on ARM as on x86 anytime soon. That's a really good thing. Bryan rants about ARM trying to go that way and thinks it's a terrible idea. I'm glad it's failing, assuming that's actually the case.
  - gjsman-1000 4y ago
    > Bryan rants about ARM trying to go that way and thinks it's a terrible idea. I'm glad it's failing, assuming that's actually the case. I'd be curious why because the PC model seems to be way more user-friendly, competition-friendly, and long-term-reliable than the current ARM model. I wish I could just download an ISO of Android 13 and install it on almost any Android phone like I could Windows.
    - mlindner 4y ago
      Well it's in the talk, he spends considerable time on the subject in fact. I suggest taking a watch.
waynecochran 4y ago
But the need the BIOS offers lives after it, and proprietary systems are interred within its bones.
zokier 4y ago
Its kinda difficult to understand what are the key differences between their approach and what coreboot does with Linux payload, which afaik does not use any BIOS/UEFI layer in between and also is pretty minimal on coreboot size => boot as early as possible to Linux
- mlindner 4y ago
  Someone from Oxide (maybe Bryan) talked a bunch about how the philosophy of Oxide differs from Coreboot somewhere. I remember either reading about it or hearing it, but I can't for the life of me remember where I heard/read it.
- panick21_ 4y ago
  The difference is that coreboot comes up and runs a payload, something like LinuxBoot. LinuxBoot is then responsible to find the actual OS that you want to boot, and uses kexec to boot that OS. LinuxBoot still has to pass hardware information to the HostOS and then hand over control. In Oxide this doesn't happen, the Pico thingy boots into their OS and that's it. There is no in-between step. The OS then simple loads the additional things it needs. It never hands of control to anther kernel. At least this is how I understand it.
  - zokier 4y ago
    > LinuxBoot is then responsible to find the actual OS that you want to boot, and uses kexec to boot that OS I'm not super familiar with coreboot, but isn't that step kinda optional? I mean could you start booting up userland directly too?
    - panick21_ 4y ago
      I have not heard of anybody doing this so I would assume there is some reason for that.
    - sounds 4y ago
      I see the sibling comment, so here's my two cents - I wonder if there's some confusion because LinuxBoot is being talked about relative to Oxide. LinuxBoot isn't trying to do that. It was originally a project called "LinuxBIOS" at Los Alamos National Labs to get their supercomputer to boot up in a reasonable amount of time. Once the team started down the path of "rip out the legacy BIOS" they discovered that a lot of things become simpler. LinuxBoot was written with the goal of booting up the machine to GRUB, like this: (CPU Reset) > LinuxBIOS > GRUB > Linux See the description of this video (though the video is worth the watch): https://vimeo.com/724454408 https://vimeo.com/724454408 "Why Linux? Because firmware always evolves to become an operating system. Rather than wait for evolution to take its course, LANL decided to save some time and use Linux as the BIOS: hence LinuxBIOS." See also https://lwn.net/Articles/10590/ https://lwn.net/Articles/10590/ ----- And the answer to your question: "could you start booting up userland directly?" I think if the LinuxBoot authors were around they would mention something about how constrained the on-board flash chip is (SPI Flash). There's room for LinuxBoot and maybe GRUB, but there is not room for all the drivers needed to get to userland.
jstgord 4y ago
What is the truth of this ? I would really like to know if my x86 'linux PC', is actually running another 'hypervisor' OS that runs Linux - it seems like a recipe for security vulnerabilities. If so, there are a lot of Qns : - what OS, is it up to date ? - can this OS be communicated with from the network ? - on which chips does it run ? - can it be re-flashed / upgraded / replaced ? - can it interrupt/schedule my normal os ? - how much CPU/power does it use ? I would certainly trust an Oxide supplied (Rust) bare metal open-source low-level OS to host linux vms on my dev machine, than say, a totally opaque binary blob that the US government has forbidden xyz company from talking about.. just to speculate wildly. I also think Oxide has wider market that just the server space - eg. one has to do all sorts of shenanigans to get a core freed up so that you can run timing/latency sensitive apps, without getting interrupted by linux threads doing noisy housekeeping on each core.
- mlindner 4y ago
  > I would really like to know if my x86 'linux PC', is actually running another 'hypervisor' OS that runs Linux - it seems like a recipe for security vulnerabilities. There's not so much a "hypervisor" OS, but instead a congealed set of many different OSes, some realtime OSes, some possibly old Linux variants, some possibly closed source proprietary one-off OSes. I suggest taking a look at the talk Bryan mentioned: https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo
userbinator 4y ago
Those summary paragraphs feel like they were written in a deliberately obfuscated style; not exactly a good first impression. Rather than have one operating system that boots another that boots another, we have returned the operating system to its roots as the software that abstracts the hardware: we execute a single, holistic system from first instruction to running user-level application code. ...also known as single-purpose firmware for an embedded system. I'm in agreement with the other comment here that this is not a good idea. Standardised interfaces like what the BIOS provides lets the OS not be tied to subtle differences in hardware.
- mappu 4y ago
  It's possible that a single-purpose firmware is too few stages, and it's also possible that the current acpi->firmware->uefi->grub->initrd->linux is too many stages. UEFI does have many of the features of a full operating system including shell, multiprocessing, networking and it really should be Linux in this position instead. LinuxBoot is another attempt at reducing layers here.
  - deleted 4y ago
    [deleted]
eternityforest 4y ago
Sounds kind of horrible. PCs have a level of compatibility that is basically unheard of anywhere in tech. Through whatever historical accident, the IBM PC is... pretty amazing. Things are standardized. Stuff just works. We can choose our OS. We don't have a ton of fragmentation. Distros aren't niche community things developed for specific hardware. If you can make a single OS image that runs on any device of any manufacturer, the way that one Linux distro runs on almost any PC with any combination of parts, great. But removing layers of abstraction seems like it could easily lead to incompatibility. I'd much rather have proprietary blobs everywhere than incompatibility.
- panick21_ 4y ago
  There is something to be said for comparability, but as the speaker points out. That great if you want to install #CrazyLinuxDistro on some old machine and have fun. However if you want to run a whole cloud or have significant amount of server hardware, you might now want to have 5 other core with random OSs doing random things, eating errors and so on and so on. Same goes if you want to make secure devices like IPhone, Chromebooks and so on. I would like my standard linux laptop to be like that too. Also we could have both. If AMD/Intel documented these low level systems then the needed patches could flow into the linux kernel. You could likely boot LinuxBoot and still put UEFI on top if you really want to. And if the system was that open you can still define layers of abstraction on top for situation where you want that compatibility. At the moment compatibility is forced on us with a really thicc layer of abstractions that do some useful things, but also do many not so useful things.
  - eternityforest 4y ago
    In theory it should work, but I'm still suspicious. In general, thicc compatibility layers seem to work in practice, and everything else never really seems to reach IBM PC levels of compatibility, fragmentation seems to happen fast if there's no consensus driving everything to follow some One True Standard. We'd probably wind up with 5 UEFI alikes, 3 distros that don't use any of them, and there would probably be subtle compatibility issues. Eventually the consensus would evolve to some new standard, the way systemd has, and then everyone would complain about that being too heavy and how they don't want a monoculture. Or, worse, we could wind up in the same place ARM is, for a decade. There's no technical reason for it, but there also doesn't seem to be a very strong pressure to standardize. Consumers aren't exactly going to unionize and engineers love to fragment, so to me a standard with that level of popularity is something special and should only be messed with if you have a real clear plan to preserve compatibility.
Gajor444 4y ago
[dead]
rationalfaith 4y ago
haskellandchill 4y ago
they talk about holistic systems and then just hacked some shit together like everybody else, very let down. "hubris" is an apt choice of name.
- wmf 4y ago
  What would your criteria be for a holistic yet non-hacked-together boot process? How do you decide what layers/abstractions are needed and what are not? (Also, user name checks out.)
  - haskellandchill 4y ago
    I think holistic implies some principles guiding design decisions, drawing something from research areas like formal methods, programming language theory, operating systems, hardware, etc and unifying them more of what the cited talk It's Time for Operating Systems to Rediscover Hardware covered.
    - chungy 4y ago
      Maybe your generation 2 will be called "Humility" to reflect lessons learned :)
      - bcantrill 4y ago
        We're one step ahead of you: Humility is actually the debugger for Hubris.[0] [0] https://github.com/oxidecomputer/humility https://github.com/oxidecomputer/humility
        chungy 4y ago
        Haha, that's pretty clever :) The intended point was more that the concept can be iterated upon. I think you'd be the last to claim Hubris ought to be the end-all-be-all of holistic operating systems.
        haskellandchill 4y ago
don-code 4y ago
I was somewhat disappointed that there was only one "need" for holistic systems presented: the fact that correctable DIMM errors were getting eaten by underlying firmware. While I don't necessarily disagree with the speaker's premise, I'd have at least liked to see more than one example of why this is important - I can't say that I've walked away with a good idea of why that is. The fact that correctable versus uncorrectable DIMM errors were differentiable tells me that there already _is_ an interface that allows propagating both up to the OS, and that the firmware of the system just wasn't fully implementing it.
- jclulow 4y ago
  Dealing with existing PC compatible servers is just a constant onslaught of tedium. Some of it is as disruptive as the DIMM stuff Bryan talked about. Some of it is just the stuff people have seemingly become desensitised to. Some of the things I've had to deal with in the last few years: * several different servers where you can set the boot order through the Redfish API, except that it only works about 60% of the time and there is no way to tell from the response if it took or not. Just have to keep doing it and rebooting until it hopefully eventually works! * PXE boot support that is incredibly slow, so for any reasonable payload size you need to chainload iPXE every time * servers with oddball Broadcom NICs where chainloading iPXE doesn't work at all * servers that sit at the BIOS screen for twenty minutes unless you pull out all the U.2 NVMe devices, then they boot OK. Maybe a firmware update will fix it! * firmware updates that don't install properly through the BMC * firmware updates that don't _show_ they're installed properly until the BIOS boots completely and can report the new version through apparently some kind of HTTP request on an internal USB NIC to the BMC, even though the BMC has control of the SPI flash and thus is lying to you until that reboot occurs * BMCs that just stop working until you remove power at the wall from the whole system * IPMI serial over LAN redirection that drops about 4% of output characters but not in a predictable way so copying and pasting, say, a serial number has be done several times to be sure you got the whole thing * an interrupt controller in a HP system that doesn't emulate fixed interrupts correctly, and so eventually after some random number of millions of interrupts the lines are just stuck on until you power cycle the system That's just stuff that comes to mind at the moment. There's literally no way to compose a reliable automated production system on this ridiculous tower of packing peanuts. In contrast, in the lab, I can pretty much just ask an Oxide machine to replace the contents of the SPI ROM and the M.2 storage device and power cycle it and it does what it's told. There are comparatively few moving parts and we have the source to almost all of them.
eschaton 4y ago
It’s simple: Everyone should use Open Firmware. It’s an IEEE standard, it works well, and device trees are already widely used. Just adopt the FORTH monitor too and embrace standardization.
Merciernmon 4y ago
Literary tangent: The title seems to be an allusion to Shakespeare. "I come to bury Caesar, not to praise him." Except, the entire purpose of Antony's funeral oration is to praise Caesar. His intent was to defend Caesar's legacy and shift public opinion against Caesar's assassins. Watch here: https://www.youtube.com/watch?v=0bi1PvXCbr8 https://www.youtube.com/watch?v=0bi1PvXCbr8
- vermilingua 4y ago
  Also Damian Lewis' (spectacular) rendition: https://www.youtube.com/watch?v=q89MLuLSJgk https://www.youtube.com/watch?v=q89MLuLSJgk
- bcantrill 4y ago
  Yeah, yeah, we know -- it's actually Frankenstein's monster.
  - pas 4y ago
    Not everyone grew up in the thick of the Anglosphere :)
- secondcoming 4y ago
  Marlon Brando did it better! https://www.youtube.com/watch?v=101sKhH-lMQ https://www.youtube.com/watch?v=101sKhH-lMQ
tenebrisalietum 4y ago
It should be illegal to hide CPU-facing hardware programming information, such as registers and functions, behind an NDA. Even DRAM training stuff. Let the open source community do it all from the ground up. If the hardware is really a smaller CPU on the other side of a bus or HCI, and what's exposed to the CPU is really a remote-RAM-loading-and-communication interface, that's fine. That's not host CPU firmware, that's device CPU firmware and while that sucks for it to be under NDA/undocumented, at least it's mostly in its own little world. The registers/process needed to setup and talk to that remote embedded CPU should be fully documented and available to anyone. ACPI is a travesty for enabling this hiding of specific hardware interfaces-often tied to things you need to have your laptop be useful like battery controllers and fans. SMM is a travesty for giving it a place to live.
musicale 4y ago
BIOS seemed to work well for CP/M, DOS, etc.. Would probably work well elsewhere.
ignaloidas 4y ago
For what it's worth, a holistic system only allows the user to bring their own code only above some layer. For Unix systems of ye olden days, that would be your C source code that you compile. For Chrome OS, that's gonna be websites and extensions. For Oxide, it seems like that is going to be VM images. But it's important to note that these days, there will always be somebody who wants to bring their code in at a lower level. And the immediate reaction of all vendors is "But why would you want to do that?". Why would you want to bring your own Unix flavor to our hardware? Why would you want to run Linux executables on Chrome OS? Why would you want to run a different hypervisor on Oxide racks? Those people will exist. And layers like BIOS and UEFI allow system vendors to say "Eh, sure, we don't think that what you're doing is wise, but whatever, here's a standard way we'll allow you to use". Holistic systems don't have these standards. They are only usable as their creators intended. And not everyone will agree with their creators intentions. But the work required to refit them to fit another purpose, which the hardware is very fit to do, but the software fights against, is usually too high, and you'll be better off using something that is worse, but left you an avenue to customize it for your purpose easily. My main takeaway from this talk, is that we need some standards for phase-based booting, as there is some very cool improvements that could be done with it. But I see no need for holistic systems as described in that talk, I just see a need for better, more transparent systems. And a system doesn't have to be holistic to be transparent, or good.
- serf 4y ago
  >But I see no need for holistic systems as described in that talk, I just see a need for better, more transparent systems. And a system doesn't have to be holistic to be transparent, or good. my takeaway is that it's not the holistic systems themselves that are needed -- it's that when holistic systems aren't enforced that vendors will go mad with power.
  - cryptonector 4y ago
    But it's terrible power for them because now they have to maintain that firmware, and that's very expensive, so they don't do a good job of it at all, and we can pretend it's OK right up until it's not. Legal liability for broken, unmaintained firmware would very quickly bring about the world that BMC wants.
welkjl 4y ago
- tadfisher 4y ago
ur-whale 4y ago
Very unclear what the business case for this actually is. I watched all the talk, and it seems to be centered around the fact that opaque, vendor-specific blobs that run on hidden cores are a security weakness. I agree with that, but to be fair, what segment of the market actually cares enough about this to pay oxide money? There may be a business case in there, but the talk certainly doesn't explain what that is.
- panick21_ 4y ago
  Why would you expect the talk at a firmware conference to explain their business plan? Also he mentions more then security.
- steveklabnik 4y ago
  From my own perspective, the business case is reliability. We can take responsibility for the entire rack. No “actually that bug is in code from our vendor, we’ll file a ticket upstream” and then maybe someday it’ll get fixed and maybe someday eventually roll out. If (let’s be real, when: nothing is perfect…) problems in our system are found, we can fix them. There’s other reasons too. But imho that’s the big one.
chungy 4y ago
My own personal war against UEFI is a losing battle (already can't use newest Intel generations), and I'd love to see this become a new standard. It might make OSes slightly more complicated, but the benefits of everything being open source and well-understood would more than make up for it.
- tsimionescu 4y ago
  Would every user having to create their own boot image for the specific device combination they own (processor, motherboard, RAM, disk, GPU, power source etc) really be worth it? I think if this became the norm, you would see Linux adoption on this brave new hardware drop even lower than it is today.
bscphil 4y ago
A lot of comments here focusing on whether this is a good thing or not, but I'm worried about a more fundamental issue: how is it supposed to happen in the first place? As I understand it, the proposal is that every bit of software running on a device that is currently considered "firmware" ought to be in the OS instead. What does the story of how we get from here to there look like? 1. We convince manufacturers that it would be a great idea to release thorough documentation for all of their hardware products, and that way open source kernel developers can write their own firmware and merge it into open source kernels. Bryan gestures in this direction several times throughout the talk, claiming that this would be good for the hardware manufacturers because it would enable us to by more of their products. But ... no one's really convinced this is going to work, right? The hardware companies don't want to release documentation for the exact same reason they don't want to open source their hardware; it's a barrier (however small) against leaking trade secrets. 2. We focus our efforts on small companies like Oxide building integrated, top-to-bottom systems designed to run without firmware. This is great because it enables BIOS-less system design from the ground up and can be every bit as open source as we want it to be. But this means waiting and hoping that a tiny number of companies manage to reverse engineer the chips (like the AMD processors discussed in the presentation), which probably means running years behind and an extremely limited selection of hardware. This is not a solution that is going to deliver my next laptop, let alone my grandma's next laptop. 3. We convince manufacturers that EFI is a broken, sucky, insecure mess and include and let them help lead a process designed to replace it. This could maybe happen, we did get rid of BIOS (in favor of EFI) after all, but what is the end result likely to look like? It looks like binary blobs running unknown code necessarily tainting nearly every Linux system under the sun. And that's if they don't just decide to integrate with Microsoft and ignore Linux completely. That's the thing about isolation. It has problems, as this talk discusses, but it also means that (putting aside a great number of bugs) an OS like Linux can exist without needing to reverse engineer firmware for the enormous amount of hardware it runs on. It's great to be able to run an OS with few-to-no blobs active, e.g. with open source Wi-Fi and graphics drivers. Closed firmware is in theory a security nightmare, but in practice it's not that often that anyone managed to exploit these vulnerabilities. Can you imagine having to run closed source Wi-Fi firmware in the Linux Ring 0?
DerSaidin 4y ago
https://www.slideshare.net/bcantrill/i-have-come-to-bury-the-bios-not-to-open-it-the-need-for-holistic-systems https://www.slideshare.net/bcantrill/i-have-come-to-bury-the...
Sirened 4y ago
One of the things that bugged me about this presentation (and, to a similar extent, with Oxide as a whole) is the assertion that the only way we can have safe and trustworthy systems is if each and every component is trustworthy. This is, obviously, a true statement—if we manage to wrangle every component of a system and put it under the control of safe and secure software, we have a really secure system. I don't, however, believe that this is the only way (or, in fact, even the most realistic). Rather, I think we will (for better or for worse) go straight in the opposite direction and end up treating the entire system as an autonomous sea of untrustworthy cores. SGX, for all of its flaws, I think got the nearest to this because it bootstrapped trusted compute in an environment where it couldn't even trust DRAM to not change underneath it. Assuming that every other system agent is hostile lets you use random proprietary garbage without needing to fully control every single core and microcontroller on the platform. It is truly a pain in the ass to program this way but it is, fundamentally, something we can do. This, of course, isn't to say I wouldn't love to have a system where we can run our own open implementation on every bit of the platform, but rather that I don't believe we ever will. Oxide has made it pretty far, no doubt, and it's an incredible feat but as Bryan mentioned, their staff is literally reverse engineering hardware and finding completely undocumented cores in the things they're putting in charge of their platform. Hell, most companies I've been at hardly even know the full capabilities of their shipped silicon (did we slap a chicken bit on that? did we fuse off that performance analysis module for production runs? did we fully remove that one feature that didn't verify before our tapeout deadline? did we backport that one bug fix to all relevant generations? and on and on) and so its many times not even a case of companies being over protective but rather people not being able to even reason about these complex systems. Both Bryan and Roscoe raise the question of who is at the helm of the ship and each find a different monster steering. The truth is that nobody is actually in control on these SoCs because nobody has the last word or some distinct power that cannot be compromised with some other random power. SoCs are not hierarchical; they're ridiculously complex systems of federated power, and we really need to treat them as such.
- pas 4y ago
  You still need at least one trusted core/chip, you need trusted comms to it (eg. it needs an embedded, or securely programmable secret/privkey, and it needs to be able to run some kind of crypto, key exchange), and then you are still left with wondering what the other cores/components are doing. Who is leaking/sniffing/corrupting data, when, and how... So what we need is compartmentalization and a blessed central core ... and that's hierarchy anyway. Yes, of course 99.99..% it's a forgotten fuse or whatever legacy junk. But apparently some folks believe there's is a business model to be built on that last 0.00.. :)
olive247 4y ago
This talk just made me realize how little I understand about what the firmware is really doing. I had previously understood the BIOS to be a small piece of firmware that runs the bootloader at startup and maybe some other coordination between other hardware devices and the CPU. Then CPU takes over and the firmware takes a backseat. The speaker mentioned a whole can of firmware worms that I had no idea even existed. I’m interested to see the progress of open source firmware now.
- soneil 4y ago
  That's pretty much where I'm at - my understanding of most of this hasn't moved since CP/M. CPU boots, PC sets to zero, starts reading instructions from ROM, optionally paging out the ROM if we really need to recover that memory window. I get that x86 is a bit messier because you have optional bioses that get run, and then you read and run the first page of the storage device, but .. man have they made life difficult since I last paid attention.
- mlindner 4y ago
  You should check out this other talk https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo . It goes into a bunch of details about how many parts there are inside modern SoCs.
germandiago 4y ago
what are the implications of replacing the BIOS actually, from a safety point of view?
fefe23 4y ago
I expected to like this talk much more than I actually did. Basically what they did is do a reverse engineering and replay attack on a boot sequence that is already so convoluted that the company that built it didn't think that would be possible. Good for them but all this work will be wasted if the next hardware version comes around and needs a different boot sequence. They basically nailed themselves to a specific version of a specific hardware platform that AMD is replacing as we speak (the next iterations are called Genoa and Bergamo and are expected this and next year respectively. I was hoping this would feel like a breath of freedom, but it really feels like a few rebellious nerds trying to prove something, but not actually proving it. The talk feels to me like they actually proved the opposite. We are already too far down the path. You will never be able to, say, boot Windows on this hardware. Can you boot a stock Linux on this hardware, or will you be dependent on patches from them? As a Linux user I love the idea. As a PC user I never asked for UEFI or the Management Engine. I should be all for this. But somehow I'm not.
- panick21_ 4y ago
  You expected a startup to solve all problems all ADM/Intel have created? That seems like expectation going out of hand. As he says in the video, what is important is documentation. Their code will serve as open documentation for a lot of those details. Sure with the next version there will be some changes, but likely much of it will still be the same. And when they upgrade their product they will have to integrate those changes. For 'stock Linux' to boot like that it would need to add those same low level drivers and I don't know if there is a reason the phased approach shouldn't work on linux. The hope is for many companies to do this and demand this documentation so that with each new version these things will quickly find their way into the open source software. This is now happening with coreboot where multiple companies collaborate on new CPU generations to get it in before the hardware is even released. But with fireware there is never this amazing solution anybody can do unless the manufacture simply does it. Its sad but its the reality.
- raxxorraxor 4y ago
  Easy and sensible why you might be against this. The simple reality is that BIOS did not prevent you to run anything you wanted on your machine. Recent developments are usually to restrict the user, so any change would very likely be bad. Extrapolated experience does not need to be true, but it very well can be. UEFI secure boot is an example. Yes, it can increase security (although I think the threat is specific or outdated), but it can well be used to limit the user practically. And if such a mechanism is established, there will be a class system of trusted and untrusted devices. This is not a development that is hard to predict. So UEFI already failed to a large degree, at least regarding the openness of systems. It is no accident that some companies push these developments enthusiastically. It is not for user security, it is simply for market dominance.
deleted 4y ago
[deleted]

287 points by pclmulqdq 4y ago

Really good talk by Bryan Cantrill (co-founder of Oxide Computer COmpany) on the problems with closed source firmware and the need to move away from BIOS and EFI and how they booted their own x86 AMD custom board system all made with open source firmware and without using AMD's firmware.

convolvatron 4y ago
I've worked on this before. We had explicit help from AMD - their documentation is better than Intels for this stuff, but still. if it weren't for linux's dependence on the bios pci and memory probes, you could really cut out all that crap. you need to program the memory controllers and bring in the kernel from storage.
- Teknoman117 4y ago
  Is Linux actually dependent on the firmware's PCIe probe? You can have the kernel do probe and enumeration if you want. (Outside of the process of getting into the kernel. Most x86 users are going to be booting off of storage that's behind PCIe somehow.) Edit - for clarity, I mean calling back into firmware after boot for PCIe functionality.
  - convolvatron 4y ago
    oh no, my recollection was that at least at the time, linux used the apci pci traversal to start driver discovery instead of doing its own. obviously after that it assumes complete control of the device tree.
pclmulqdq 4y ago
I have booted a few SoCs without a BIOS or anything of the sort before (nothing nearly as big as an AMD Milan chip). Doing this with a huge, fast chip is really impressive from the folks at Oxide. DDR5 DRAM training is also a ridiculously complicated and touchy exercise, as is some of the PCIe link training that (according to the talk) Linux/Unix handles. Edit: this board uses DDR4, not DDR5.
- mlindner 4y ago
  Apparently the DRAM training is apparently one piece of AMD firmware I believe they re-used. Though I had trouble following that part. I think they said that, but they only didn't do it because they didn't need to because they had to pick and choose battles. Edit: Yes, I believe he says they used the AMD firmware for the PSP (Platform Security Processor). Edit2: This post may actually be incorrect. Please go watch the talk. I'm not sure anymore.
  - bcantrill 4y ago
    No, you're correct: the PSP does DIMM training. Also, note that this is AMD Milan, so it's DDR4, not DDR5 -- DDR5 is still forthcoming from both AMD and Intel.
    - pclmulqdq 4y ago
      DDR4 link training is a bit easier to comprehend than DDR5, but still a headache. I think almost every high-performance SoC uses an auxiliary processing core for link training at this point (including large FPGAs). I have been waiting for someone to find a security vulnerability in one of these cores.
    - mlindner 4y ago
      One part I couldn't follow on your talk is which software is Oxide designed and which is vendor supplied software? Have you really stripped out every piece of vendor software from the product? Or are there still some parts that are vendor supplied black boxes? The question after your talk implied that DIMM training was still being done by non-Oxide software.
      - bcantrill 4y ago
        On AMD, DIMM training is done by the PSP. And you have to run the PSP to run the SoC, so we have no alternative there. More generally, we have implemented and opened everything we can; there still remain opaque bits like the PSP, as well as some smaller bits scattered through the machine (e.g., SSD firmware, MCU boot ROMs, VR firmware, etc.). We have endeavored to make as open a system as possible within the constraint of not making our own silicon. And while we haven't talked about it publicly, we will also open our schematics when we ship the rack, allowing everyone to see every component in the BOM. While we do have some (necessary) proprietary blobs in the system, we want to at least be transparant about where they are!

aidenn0 4y ago

Anyone else worried by the title that Bryan is going to setup an autocratic triumvirate?

encryptluks2 4y ago
I'm not exactly worried but it wouldn't surprise me if he wanted to.
rrss 4y ago
Here was the BIOS, when comes such another?
- aidenn0 4y ago
  My heart is in the coffin there with the BIOS, And I must pause until it returns to me.
lifeisstillgood 4y ago
Friends, Romans, Countrymen, lend me your binaries?
xdmr 4y ago
No, I am Ole Torvalds the poet!
mlindner 4y ago
I appear to missing some inside joke. Anyone want to explain it?
- skybrian 4y ago
  The title alludes to the second line of a speech from one of Shakespeare's plays. https://www.poetryfoundation.org/poems/56968/speech-friends-romans-countrymen-lend-me-your-ears https://www.poetryfoundation.org/poems/56968/speech-friends-...
- wmf 4y ago
  https://en.wikipedia.org/wiki/Friends,_Romans,_countrymen,_lend_me_your_ears https://en.wikipedia.org/wiki/Friends,_Romans,_countrymen,_l...

lifeisstillgood 4y ago

Link to the other talk he references in this talk https://youtu.be/36myc8wQhLo https://youtu.be/36myc8wQhLo And yes, this is (as ever) a call to arms on fixing a massive security problem we all have.

pclmulqdq 4y ago
This is an amazing talk, and I think I have seen it on HN before. It lays out the problem of operating systems on modern computers well. I'm not sure that it offers the right solution by continuing to offer a system with the same set of abstraction layers as a solution to the problem of the "operating system" abstraction. I think the folks at Oxide are probably closer to the right track by silo-busting the BIOS, OS, and hypervisor layers of a modern VM-hosting stack. Edit: I should also add that this talk lays out a huge, gaping hole in the field of OS research, which might be its most important contribution.
- imglorp 4y ago
  But it's not an OS problem; the OS being written that way is only the symptom because it can't touch the real hardware. Cantrill and Roscoe both pointed at the SOC vendors: if there's a new problem to solve, they add more proprietary, undocumented, cores and enclaves with tightly held secret functions and OSes of their own, all of which are out of visibility of the OS. Some of them are even designed at odds with the user's interests, such as DRM goop. This gets back to the war on general purpose computing as Cory Doctrow put it. The hardware is ceasing to work for the user's interest and is starting to work for everyone else in the stack against the user.
  - pclmulqdq 4y ago
    I have a background in digital circuits and computer architecture, and I completely disagree with you that these SoC components should be run by the traditional CPU cores. Most of these cores have strict latency guarantees or security boundaries that are much harder to achieve when you are trying to share silicon with the general-purpose code running on the machine, and since the cores are so small, it does not meaningfully save silicon to use them. OSes are generally terrible at providing strict latency guarantees. If you will crash the CPU if you don't do [X] every 10 microseconds, you should not be telling the operating system to do [X]. This is the case of audio systems, radio systems, PCIe, and almost every other complicated I/O function. These could all be done with hardware state machines, but it is better (cheaper) to use a small dedicated CPU instead. When I refer to security boundaries, a lot of people think that "security through obscurity" is the idea. It is not. It is more similar to closing all the ports on a server and applying "API security" ideas to hardware. It is a lot easier to secure a function that has a simple queue interface to the main cores than to secure a function that shares the main cores - Spectre and Meltdown have showed us that it might be impossible. Yes, these secure enclaves are used for DRM crap and other nonsense, so I can see why you might not like the existence of that core, but even if you erase the DRM software from that core and make it work for the user completely, you still will want the boundary. Not to mention that every modern motherboard these days has a board management controller, which cannot be part of the CPU, and controls power and resets. From a hardware perspective, these SoCs and motherboards really need to be heterogeneous systems. It's up to the system software to work with that. Heterogeneous SoCs really have nothing to do with taking power away from the user. The user can program all of these cores, but we live with abstractions that make it very hard.

captainmuon 4y ago

Having worked with devicetrees and hated it, I don't like this idea. I don't like the modern world of ARM and embedded where you have hardcoded firmware images for every single device. I like the x86 model, where you have a one size fits all boot disk which just loads different drivers. You don't port your OS to a new computer, you just write drivers for the new hardware parts. And your hardware describes itself to the OS. Granted, UEFI is grotesquely complicated. Probably you could replace most of it with a EEPROM that has a mapping of device ID to memory address, and some instructions how to speak to the embedded controller (basically, ACPI). Unfortunately, UEFI/ACPI or simliar seems to be going nowhere in the ARM world, so it doesn't seem installing Windows/Linux is going to be as easy on ARM as on x86 anytime soon.

mlindner 4y ago
You realize that you're just moving the ball around on who provides what, right? Someone has to write that software that "describes itself to the OS" and rather than having the hardware describe itself to the OS, why can't the OS itself determine what hardware it has? That "describes itself to the OS" part is actually often being done by a complete OS as well. > Unfortunately, UEFI/ACPI or simliar seems to be going nowhere in the ARM world, so it doesn't seem installing Windows/Linux is going to be as easy on ARM as on x86 anytime soon. That's a really good thing. Bryan rants about ARM trying to go that way and thinks it's a terrible idea. I'm glad it's failing, assuming that's actually the case.
- gjsman-1000 4y ago
  > Bryan rants about ARM trying to go that way and thinks it's a terrible idea. I'm glad it's failing, assuming that's actually the case. I'd be curious why because the PC model seems to be way more user-friendly, competition-friendly, and long-term-reliable than the current ARM model. I wish I could just download an ISO of Android 13 and install it on almost any Android phone like I could Windows.
  - mlindner 4y ago
    Well it's in the talk, he spends considerable time on the subject in fact. I suggest taking a watch.

waynecochran 4y ago

But the need the BIOS offers lives after it, and proprietary systems are interred within its bones.

zokier 4y ago

Its kinda difficult to understand what are the key differences between their approach and what coreboot does with Linux payload, which afaik does not use any BIOS/UEFI layer in between and also is pretty minimal on coreboot size => boot as early as possible to Linux

mlindner 4y ago
Someone from Oxide (maybe Bryan) talked a bunch about how the philosophy of Oxide differs from Coreboot somewhere. I remember either reading about it or hearing it, but I can't for the life of me remember where I heard/read it.
panick21_ 4y ago
The difference is that coreboot comes up and runs a payload, something like LinuxBoot. LinuxBoot is then responsible to find the actual OS that you want to boot, and uses kexec to boot that OS. LinuxBoot still has to pass hardware information to the HostOS and then hand over control. In Oxide this doesn't happen, the Pico thingy boots into their OS and that's it. There is no in-between step. The OS then simple loads the additional things it needs. It never hands of control to anther kernel. At least this is how I understand it.
- zokier 4y ago
  > LinuxBoot is then responsible to find the actual OS that you want to boot, and uses kexec to boot that OS I'm not super familiar with coreboot, but isn't that step kinda optional? I mean could you start booting up userland directly too?
  - panick21_ 4y ago
    I have not heard of anybody doing this so I would assume there is some reason for that.
  - sounds 4y ago
    I see the sibling comment, so here's my two cents - I wonder if there's some confusion because LinuxBoot is being talked about relative to Oxide. LinuxBoot isn't trying to do that. It was originally a project called "LinuxBIOS" at Los Alamos National Labs to get their supercomputer to boot up in a reasonable amount of time. Once the team started down the path of "rip out the legacy BIOS" they discovered that a lot of things become simpler. LinuxBoot was written with the goal of booting up the machine to GRUB, like this: (CPU Reset) > LinuxBIOS > GRUB > Linux See the description of this video (though the video is worth the watch): https://vimeo.com/724454408 https://vimeo.com/724454408 "Why Linux? Because firmware always evolves to become an operating system. Rather than wait for evolution to take its course, LANL decided to save some time and use Linux as the BIOS: hence LinuxBIOS." See also https://lwn.net/Articles/10590/ https://lwn.net/Articles/10590/ ----- And the answer to your question: "could you start booting up userland directly?" I think if the LinuxBoot authors were around they would mention something about how constrained the on-board flash chip is (SPI Flash). There's room for LinuxBoot and maybe GRUB, but there is not room for all the drivers needed to get to userland.

jstgord 4y ago

What is the truth of this ? I would really like to know if my x86 'linux PC', is actually running another 'hypervisor' OS that runs Linux - it seems like a recipe for security vulnerabilities. If so, there are a lot of Qns : - what OS, is it up to date ? - can this OS be communicated with from the network ? - on which chips does it run ? - can it be re-flashed / upgraded / replaced ? - can it interrupt/schedule my normal os ? - how much CPU/power does it use ? I would certainly trust an Oxide supplied (Rust) bare metal open-source low-level OS to host linux vms on my dev machine, than say, a totally opaque binary blob that the US government has forbidden xyz company from talking about.. just to speculate wildly. I also think Oxide has wider market that just the server space - eg. one has to do all sorts of shenanigans to get a core freed up so that you can run timing/latency sensitive apps, without getting interrupted by linux threads doing noisy housekeeping on each core.

mlindner 4y ago
> I would really like to know if my x86 'linux PC', is actually running another 'hypervisor' OS that runs Linux - it seems like a recipe for security vulnerabilities. There's not so much a "hypervisor" OS, but instead a congealed set of many different OSes, some realtime OSes, some possibly old Linux variants, some possibly closed source proprietary one-off OSes. I suggest taking a look at the talk Bryan mentioned: https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo

userbinator 4y ago

Those summary paragraphs feel like they were written in a deliberately obfuscated style; not exactly a good first impression. Rather than have one operating system that boots another that boots another, we have returned the operating system to its roots as the software that abstracts the hardware: we execute a single, holistic system from first instruction to running user-level application code. ...also known as single-purpose firmware for an embedded system. I'm in agreement with the other comment here that this is not a good idea. Standardised interfaces like what the BIOS provides lets the OS not be tied to subtle differences in hardware.

mappu 4y ago
It's possible that a single-purpose firmware is too few stages, and it's also possible that the current acpi->firmware->uefi->grub->initrd->linux is too many stages. UEFI does have many of the features of a full operating system including shell, multiprocessing, networking and it really should be Linux in this position instead. LinuxBoot is another attempt at reducing layers here.
- deleted 4y ago
  [deleted]

eternityforest 4y ago

Sounds kind of horrible. PCs have a level of compatibility that is basically unheard of anywhere in tech. Through whatever historical accident, the IBM PC is... pretty amazing. Things are standardized. Stuff just works. We can choose our OS. We don't have a ton of fragmentation. Distros aren't niche community things developed for specific hardware. If you can make a single OS image that runs on any device of any manufacturer, the way that one Linux distro runs on almost any PC with any combination of parts, great. But removing layers of abstraction seems like it could easily lead to incompatibility. I'd much rather have proprietary blobs everywhere than incompatibility.

panick21_ 4y ago
There is something to be said for comparability, but as the speaker points out. That great if you want to install #CrazyLinuxDistro on some old machine and have fun. However if you want to run a whole cloud or have significant amount of server hardware, you might now want to have 5 other core with random OSs doing random things, eating errors and so on and so on. Same goes if you want to make secure devices like IPhone, Chromebooks and so on. I would like my standard linux laptop to be like that too. Also we could have both. If AMD/Intel documented these low level systems then the needed patches could flow into the linux kernel. You could likely boot LinuxBoot and still put UEFI on top if you really want to. And if the system was that open you can still define layers of abstraction on top for situation where you want that compatibility. At the moment compatibility is forced on us with a really thicc layer of abstractions that do some useful things, but also do many not so useful things.
- eternityforest 4y ago
  In theory it should work, but I'm still suspicious. In general, thicc compatibility layers seem to work in practice, and everything else never really seems to reach IBM PC levels of compatibility, fragmentation seems to happen fast if there's no consensus driving everything to follow some One True Standard. We'd probably wind up with 5 UEFI alikes, 3 distros that don't use any of them, and there would probably be subtle compatibility issues. Eventually the consensus would evolve to some new standard, the way systemd has, and then everyone would complain about that being too heavy and how they don't want a monoculture. Or, worse, we could wind up in the same place ARM is, for a decade. There's no technical reason for it, but there also doesn't seem to be a very strong pressure to standardize. Consumers aren't exactly going to unionize and engineers love to fragment, so to me a standard with that level of popularity is something special and should only be messed with if you have a real clear plan to preserve compatibility.

Gajor444 4y ago

[dead]

rationalfaith 4y ago

haskellandchill 4y ago

they talk about holistic systems and then just hacked some shit together like everybody else, very let down. "hubris" is an apt choice of name.

wmf 4y ago
What would your criteria be for a holistic yet non-hacked-together boot process? How do you decide what layers/abstractions are needed and what are not? (Also, user name checks out.)
- haskellandchill 4y ago
  I think holistic implies some principles guiding design decisions, drawing something from research areas like formal methods, programming language theory, operating systems, hardware, etc and unifying them more of what the cited talk It's Time for Operating Systems to Rediscover Hardware covered.
  - chungy 4y ago
    Maybe your generation 2 will be called "Humility" to reflect lessons learned :)
    - bcantrill 4y ago
      We're one step ahead of you: Humility is actually the debugger for Hubris.[0] [0] https://github.com/oxidecomputer/humility https://github.com/oxidecomputer/humility
      - chungy 4y ago
        Haha, that's pretty clever :) The intended point was more that the concept can be iterated upon. I think you'd be the last to claim Hubris ought to be the end-all-be-all of holistic operating systems.
      - haskellandchill 4y ago

don-code 4y ago

I was somewhat disappointed that there was only one "need" for holistic systems presented: the fact that correctable DIMM errors were getting eaten by underlying firmware. While I don't necessarily disagree with the speaker's premise, I'd have at least liked to see more than one example of why this is important - I can't say that I've walked away with a good idea of why that is. The fact that correctable versus uncorrectable DIMM errors were differentiable tells me that there already _is_ an interface that allows propagating both up to the OS, and that the firmware of the system just wasn't fully implementing it.

jclulow 4y ago
Dealing with existing PC compatible servers is just a constant onslaught of tedium. Some of it is as disruptive as the DIMM stuff Bryan talked about. Some of it is just the stuff people have seemingly become desensitised to. Some of the things I've had to deal with in the last few years: * several different servers where you can set the boot order through the Redfish API, except that it only works about 60% of the time and there is no way to tell from the response if it took or not. Just have to keep doing it and rebooting until it hopefully eventually works! * PXE boot support that is incredibly slow, so for any reasonable payload size you need to chainload iPXE every time * servers with oddball Broadcom NICs where chainloading iPXE doesn't work at all * servers that sit at the BIOS screen for twenty minutes unless you pull out all the U.2 NVMe devices, then they boot OK. Maybe a firmware update will fix it! * firmware updates that don't install properly through the BMC * firmware updates that don't _show_ they're installed properly until the BIOS boots completely and can report the new version through apparently some kind of HTTP request on an internal USB NIC to the BMC, even though the BMC has control of the SPI flash and thus is lying to you until that reboot occurs * BMCs that just stop working until you remove power at the wall from the whole system * IPMI serial over LAN redirection that drops about 4% of output characters but not in a predictable way so copying and pasting, say, a serial number has be done several times to be sure you got the whole thing * an interrupt controller in a HP system that doesn't emulate fixed interrupts correctly, and so eventually after some random number of millions of interrupts the lines are just stuck on until you power cycle the system That's just stuff that comes to mind at the moment. There's literally no way to compose a reliable automated production system on this ridiculous tower of packing peanuts. In contrast, in the lab, I can pretty much just ask an Oxide machine to replace the contents of the SPI ROM and the M.2 storage device and power cycle it and it does what it's told. There are comparatively few moving parts and we have the source to almost all of them.

eschaton 4y ago

It’s simple: Everyone should use Open Firmware. It’s an IEEE standard, it works well, and device trees are already widely used. Just adopt the FORTH monitor too and embrace standardization.

Merciernmon 4y ago

Literary tangent: The title seems to be an allusion to Shakespeare. "I come to bury Caesar, not to praise him." Except, the entire purpose of Antony's funeral oration is to praise Caesar. His intent was to defend Caesar's legacy and shift public opinion against Caesar's assassins. Watch here: https://www.youtube.com/watch?v=0bi1PvXCbr8 https://www.youtube.com/watch?v=0bi1PvXCbr8

vermilingua 4y ago
Also Damian Lewis' (spectacular) rendition: https://www.youtube.com/watch?v=q89MLuLSJgk https://www.youtube.com/watch?v=q89MLuLSJgk
bcantrill 4y ago
Yeah, yeah, we know -- it's actually Frankenstein's monster.
- pas 4y ago
  Not everyone grew up in the thick of the Anglosphere :)
secondcoming 4y ago
Marlon Brando did it better! https://www.youtube.com/watch?v=101sKhH-lMQ https://www.youtube.com/watch?v=101sKhH-lMQ

tenebrisalietum 4y ago

It should be illegal to hide CPU-facing hardware programming information, such as registers and functions, behind an NDA. Even DRAM training stuff. Let the open source community do it all from the ground up. If the hardware is really a smaller CPU on the other side of a bus or HCI, and what's exposed to the CPU is really a remote-RAM-loading-and-communication interface, that's fine. That's not host CPU firmware, that's device CPU firmware and while that sucks for it to be under NDA/undocumented, at least it's mostly in its own little world. The registers/process needed to setup and talk to that remote embedded CPU should be fully documented and available to anyone. ACPI is a travesty for enabling this hiding of specific hardware interfaces-often tied to things you need to have your laptop be useful like battery controllers and fans. SMM is a travesty for giving it a place to live.

musicale 4y ago

BIOS seemed to work well for CP/M, DOS, etc.. Would probably work well elsewhere.

ignaloidas 4y ago

For what it's worth, a holistic system only allows the user to bring their own code only above some layer. For Unix systems of ye olden days, that would be your C source code that you compile. For Chrome OS, that's gonna be websites and extensions. For Oxide, it seems like that is going to be VM images. But it's important to note that these days, there will always be somebody who wants to bring their code in at a lower level. And the immediate reaction of all vendors is "But why would you want to do that?". Why would you want to bring your own Unix flavor to our hardware? Why would you want to run Linux executables on Chrome OS? Why would you want to run a different hypervisor on Oxide racks? Those people will exist. And layers like BIOS and UEFI allow system vendors to say "Eh, sure, we don't think that what you're doing is wise, but whatever, here's a standard way we'll allow you to use". Holistic systems don't have these standards. They are only usable as their creators intended. And not everyone will agree with their creators intentions. But the work required to refit them to fit another purpose, which the hardware is very fit to do, but the software fights against, is usually too high, and you'll be better off using something that is worse, but left you an avenue to customize it for your purpose easily. My main takeaway from this talk, is that we need some standards for phase-based booting, as there is some very cool improvements that could be done with it. But I see no need for holistic systems as described in that talk, I just see a need for better, more transparent systems. And a system doesn't have to be holistic to be transparent, or good.

serf 4y ago
>But I see no need for holistic systems as described in that talk, I just see a need for better, more transparent systems. And a system doesn't have to be holistic to be transparent, or good. my takeaway is that it's not the holistic systems themselves that are needed -- it's that when holistic systems aren't enforced that vendors will go mad with power.
- cryptonector 4y ago
  But it's terrible power for them because now they have to maintain that firmware, and that's very expensive, so they don't do a good job of it at all, and we can pretend it's OK right up until it's not. Legal liability for broken, unmaintained firmware would very quickly bring about the world that BMC wants.

welkjl 4y ago

tadfisher 4y ago

ur-whale 4y ago

Very unclear what the business case for this actually is. I watched all the talk, and it seems to be centered around the fact that opaque, vendor-specific blobs that run on hidden cores are a security weakness. I agree with that, but to be fair, what segment of the market actually cares enough about this to pay oxide money? There may be a business case in there, but the talk certainly doesn't explain what that is.

panick21_ 4y ago
Why would you expect the talk at a firmware conference to explain their business plan? Also he mentions more then security.
steveklabnik 4y ago
From my own perspective, the business case is reliability. We can take responsibility for the entire rack. No “actually that bug is in code from our vendor, we’ll file a ticket upstream” and then maybe someday it’ll get fixed and maybe someday eventually roll out. If (let’s be real, when: nothing is perfect…) problems in our system are found, we can fix them. There’s other reasons too. But imho that’s the big one.

chungy 4y ago

My own personal war against UEFI is a losing battle (already can't use newest Intel generations), and I'd love to see this become a new standard. It might make OSes slightly more complicated, but the benefits of everything being open source and well-understood would more than make up for it.

tsimionescu 4y ago
Would every user having to create their own boot image for the specific device combination they own (processor, motherboard, RAM, disk, GPU, power source etc) really be worth it? I think if this became the norm, you would see Linux adoption on this brave new hardware drop even lower than it is today.

bscphil 4y ago

A lot of comments here focusing on whether this is a good thing or not, but I'm worried about a more fundamental issue: how is it supposed to happen in the first place? As I understand it, the proposal is that every bit of software running on a device that is currently considered "firmware" ought to be in the OS instead. What does the story of how we get from here to there look like? 1. We convince manufacturers that it would be a great idea to release thorough documentation for all of their hardware products, and that way open source kernel developers can write their own firmware and merge it into open source kernels. Bryan gestures in this direction several times throughout the talk, claiming that this would be good for the hardware manufacturers because it would enable us to by more of their products. But ... no one's really convinced this is going to work, right? The hardware companies don't want to release documentation for the exact same reason they don't want to open source their hardware; it's a barrier (however small) against leaking trade secrets. 2. We focus our efforts on small companies like Oxide building integrated, top-to-bottom systems designed to run without firmware. This is great because it enables BIOS-less system design from the ground up and can be every bit as open source as we want it to be. But this means waiting and hoping that a tiny number of companies manage to reverse engineer the chips (like the AMD processors discussed in the presentation), which probably means running years behind and an extremely limited selection of hardware. This is not a solution that is going to deliver my next laptop, let alone my grandma's next laptop. 3. We convince manufacturers that EFI is a broken, sucky, insecure mess and include and let them help lead a process designed to replace it. This could maybe happen, we did get rid of BIOS (in favor of EFI) after all, but what is the end result likely to look like? It looks like binary blobs running unknown code necessarily tainting nearly every Linux system under the sun. And that's if they don't just decide to integrate with Microsoft and ignore Linux completely. That's the thing about isolation. It has problems, as this talk discusses, but it also means that (putting aside a great number of bugs) an OS like Linux can exist without needing to reverse engineer firmware for the enormous amount of hardware it runs on. It's great to be able to run an OS with few-to-no blobs active, e.g. with open source Wi-Fi and graphics drivers. Closed firmware is in theory a security nightmare, but in practice it's not that often that anyone managed to exploit these vulnerabilities. Can you imagine having to run closed source Wi-Fi firmware in the Linux Ring 0?

DerSaidin 4y ago

https://www.slideshare.net/bcantrill/i-have-come-to-bury-the-bios-not-to-open-it-the-need-for-holistic-systems https://www.slideshare.net/bcantrill/i-have-come-to-bury-the...

Sirened 4y ago

One of the things that bugged me about this presentation (and, to a similar extent, with Oxide as a whole) is the assertion that the only way we can have safe and trustworthy systems is if each and every component is trustworthy. This is, obviously, a true statement—if we manage to wrangle every component of a system and put it under the control of safe and secure software, we have a really secure system. I don't, however, believe that this is the only way (or, in fact, even the most realistic). Rather, I think we will (for better or for worse) go straight in the opposite direction and end up treating the entire system as an autonomous sea of untrustworthy cores. SGX, for all of its flaws, I think got the nearest to this because it bootstrapped trusted compute in an environment where it couldn't even trust DRAM to not change underneath it. Assuming that every other system agent is hostile lets you use random proprietary garbage without needing to fully control every single core and microcontroller on the platform. It is truly a pain in the ass to program this way but it is, fundamentally, something we can do. This, of course, isn't to say I wouldn't love to have a system where we can run our own open implementation on every bit of the platform, but rather that I don't believe we ever will. Oxide has made it pretty far, no doubt, and it's an incredible feat but as Bryan mentioned, their staff is literally reverse engineering hardware and finding completely undocumented cores in the things they're putting in charge of their platform. Hell, most companies I've been at hardly even know the full capabilities of their shipped silicon (did we slap a chicken bit on that? did we fuse off that performance analysis module for production runs? did we fully remove that one feature that didn't verify before our tapeout deadline? did we backport that one bug fix to all relevant generations? and on and on) and so its many times not even a case of companies being over protective but rather people not being able to even reason about these complex systems. Both Bryan and Roscoe raise the question of who is at the helm of the ship and each find a different monster steering. The truth is that nobody is actually in control on these SoCs because nobody has the last word or some distinct power that cannot be compromised with some other random power. SoCs are not hierarchical; they're ridiculously complex systems of federated power, and we really need to treat them as such.

pas 4y ago
You still need at least one trusted core/chip, you need trusted comms to it (eg. it needs an embedded, or securely programmable secret/privkey, and it needs to be able to run some kind of crypto, key exchange), and then you are still left with wondering what the other cores/components are doing. Who is leaking/sniffing/corrupting data, when, and how... So what we need is compartmentalization and a blessed central core ... and that's hierarchy anyway. Yes, of course 99.99..% it's a forgotten fuse or whatever legacy junk. But apparently some folks believe there's is a business model to be built on that last 0.00.. :)

olive247 4y ago

This talk just made me realize how little I understand about what the firmware is really doing. I had previously understood the BIOS to be a small piece of firmware that runs the bootloader at startup and maybe some other coordination between other hardware devices and the CPU. Then CPU takes over and the firmware takes a backseat. The speaker mentioned a whole can of firmware worms that I had no idea even existed. I’m interested to see the progress of open source firmware now.

soneil 4y ago
That's pretty much where I'm at - my understanding of most of this hasn't moved since CP/M. CPU boots, PC sets to zero, starts reading instructions from ROM, optionally paging out the ROM if we really need to recover that memory window. I get that x86 is a bit messier because you have optional bioses that get run, and then you read and run the first page of the storage device, but .. man have they made life difficult since I last paid attention.
mlindner 4y ago
You should check out this other talk https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo . It goes into a bunch of details about how many parts there are inside modern SoCs.

germandiago 4y ago

what are the implications of replacing the BIOS actually, from a safety point of view?

fefe23 4y ago

I expected to like this talk much more than I actually did. Basically what they did is do a reverse engineering and replay attack on a boot sequence that is already so convoluted that the company that built it didn't think that would be possible. Good for them but all this work will be wasted if the next hardware version comes around and needs a different boot sequence. They basically nailed themselves to a specific version of a specific hardware platform that AMD is replacing as we speak (the next iterations are called Genoa and Bergamo and are expected this and next year respectively. I was hoping this would feel like a breath of freedom, but it really feels like a few rebellious nerds trying to prove something, but not actually proving it. The talk feels to me like they actually proved the opposite. We are already too far down the path. You will never be able to, say, boot Windows on this hardware. Can you boot a stock Linux on this hardware, or will you be dependent on patches from them? As a Linux user I love the idea. As a PC user I never asked for UEFI or the Management Engine. I should be all for this. But somehow I'm not.

panick21_ 4y ago
You expected a startup to solve all problems all ADM/Intel have created? That seems like expectation going out of hand. As he says in the video, what is important is documentation. Their code will serve as open documentation for a lot of those details. Sure with the next version there will be some changes, but likely much of it will still be the same. And when they upgrade their product they will have to integrate those changes. For 'stock Linux' to boot like that it would need to add those same low level drivers and I don't know if there is a reason the phased approach shouldn't work on linux. The hope is for many companies to do this and demand this documentation so that with each new version these things will quickly find their way into the open source software. This is now happening with coreboot where multiple companies collaborate on new CPU generations to get it in before the hardware is even released. But with fireware there is never this amazing solution anybody can do unless the manufacture simply does it. Its sad but its the reality.
raxxorraxor 4y ago
Easy and sensible why you might be against this. The simple reality is that BIOS did not prevent you to run anything you wanted on your machine. Recent developments are usually to restrict the user, so any change would very likely be bad. Extrapolated experience does not need to be true, but it very well can be. UEFI secure boot is an example. Yes, it can increase security (although I think the threat is specific or outdated), but it can well be used to limit the user practically. And if such a mechanism is established, there will be a class system of trusted and untrusted devices. This is not a development that is hard to predict. So UEFI already failed to a large degree, at least regarding the openness of systems. It is no accident that some companies push these developments enthusiastically. It is not for user security, it is simply for market dominance.

deleted 4y ago

[deleted]