11 ms·
Booting Modern Intel CPUs
- rkagerer 4y agoThanks for educating me as to the dumpster fire of patched together steps that take place in the first milliseconds of booting my PC. With regard to this: Intel CPUs ship with built-in microcode, but it's frequently old and buggy and it's up to the system firmware to include a copy that's new enough that it's actually expected to work reliably. ...I wish companies put more effort into releasing finished, polished products. Not just Intel. I played with an ESP32-S3 recently and learned not only was the entire digi controller for ADC2 'dropped' from support due to bad silicon, but using ADC1 with DMA was fraught with gotchas and blatently incorrect info in the documentation (reported at least a dozen to EspressIf and they acknowledged).
- taspeotis 4y agoSure but I have some sympathy here … a contemporary CPU probably has a million different capabilities and if some not-even-single-digit-percentage of them don’t work that’s a lot of errata. Plus it’s hardware! So it’s hard to fix it once it’s out.
- sgtnoodle 4y agoThe ESP32 ADC is a dumpster fire. I used one to sample an analog signal at 8Khz, looking for 400uS wide pulses (pacemaker capture pulses.) I originally started with an interrupt that triggered conversion from software. It would completely miss pulses 10-20% of the time and instead drift around, as if the ADC was floating rather than connected to anything. I finally got it to work reliably by switching to DMA through the I2S peripheral. Pretty much every hobby project I look at, people complain about how bad the ADC is, and then just proceed to take N samples and average them. It's not that the ADC is bad, it's that the underlying firmware is buggy. I suspect there's a race condition with the main CPU and a coprocessor that's using the ADC for some internal wifi stuff.
- rkagerer 4y agoYeah I felt that pain :-). On the ESP32-S3 ADC2 is shared with Wifi, but I don't think that's the case for ADC1. I made headway by throwing away all their boilerplate code and setting up the raw peripheral registers myself. I actually prefer it that way without all the abstraction underneath. Was getting nice results at 80kHz sample rate. Would love to turn it into a write-up as I'm not sure anyone has done this before with that chip (at least I couldn't find any sample code around). It's actually all in MicroPython at the moment to boot... but wouldn't be hard to convert to C (and might even clean things up due to the more natural fit of that language working with bitfields). What I'd really like is to make it into a DMA-based ADC module for MicroPython but I'm not familiar enough with compiling that platform from scratch, particularly on Windows.
- matthewfcarlson 4y agoIf you do end up writing it up, consider dropping it in the hackaday tip line (https://hackaday.com/submit-a-tip/ https://hackaday.com/submit-a-tip/). It sounds incredibly useful.
- gkhartman 4y agoThanks for the tip about the ESP32-S3 ADC2. Coincidentally, I was about to embark on that trail only to learn the hard way. You've likely saved me a good deal of frustration.
- ChuckNorris89 4y agoThis is why companies still choose to buy expensive microcontrollers from big name reputable manufacturers (Microchip, TI, SiLabs, NXP, Nordic, STM, Infineon, Renesas, etc.) instead of going for the cheaper Chinese ARM microcontrollers at half the price. They tend to usually test their designs a lot more thoroughly, which adds to the final cost, and even when their products do come with flaws, they're more likely to be transparent about it and assist you with workarounds and sometimes send over field engineers on-site to help you. They also tend to be more honest about the specs, capabilities and limitations of their products in the datasheets. Finding out half-way through the design phase that some claims in the datasheet are bogus is a no-go for most companies. If you're just doing hobby work, or tinkering, or planning to ship millions of bottom of the barrel products on AliExpress with no intention of providing any warranty or customer support for them, then it's fine to go with whatever's the absolute cheapest, but serious companies like Apple, Sony, etc. who care about the customer experience, won't risk delaying a product launch because they wanted to save 50 cents on a new unproven cheap microcontroller who's ADCs don't work right.
- RicoElectrico 4y agoUh, have you ever seen the length of a typical STM32 errata sheet? ;)
- ChuckNorris89 4y agoUh, have you read the part in my comments where I said that they're morel likely to be honest about it? Having a short errata or no errata at all doesn't mean the product is flawless. It could be that they don't know all the faults yet, or aren't sharing them, or both. For ST it could be that they seem to be the most popular cheap ARM microcontrollers for hobbyists and consumer products, so it's easier to find faults in them since so many companies use them, similar how the most used pieces of software also have the most vulnerabilities reported on them. Also, maybe ST was not a great example on my end, as they seem to have an obsession lately with outsourcing and farming out everything to the cheapest offshore location possible and penny pinching to the extreme for their consumer oriented parts. I don't blame them though, competition in the generic ARM microcontroller market is cutthroats and margins are slim and salaries are low and your major customers (Sony, Apple, Samsung, LG, etc) keep putting the pressure on you to lower your prices or threaten to look somewhere else.
- rbanffy 4y ago> dumpster fire of patched together That's more or less the history of the x86, from the 8086 and on. I'd add a couple extra expletives as well. I can imagine a number of better ways seeding CPU state prior to starting it in a way they could all be neatly started in parallel. > I wish companies put more effort into releasing finished, polished products. Some do. It's just that their hardware is expensive and not very available outside some niches.
- SassyGrapefruit 4y agoI think my favorite part of this discussion is that most developers give this sort of process/code a mystical vibe. Like it's some sacred code built by a priesthood of the world's greatest and smartest programmers. Not accessible to mere mortals. I encourage developers that work for me to just read the code and documentation. My advice to them is usually something along the lines of... Think of how you would have done it as a sophomore in college completing an overdue assignment... chances are it works just like that
- booi 4y agoThis is insanity. I wonder if ARM CPUs were able to start from scratch or a better place.
- grishka 4y agoFor one, there never was, and still isn't, any universal BIOS-like spec for how an ARM CPU should boot and which peripherals it should have.
- penguin_booze 4y agoArm v7 was a Wild West, but with v8, Arm tried to standardize a lot. The Arm Trusted Firmware is the reference boot firmware implementation for v8+ CPUs: https://github.com/ARM-software/arm-trusted-firmware https://github.com/ARM-software/arm-trusted-firmware. I'd think most of the referece documents can be discovered from that code base. Relatedly, from the perspective of hands-on programming, the System Programmer's guide is the manual to start with: https://developer.arm.com/documentation/den0024/a/ https://developer.arm.com/documentation/den0024/a/.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- yencabulator 4y agoTo drive home how far from sane common ARM bootup sequences are, the Raspberry Pi is started by its GPU. You can think of an RPi as a proprietary GPU with an auxiliary ARM CPU.
- jsmith45 4y agoYeah, and the GPU code is designed to read the linux kernel into RAM, and set it up so it begins executing the linux kernel as the inital instructions of the ARM CPU. If you want some more normal bootloader like u-boot you need to jump through hoops to make sure the GPU based bootloader can treat it like a weird Linux Kernel. (In theory, with source for the GPU 2nd stage bootloader one could change things, but RPI foundation does not provide access to that source).
- moose_man 4y agoWhen technical debt strangles your entire business. "We'll fix this next release"
- mnd999 4y agoIt’s not technical debt, it’s a feature. Being able to boot and run software from decades ago has served them well in the past.
- mjg59 4y agoThat's an argument for including support for real mode, rather than for coming up in it. Modern systems booting legacy software are already transitioning into protected mode to run the UEFI stack, and then switching back to real mode before passing control to the Compatibility Services Module.
- LoganDark 4y ago> That's an argument for including support for real mode, rather than for coming up in it. Not really. You have to come up in it to boot software that expects to already be in it.
- mjg59 4y agoNo, firmware needs to hand off in the state the software you're booting needs. That says nothing about the mode the CPU needs to be in when it starts running firmware.
- LoganDark 4y ago> No, firmware needs to hand off in the state the software you're booting needs. That says nothing about the mode the CPU needs to be in when it starts running firmware. This assumes that "firmware" doesn't count against backwards compatibility, which isn't necessarily the case. Maybe Intel (or AMD) doesn't have a 100% monopoly on firmware to be confident enough that the CPU mode is an implementation detail. Or maybe some customers do indeed run their own "firmware" (maybe embedded?). No way to be sure.
- cperciva 4y agoAnother fun thing with SMP: The x86 multiprocessor spec says that to start an AP you need to send an IPI, wait 10 ms, then send another IPI (IIRC it's a "reset" followed by an "init"). On large systems, this adds up! Except that you don't need to wait 10 ms for each AP -- you can start up the APs in parallel. There's just one small problem: All of the APs start up in the same state -- executing from the same CS:IP, and also the same stack pointer. Good luck having hundreds of CPUs stomping over each other's stacks. Except that if you're careful, it doesn't matter -- you can even make a function call if you want because all of the CPUs will push the same return address onto the stack. Implementing this in FreeBSD is on my "speeding up the boot" to-do list. I know it's possible though, because someone told me that they had already done exactly this in a different (non open source) system. Lexicon for the non-x86 people: SMP = Symmetric MultiProcessing, aka more than one "virtual CPU". AP = Auxiliary Processor, any CPU other than the one which the BIOS starts up for you. IPI = InterProcessor Interrupt, how CPUs wake each other up. CS:IP = Code Segment + Instruction Pointer, where the CPU is reading instructions from.
- JoshTriplett 4y agoAlso, on any modern system, you really don't need the second SIPI. The CPU will come up with the first SIPI, and then ignore the second SIPI. So you can just send a pile of INITs and then a pile of SIPIs (or in theory one broadcast SIPI), and expect the CPUs to come up. For the startup code, you shouldn't need to make a function call. A few lines of memory-less stack-less assembly could get the CPU number and then change the stack, assuming you have a global value that gives the base of a preallocated array of stacks.
- Dwedit 4y agoHow "modern" are we talking here? Core 2 Duo? Arrandale? Haswell? Skylake?
- mananaysiempre 4y agoPer the OSDev Wiki, Pentium Pro and later[1]: > For newer CPUs (P6, Pentium 4) one SIPI is enough, but I'm not sure if older Intel CPUs (Pentium) or CPUs from other manufacturers need a second SIPI or not. When (and if) that became officially sanctioned behaviour is another question. [1] https://wiki.osdev.org/Symmetric_Multiprocessing#Initialisation_of_an_old_SMP_system https://wiki.osdev.org/Symmetric_Multiprocessing#Initialisat...
- jeffbee 4y agoHuh, I did not realize until reading this that the IME is also x86. I assumed it was just whatever was most convenient, which seems like it rules out x86, but I guess not.
- mjg59 4y agoIt was initially ARC, but transitioned to x86 with version 11.
- p_l 4y agoIIRC some of the variants used embedded SPARC, though they are a rare find.
- usr1106 4y agoAMD uses ARM for that purpose IIRC.
- anonymfus 4y agoNo, the role of the AMD's Platform Security Processor in the boot process is completely different from Intel's IME, as PSP is located on the CPU side and so entire boot process and security checks are completely different and would require a separate writeup.
- usr1106 4y agoUse the cache as RAM. So I guess you could run a small Linux system just in cache without any RAM? Why? As a fun project with some old motherboard for example :)
- JoshTriplett 4y agoWith a fair bit of effort; notably, DMA and IOMMUs probably won't work, and most modern devices don't really support PIO. You might be able to boot a really simple environment that runs out of an initramfs though. It's also entirely not obvious to what degree you can use the paging system. It'd likely be a substantial effort to port Linux.
- usr1106 4y agoSure, I would not expect to do any advanced IO. But right, probably there is no serial console really close to the CPU either, this is not a Raspberry PI. No idea what signals could be used for that. Page tables I am not sure either. Could you still do it like in Linux 1.0? No idea what things looked like then, but I assume much less dedicated hardware support.
- JoshTriplett 4y ago> probably there is no serial console really close to the CPU either outb to 0x3f8 might work. I know there were once versions of Linux that supported running without an MMU. Those versions, with very limited hardware support, might work in this mode.
- mschuster91 4y agoIt's still part of the kernel code: https://www.kernel.org/doc/Documentation/nommu-mmap.txt https://www.kernel.org/doc/Documentation/nommu-mmap.txt And apparently, good enough to run DOOM: https://hackaday.com/2022/12/07/a-tiny-risc-v-emulator-runs-linux-with-no-mmu-and-yes-it-runs-doom/ https://hackaday.com/2022/12/07/a-tiny-risc-v-emulator-runs-...
- 4y ago
- yuuta 4y agoThat's a pretty good article, but I expect more in-depth information on booting modern Intel CPUs ... I am very interested in modern UEFI / BIOS firmware development and how do they bring up x86 CPUs, but unfortunately there are very little source (I guess, except for EDK2), and the majority (?) of x86 firmwares are proprietary. Booting x86 is much more complicated than writing a linker script with a vector table for your microcontroller ... so, this seems very interesting.
- evilos 4y agoSo I guess this is what Bryan Cantrill meant when he said that a million kittens were slaughtered every time you boot your CPU.
- empiricus 4y agoSometimes I think our universe works by killing kittens (1 mil every hour). (The alternative explanation for all the dead kittens is just Moloch).
- vrglvrglvrgl 4y ago[dead]
- concerned_ 4y ago[flagged]
- yjftsjthsd-h 4y ago? The article very specifically talks about how the process has changed over the years, even to the point of UEFI not starting the OS/bootloader in real mode anymore.
- concerned_ 4y ago[flagged]
- yjftsjthsd-h 4y ago> It does not It literally does: >>> For modern UEFI systems, the firmware that's launched from the reset vector then reprograms the CPU into a sensible mode (ie, one without all this segmentation bullshit), > you could not use this article to boot any Intel CPU. I mean, the article doesn't contain machine code or assembly listings, but it does a decent job of describing the process. What is your actual criticism, either of the article or the described CPUs?
- concerned_ 4y ago[flagged]
- rwmj 4y agoIntel actually released a variant of the 80386 which booted into protected mode and lacked real mode entirely. It was as far as I know not very successsful: https://en.wikipedia.org/wiki/Intel_80376 https://en.wikipedia.org/wiki/Intel_80376
- mjg59 4y agoSimultaneously lacking real mode and paging support did kind of restrict it to embedded use cases
- senko 4y agoIn 1989, people very much used real mode, so it's not surprising this failed. (also, it was for embedded systems only according to that Wikipedia article) In 2023, not so much. It's baffling to me this is still supported, when all the other changes in the hardware basically make it impossible to run anything that old on the modern hardware (you can do that in QEMU, but you can then emulate x86 in software fast enough anyway).
- voxadam 4y agoWhat's the actual reason for real mode still being supported on modern processors in this day in age? Why didn't it die with the advent of AMD64 (aka x86-64)? Why didn't AMD skip real mode and boot directly into something more modern?
- pwg 4y agoMost likely because for protected mode to function, there is a certain minimal amount of housekeeping data tables that need to be setup properly (i.e. LDT, GDT, IDT, etc), otherwise you'll just immediately take a double fault and the CPU will halt. Real mode exists today as a gateway to setting up all those housekeeping data tables so that once the "protected mode" switch is flipped on, the CPU will actually find code to execute.
- gpderetta 4y ago- There is likely very little to no cost in keeping it in - There might be even a cost in removing it. - Complexity is a barrier to entry to any competitor that want to produce compatible CPUs.
- lynguist 4y agoThere are so many steps and sidesteps there! Could someone who has access to GPT-4 API feed this text in and ask it to produce the topological or chronological step-by-step form of it? I’d be curious to see it.
- porphyry3 4y ago> ...reprograms the CPU into a sensible mode (ie, one without all this segmentation bullshit)... Protected mode still has segmentation except that segment registers now index into either a global or local descriptor table (DT). Linux tries to make this transparent by setting up descriptors where logical addresses coincide with linear addresses (at least for CS and DS).
- TacticalCoder 4y agoThat is very interesting. > I'm also missing out the fact that this entire process only kicks off after the Management Engine says it can, which means we're waiting for an entirely independent x86 to boot an entire OS before our CPU even starts pretending to execute the system firmware. I take it that that OS is Minix? > But what verifies the first component in the boot chain? You can't simply ask the BIOS to verify itself - if an attacker can replace the BIOS, they can replace it with one that simply lies about having done so. Intel's solution to this is called Boot Guard. Wait... How can an attacker replace the BIOS? Aren't motherboard nowadays protected from unwarranted BIOS flashing? Say I'm an attacker and I got root on some PC (Intel or AMD) running Linux, how do I replace the BIOS with a backdoored BIOS without the user noticing?
- mjg59 4y ago> I take it that that OS is Minix? It's the Minix kernel, I don't think the userland contains much Minix. > Wait... How can an attacker replace the BIOS? If you have physical access you can just attach to the flash directly and reprogram it. This is very much in-scope for various people.
- LoganDark 4y ago> If you have physical access you can just attach to the flash directly and reprogram it. This is very much in-scope for various people. Not to mention just replacing the motherboard since the CPU is socketed and could go anywhere.
- smikhanov 4y agoThere's a very detailed but less emotional writeup on the same topic from 2018, highly recommended: https://binarydebt.wordpress.com/2018/10/06/how-does-an-x86-processor-boot/ https://binarydebt.wordpress.com/2018/10/06/how-does-an-x86-...
- ignoramous 4y agoAnd a how computers boot discussion this previous month: https://news.ycombinator.com/item?id=35229045 https://news.ycombinator.com/item?id=35229045
- deleted 4y ago[deleted]
- vaxman 4y ago"It then reads the initial block of the firmware (the Initial Boot Block, or IBB) into RAM (or, well, cache, as previously described) and parses it. There's a block that contains a public key - it hashes that key and verifies that it matches the SHA256 from the fuses. It then uses that key to validate a signature on the IBB. If it all checks out, it executes the IBB and everything starts looking like the nice simple model we had before." Should all be performed by ME/PSP now, but the Intel ME/AMD PSP can't be updated (even with new microcode) and older versions are known to contain serious vulnerabilities. Thus we can reduce all of our options down to either (a) not running with Secure Boot or (b) running with Secure Boot, but only on the very latest Intel/AMD processors. So if we have the very latest processors, then we only need to worry about validating "critical" microcode updates to them and that doesn't need to happen at power-on at all, it can happen in the ME/PSP, but we do need to know about them at the C-suite level, since it will impact purchasing of replacement gear, financial statements and risk analysis decisions. That means we should power-on directly into the on-die ME/PSP and it, in turn, can bring up the RAM, PCI, etc. and check (online or on attached storage) for "critical" updates and, if found, emit a "distress beacon" on the network while also checking for a signal (from a resistor, the network or attached storage) that will inform it to download and update (rather than simply halting) the processor. This allows Management (the ones who have to sign SarBox, not just the system admins) to be informed of emerging vulnerabilities in their infrastructure (because of the "distress beacons" triggering indicators on their enterprise dashboards) so that they may choose to take explicit action to allow processing to continue (by emitting the signal that all ME/PSP would look for to enable critical microcode updates instead of halting). Likewise, we need ME/PSP to also start verifying all connected communication interfaces (before and after boot) by checking digital signatures that chip manufacturers obtain from Intel/AMD --much like what developers have to do to get their code to run in kernel space on an Apple Mac. I mean, CVE-2022-21742, et. al. --not just Thunderbolt and USB attacks. The only other rational solution really is to run with Secure Boot disabled (because older Intel/AMD processors cannot be trusted anyway --thanks to their non-updatable components like ME/PSP-- even with post-production microcode updates, even with Secure Boot). Anyway, 11 years later, will leave this here .. https://www.extremetech.com/defense/133773-rakshasa-the-hardware-backdoor-that-china-could-embed-in-every-computer https://www.extremetech.com/defense/133773-rakshasa-the-hard...
- userbinator 4y agoJust like the locked-down mobile devices that have unfortunately perverted the nature of general-purpose computing, Boot Guard is a tool of planned obsolescence and manufacturer control disguised as "security". Want to fix something in the BIOS that they didn't want you to have[1][2][3]? Too bad, it's locked and they won't release a newer version to force you to buy another. Absolute bastards. [1] https://news.ycombinator.com/item?id=29837884 https://news.ycombinator.com/item?id=29837884 [2] https://news.ycombinator.com/item?id=28254571 https://news.ycombinator.com/item?id=28254571 [3] https://news.ycombinator.com/item?id=33650347 https://news.ycombinator.com/item?id=33650347
- superkuh 4y agohttps://archive.is/XoghM https://archive.is/XoghM These days dreamwidth.org is harder to view and interact with than facebook.com if you don't have an account. We really need to stop linking to it and link instead to an archive.is copy or the like. After the run-around to the archived copy I see mgj is still complaining about people being able to boot in modes other than UEFI. I'm glad these options still exist. Throwing away all the legacy computing options would remove many abilities no longer possible on modern hardware and software stacks.
- cronix 4y ago> These days dreamwidth.org is harder to view and interact with than facebook.com if you don't have an account. We really need to stop linking to it and link instead to an archive.is copy or the like. I've made the suggestion to Dang in the past to just autogenerate an archive.is link for every story/link posted to HN and have that be an "alternate link" after the main one. I think it's kind of silly some people just post an archive link for every post and gets a buttload of points for it as everyone upvotes it which games the system. I think it would also be good in general to preserve the article as it appeared at the time when initially discussed and hasn't been edited, or removed, since.
- masfuerte 4y agoIt's fine with js disabled.
- mjg59 4y ago> I see mgj is still complaining about people being able to boot in modes other than UEFI I'm not sure how you get that impression, since I'm mostly talking about what happens before you get to that point. Having the CPU hand off control to the firmware in protected mode doesn't preclude the firmware switching back to real mode.
- MichaelZuo 4y agoWhat exactly does "...the CPU is hardcoded to start reading instructions from when power is applied." mean? How do the hardcoded parts read anything with just an electrical current?
- deepspace 4y agoI am not sure what you mean by "just an electrical current". The CPU is typically kept in reset until the clock is stable, so it has a valid clock signal on startup. It is therefore able to start executing the microcode which performs the power-on-reset sequence, i.e. start reading instructions from the reset vector and so on.
- MichaelZuo 4y ago> CPU is typically kept in reset until the clock is stable > It is therefore able to start executing the microcode At the very beginning, what is keeping it in 'reset' and initiating the execution of the microcode? Through what means? From what I understand, in the first few hundred nanoseconds it's just an electrical current that's flowing through the CPU and nothing else.
- convolvatron 4y agoyeah, ok, reset would normally be held low - whether that's the presence or absence of a current isn't important. in the the old days there was usually a hardware power controller that doesn't raise reset until the power is stable. on those machines I think that just took long enough that the clock chain had settled out. these days its more likely that there is a system controller orchestrating the bringup. potentially waiting for all the voltage converters and the clock generator to settle before raising the reset line. depending on the architecture it may not really be a simple pin, but addressed by the system board controller through the scan network.
- MichaelZuo 4y agoHow is the system controller initiated? I assume via a simpler process?
- wg0 4y agoThis might seem far fetched but a RISC-V take over in a decade or two is imminent even more so looking at the geopolitical vectors and trajectories.