18 ms·

Modern CPUs have a backstage cast

212 points by hlandau 3y ago

giuliomagnifico 3y ago
I understood nothing (as a sysadmin) but this looks like a very interesting article for who can understand it.
- bicolao 3y ago
  I think you can see a modern CPU as a network. There are some beefy servers doing all the heavy lifting which is what the outsiders see. But there's also a few smaller servers here and there monitoring the system (or even responsible for powering on the entire network).
  - hlandau 3y ago
    Author here. This is very much the case for a computer system as a whole also. Basically a network of cooperating microprocessors, including in I/O peripherals etc. PCIe in particular is literally a packet-switched computer network - it has a physical layer, data link layer, and a transaction layer which is basically packet switched. There are even proprietary solutions for tunnelling PCIe over Ethernet.
    - p_l 3y ago
      To make it even funnier - Digital's last Alpha CPU, EV7, which was essentially the ancestor of AMD K8 (which finally brought "mesh" networking to mainstream PCs), actually had IP-based internal management network! Each EV7 computer had, instead of normal BMC, a bigger management node connected to 10MBit ethernet hub (twisted ethernet, fortunately :P), and this network was then connected to things like I/O boards, power control, system boards... including to each individual EV7 CPU. Each so connected component had a small CPU with ethernet that was responsible for interfacing their specific component to the network, and when the system booted part of it involved prodding the CPUs over ethernet to put them into appropriate halt state from which they could start booting.
      - classichasclass 3y ago
        This kind of thing with functional domains accessible over Ethernet existed in at least one laptop as well, where you could connect to the "nodes" once you busted into it (my article): https://oldvcr.blogspot.com/2023/04/of-sun-ray-laptops-mips-and-getting.html https://oldvcr.blogspot.com/2023/04/of-sun-ray-laptops-mips-...
    - di4na 3y ago
      And you have smaller one that basically pxe boot the bigger one and manage the power, cooling, etc. It is datacenters all the way down. As someone that used to do embedded, there is a reason i felt most at home in erlang and elixir. Their processes that share nothing and use message passing was really close to how it looks to build and code for an embedded platform.
travisgriggs 3y ago
I miss these kinds of articles on the net. Is anyone else reminded of the CPU Praxis articles that were part of ARS Technica's early rise to popularity? I really miss those. This article, is of course, much shorter, but still, I miss that sort of content on the internet.
buildbot 3y ago
“Turtles all the way down” Modern CPUs are so complex you need simpler ones to abstract it! Very cool breakdown of how power9 does this.
chasil 3y ago
"...this is interesting is because POWER9 is basically the first time the public got a real view of how sophisticated the backstage cast actually is of a modern server CPU." Not quite correct; the OpenSPARC T1 and T2 were publicly released and available by 2008. https://www.oracle.com/servers/technologies/opensparc.html https://www.oracle.com/servers/technologies/opensparc.html "Large parts of this process are handled by vendor-supplied mystery firmware blobs, which may as well be boxes with “???” written in them. The maintainers of the me_cleaner script likely have the clearest view of what is known. https://github.com/corna/me_cleaner https://github.com/corna/me_cleaner
- hlandau 3y ago
  >Not quite correct; the OpenSPARC T1 and T2 were publicly released and available by 2008. Points for mentioning this! But things have come a long way since 2008. You can get Intel ME-less machines from the 2008 era. Not sure if OpenSPARC T2 has any management cores. >The maintainers of the me_cleaner script likely have the clearest view of what is known. Yep, absolutely. Much of what we know is thanks to the efforts of researchers like these. See also the talks on finding the 'Red Unlock' mode of modern Intel CPUs.
  - jakeogh 3y ago
    Microcode access on Atom https://www.youtube.com/watch?v=5Pq1FmxS6H8 https://www.youtube.com/watch?v=5Pq1FmxS6H8
kccqzy 3y ago
> It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode The somewhat surprising but true implication is that on boot, the CPU is initialized before the RAM is initialized. So there is a window of time during boot when the main core on the CPU is running instructions that cannot access the RAM. Even on register-starved x86 it is possible to write code without using RAM, but it certainly seems more convenient to me to treat the cache as RAM. Documentation for a special compiler that compiles to code that doesn't use RAM: https://github.com/wt/coreboot/blob/master/util/romcc/romcc.1 https://github.com/wt/coreboot/blob/master/util/romcc/romcc....
- intelVISA 3y ago
  CAR is a gem. It's great for lite OSes in hostile envs.
  - derefr 3y ago
    Funny enough, a modern CPU doing CAR still has more memory than a PC from the 1980s. Presuming you statically recompiled them, you could run entire SNES games from a modern CPU’s cache! (And that being said, now I’m wondering whether you could force eviction and retainment into L3 cache on demand, to achieve something like memory bank switching…)
    - JonathonW 3y ago
      SNES games? You could comfortably run Windows 95 within the L3 cache on many recent Intel processors (the one in my 2019-era MBP has 16 MB of L3 onboard; current generation processors go even bigger and Windows 95 only needs 4 MB). It's not really clear to me from the limited bits of info that I've read whether or not L3 is guaranteed to be accessible when doing CAR, but, if it is, you've got enough memory available to do a lot of stuff. (And even the L2 cache is starting to get pretty big on the higher-end current-gen chips.)
      - hlandau 3y ago
dist-epoch 3y ago
A modern motherboard can update it's BIOS from a USB stick WITHOUT a CPU or memory installed. Think about that. The motherboard "knows" how to read a FAT file system from a USB mass storage device, verify it's digital signature and flash it with no main CPU or memory.
- wmf 3y ago
  I assume this is done with a microcontroller on the board.
  - chasil 3y ago
    The ARC processor was formerly in the northbridge of the chipset. Intel has since replaced this with an 80486 in modern designs; perhaps it also is implemented in the northbridge. https://en.wikipedia.org/wiki/ARC_(processor) https://en.wikipedia.org/wiki/ARC_(processor)
    - wmf 3y ago
      I think you're talking about the ME but I don't think the ME is responsible for "BIOS" flashing. I think it must be a separate microcontroller. This is kind of the point of the original blog post: don't go looking for "the microcontroller" because there isn't just one; there are many.
  - sebazzz 3y ago
    And also born out necessity, given that many Intel and AMD boards can't be booted with a too new CPU if the BIOS doesn't know about it - not even for flashing a new BIOS - so you needed to borrow an old CPU just for the sake of upgrading the BIOS.
    - ls612 3y ago
      It was originally to solve the issue where if you lost power flashing your bios you’d brick the system irrevocably. Now even if the bios is corrupt and the system won’t boot you can reflash a known good firmware with stock settings to get back up and running.
wmf 3y ago
Much of the openness of Power7/8/9 was encouraged by Google who wanted to have control over all the firmware, even the secret firmware. I think Google is also auditing PSP/ME source code but the public only sees the audit results.
shrubble 3y ago
A while ago I bought some older AMD 8350 systems, which apparently are the last without a PSP, the platform security processor. I did this as a sort of 'just in case' setup, was planning to put OpenSolaris on it and run things under Zones or LX zones and to run it as a backup server. Fast enough to get some work done and possibly more secure if the PSP is ever used/broken maliciously...
- LeonTheremin 3y ago
  [dead]
- jacquesm 3y ago
  That may well end up being a very prescient move. Be prepared to be labeled a tinfoil hat type until then, but I definitely think you are wise to take a precaution.
JdeBP 3y ago
As the author of https://superuser.com/a/347115/38062 https://superuser.com/a/347115/38062 and https://superuser.com/a/345333/38062 https://superuser.com/a/345333/38062, you have my sympathy about the "pack of lies" involving real mode and several wrong combinations of selector and offset.
- hlandau 3y ago
  Had to reverse engineer a real mode PCI option ROM once... that was extremely unpleasant [1]. And then of course there's "Unreal Mode". Moreover Intel is just this week actually finally proposing removing real mode. [2] I'm a bit worried for what this means for emulation of old 16-bit Windows and DOS software under Wine (one of the great ironies that Wine can still run Win16 programs on an x64 host OS when Windows can't) - though I suspect the performance requirements of such software is so low by modern standards that emulating such programs wouldn't pose any challenge. [1] https://www.devever.net/~hl/ortega https://www.devever.net/~hl/ortega [2] https://www.phoronix.com/news/Intel-X86-S-64-bit-Only https://www.phoronix.com/news/Intel-X86-S-64-bit-Only
  - JdeBP 3y ago
    See https://news.ycombinator.com/item?id=36074093 https://news.ycombinator.com/item?id=36074093 for a more significant worry. Emulating a CPU is not affected as much as code that would otherwise have still run on the bare hardware.
- JdeBP 3y ago
  It's also worth adding that none of this is new. There's always been a reason that the "C" in "CPU" has stood for "central". The idea that there are other, non-central, processors around the place goes back a long time. Four particular ones come to mind: * The DPT range of SCSI host bus adapter cards, many years ago, had an full blown MC680x0 processor on the card. * Connor Krukosky, who famously installed a mainframe in his basement with a console front-end processor that was a PC machine running OS/2. * PC/AT keyboards had on-board microcontrollers running programs. * And of course who can forget the BBC Micro's Tube? It's the short period in history where people thought that computers came with only one processor that is the real oddity. (-:
Simplicitas 3y ago
Doesn’t mention the special core reserved for the NSA and other national security agencies :-)
- LeonTheremin 3y ago
  [dead]
tyingq 3y ago
I was surprised when I read how the Raspberry PI's GPU handles the booting. https://forums.raspberrypi.com/viewtopic.php?t=266130 https://forums.raspberrypi.com/viewtopic.php?t=266130
MichaelZuo 3y ago
> The Self-Boot Engine (SBE) (quantity: 1) is a core which is responsible for booting the entire system. It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode; it does little after that point. It has some SRAM to do its work in and uses a slightly custom variant of the 32-bit Power ISA, extended to support 64-bit loads and stores using adjacent GPR pairs. This core design is known as a PPE. It's the first thing that runs on the CPU die. What I’m curious about is how does the ‘Self Boot Engine’ initialize itself in the first few miliseconds? Maybe, a motherboard chip does the actual work, but that just invites the same question, at some point something must be initializing itself, how?
- tux3 3y ago
  A CPU can contain hardcoded memory, it can detect the power-on event. That we have an incredibly fancy chain of smaller CPUs booting bigger CPU is more surprising! Things should be initializing themselves. Those are 'just' voltage levels in the chip. I would 'simply' have the main core of the main CPU come out of power-on-reset in a vaguely usable state, and then let regular firmware on a NAND spin up the RAM and setup everything else! It's still not clear to me why is a second core needed to setup the first core the right way, instead of coming out of reset the right way
  - thrtythreeforty 3y ago
    Frequently (I work on large SOCs) it's less risk and more flexible to let software do boot. For example, there is some initialization step in the way between the big core and the instructions, and it is deemed easier/less risky to let software perform this initialization rather than do it in hardware and not have a fallback in case of hardware bugs. Or, if you have the fallback, then you've just reinvented the simple-core boot mechanism so you might as well ditch the hardware work. I hate this trade-off because it means so much irritating work for the software engineer and frequently has downstream impact on other parts of the system (e.g. does PCIe link come up in time?) but the risk of silicon problems drives this design decision.
- thrtythreeforty 3y ago
pabs3 3y ago
This reminds me of the "It's Time for Operating Systems to Rediscover Hardware" talk by Timothy Roscoe: https://www.usenix.org/conference/osdi21/presentation/fri-keynote https://www.usenix.org/conference/osdi21/presentation/fri-ke...

212 points by hlandau 3y ago

giuliomagnifico 3y ago

I understood nothing (as a sysadmin) but this looks like a very interesting article for who can understand it.

bicolao 3y ago
I think you can see a modern CPU as a network. There are some beefy servers doing all the heavy lifting which is what the outsiders see. But there's also a few smaller servers here and there monitoring the system (or even responsible for powering on the entire network).
- hlandau 3y ago
  Author here. This is very much the case for a computer system as a whole also. Basically a network of cooperating microprocessors, including in I/O peripherals etc. PCIe in particular is literally a packet-switched computer network - it has a physical layer, data link layer, and a transaction layer which is basically packet switched. There are even proprietary solutions for tunnelling PCIe over Ethernet.
  - p_l 3y ago
    To make it even funnier - Digital's last Alpha CPU, EV7, which was essentially the ancestor of AMD K8 (which finally brought "mesh" networking to mainstream PCs), actually had IP-based internal management network! Each EV7 computer had, instead of normal BMC, a bigger management node connected to 10MBit ethernet hub (twisted ethernet, fortunately :P), and this network was then connected to things like I/O boards, power control, system boards... including to each individual EV7 CPU. Each so connected component had a small CPU with ethernet that was responsible for interfacing their specific component to the network, and when the system booted part of it involved prodding the CPUs over ethernet to put them into appropriate halt state from which they could start booting.
    - classichasclass 3y ago
      This kind of thing with functional domains accessible over Ethernet existed in at least one laptop as well, where you could connect to the "nodes" once you busted into it (my article): https://oldvcr.blogspot.com/2023/04/of-sun-ray-laptops-mips-and-getting.html https://oldvcr.blogspot.com/2023/04/of-sun-ray-laptops-mips-...
  - di4na 3y ago
    And you have smaller one that basically pxe boot the bigger one and manage the power, cooling, etc. It is datacenters all the way down. As someone that used to do embedded, there is a reason i felt most at home in erlang and elixir. Their processes that share nothing and use message passing was really close to how it looks to build and code for an embedded platform.

travisgriggs 3y ago

I miss these kinds of articles on the net. Is anyone else reminded of the CPU Praxis articles that were part of ARS Technica's early rise to popularity? I really miss those. This article, is of course, much shorter, but still, I miss that sort of content on the internet.

buildbot 3y ago

“Turtles all the way down” Modern CPUs are so complex you need simpler ones to abstract it! Very cool breakdown of how power9 does this.

chasil 3y ago

"...this is interesting is because POWER9 is basically the first time the public got a real view of how sophisticated the backstage cast actually is of a modern server CPU." Not quite correct; the OpenSPARC T1 and T2 were publicly released and available by 2008. https://www.oracle.com/servers/technologies/opensparc.html https://www.oracle.com/servers/technologies/opensparc.html "Large parts of this process are handled by vendor-supplied mystery firmware blobs, which may as well be boxes with “???” written in them. The maintainers of the me_cleaner script likely have the clearest view of what is known. https://github.com/corna/me_cleaner https://github.com/corna/me_cleaner

hlandau 3y ago
>Not quite correct; the OpenSPARC T1 and T2 were publicly released and available by 2008. Points for mentioning this! But things have come a long way since 2008. You can get Intel ME-less machines from the 2008 era. Not sure if OpenSPARC T2 has any management cores. >The maintainers of the me_cleaner script likely have the clearest view of what is known. Yep, absolutely. Much of what we know is thanks to the efforts of researchers like these. See also the talks on finding the 'Red Unlock' mode of modern Intel CPUs.
- jakeogh 3y ago
  Microcode access on Atom https://www.youtube.com/watch?v=5Pq1FmxS6H8 https://www.youtube.com/watch?v=5Pq1FmxS6H8

kccqzy 3y ago

> It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode The somewhat surprising but true implication is that on boot, the CPU is initialized before the RAM is initialized. So there is a window of time during boot when the main core on the CPU is running instructions that cannot access the RAM. Even on register-starved x86 it is possible to write code without using RAM, but it certainly seems more convenient to me to treat the cache as RAM. Documentation for a special compiler that compiles to code that doesn't use RAM: https://github.com/wt/coreboot/blob/master/util/romcc/romcc.1 https://github.com/wt/coreboot/blob/master/util/romcc/romcc....

intelVISA 3y ago
CAR is a gem. It's great for lite OSes in hostile envs.
- derefr 3y ago
  Funny enough, a modern CPU doing CAR still has more memory than a PC from the 1980s. Presuming you statically recompiled them, you could run entire SNES games from a modern CPU’s cache! (And that being said, now I’m wondering whether you could force eviction and retainment into L3 cache on demand, to achieve something like memory bank switching…)
  - JonathonW 3y ago
    SNES games? You could comfortably run Windows 95 within the L3 cache on many recent Intel processors (the one in my 2019-era MBP has 16 MB of L3 onboard; current generation processors go even bigger and Windows 95 only needs 4 MB). It's not really clear to me from the limited bits of info that I've read whether or not L3 is guaranteed to be accessible when doing CAR, but, if it is, you've got enough memory available to do a lot of stuff. (And even the L2 cache is starting to get pretty big on the higher-end current-gen chips.)
    - hlandau 3y ago

dist-epoch 3y ago

A modern motherboard can update it's BIOS from a USB stick WITHOUT a CPU or memory installed. Think about that. The motherboard "knows" how to read a FAT file system from a USB mass storage device, verify it's digital signature and flash it with no main CPU or memory.

wmf 3y ago
I assume this is done with a microcontroller on the board.
- chasil 3y ago
  The ARC processor was formerly in the northbridge of the chipset. Intel has since replaced this with an 80486 in modern designs; perhaps it also is implemented in the northbridge. https://en.wikipedia.org/wiki/ARC_(processor) https://en.wikipedia.org/wiki/ARC_(processor)
  - wmf 3y ago
    I think you're talking about the ME but I don't think the ME is responsible for "BIOS" flashing. I think it must be a separate microcontroller. This is kind of the point of the original blog post: don't go looking for "the microcontroller" because there isn't just one; there are many.
- sebazzz 3y ago
  And also born out necessity, given that many Intel and AMD boards can't be booted with a too new CPU if the BIOS doesn't know about it - not even for flashing a new BIOS - so you needed to borrow an old CPU just for the sake of upgrading the BIOS.
  - ls612 3y ago
    It was originally to solve the issue where if you lost power flashing your bios you’d brick the system irrevocably. Now even if the bios is corrupt and the system won’t boot you can reflash a known good firmware with stock settings to get back up and running.

wmf 3y ago

Much of the openness of Power7/8/9 was encouraged by Google who wanted to have control over all the firmware, even the secret firmware. I think Google is also auditing PSP/ME source code but the public only sees the audit results.

shrubble 3y ago

A while ago I bought some older AMD 8350 systems, which apparently are the last without a PSP, the platform security processor. I did this as a sort of 'just in case' setup, was planning to put OpenSolaris on it and run things under Zones or LX zones and to run it as a backup server. Fast enough to get some work done and possibly more secure if the PSP is ever used/broken maliciously...

LeonTheremin 3y ago
[dead]
jacquesm 3y ago
That may well end up being a very prescient move. Be prepared to be labeled a tinfoil hat type until then, but I definitely think you are wise to take a precaution.

JdeBP 3y ago

As the author of https://superuser.com/a/347115/38062 https://superuser.com/a/347115/38062 and https://superuser.com/a/345333/38062 https://superuser.com/a/345333/38062, you have my sympathy about the "pack of lies" involving real mode and several wrong combinations of selector and offset.

hlandau 3y ago
Had to reverse engineer a real mode PCI option ROM once... that was extremely unpleasant [1]. And then of course there's "Unreal Mode". Moreover Intel is just this week actually finally proposing removing real mode. [2] I'm a bit worried for what this means for emulation of old 16-bit Windows and DOS software under Wine (one of the great ironies that Wine can still run Win16 programs on an x64 host OS when Windows can't) - though I suspect the performance requirements of such software is so low by modern standards that emulating such programs wouldn't pose any challenge. [1] https://www.devever.net/~hl/ortega https://www.devever.net/~hl/ortega [2] https://www.phoronix.com/news/Intel-X86-S-64-bit-Only https://www.phoronix.com/news/Intel-X86-S-64-bit-Only
- JdeBP 3y ago
  See https://news.ycombinator.com/item?id=36074093 https://news.ycombinator.com/item?id=36074093 for a more significant worry. Emulating a CPU is not affected as much as code that would otherwise have still run on the bare hardware.
JdeBP 3y ago
It's also worth adding that none of this is new. There's always been a reason that the "C" in "CPU" has stood for "central". The idea that there are other, non-central, processors around the place goes back a long time. Four particular ones come to mind: * The DPT range of SCSI host bus adapter cards, many years ago, had an full blown MC680x0 processor on the card. * Connor Krukosky, who famously installed a mainframe in his basement with a console front-end processor that was a PC machine running OS/2. * PC/AT keyboards had on-board microcontrollers running programs. * And of course who can forget the BBC Micro's Tube? It's the short period in history where people thought that computers came with only one processor that is the real oddity. (-:

Simplicitas 3y ago

Doesn’t mention the special core reserved for the NSA and other national security agencies :-)

LeonTheremin 3y ago
[dead]

tyingq 3y ago

I was surprised when I read how the Raspberry PI's GPU handles the booting. https://forums.raspberrypi.com/viewtopic.php?t=266130 https://forums.raspberrypi.com/viewtopic.php?t=266130

MichaelZuo 3y ago

> The Self-Boot Engine (SBE) (quantity: 1) is a core which is responsible for booting the entire system. It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode; it does little after that point. It has some SRAM to do its work in and uses a slightly custom variant of the 32-bit Power ISA, extended to support 64-bit loads and stores using adjacent GPR pairs. This core design is known as a PPE. It's the first thing that runs on the CPU die. What I’m curious about is how does the ‘Self Boot Engine’ initialize itself in the first few miliseconds? Maybe, a motherboard chip does the actual work, but that just invites the same question, at some point something must be initializing itself, how?

tux3 3y ago
A CPU can contain hardcoded memory, it can detect the power-on event. That we have an incredibly fancy chain of smaller CPUs booting bigger CPU is more surprising! Things should be initializing themselves. Those are 'just' voltage levels in the chip. I would 'simply' have the main core of the main CPU come out of power-on-reset in a vaguely usable state, and then let regular firmware on a NAND spin up the RAM and setup everything else! It's still not clear to me why is a second core needed to setup the first core the right way, instead of coming out of reset the right way
- thrtythreeforty 3y ago
  Frequently (I work on large SOCs) it's less risk and more flexible to let software do boot. For example, there is some initialization step in the way between the big core and the instructions, and it is deemed easier/less risky to let software perform this initialization rather than do it in hardware and not have a fallback in case of hardware bugs. Or, if you have the fallback, then you've just reinvented the simple-core boot mechanism so you might as well ditch the hardware work. I hate this trade-off because it means so much irritating work for the software engineer and frequently has downstream impact on other parts of the system (e.g. does PCIe link come up in time?) but the risk of silicon problems drives this design decision.
thrtythreeforty 3y ago

pabs3 3y ago

This reminds me of the "It's Time for Operating Systems to Rediscover Hardware" talk by Timothy Roscoe: https://www.usenix.org/conference/osdi21/presentation/fri-keynote https://www.usenix.org/conference/osdi21/presentation/fri-ke...