5 ms·

> The Self-Boot Engine (SBE) (quantity: 1) is a core which is responsible for booting the entire system. It's responsible for initialising the chip and getting

> The Self-Boot Engine (SBE) (quantity: 1) is a core which is responsible for booting the entire system. It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode; it does little after that point. It has some SRAM to do its work in and uses a slightly custom variant of the 32-bit Power ISA, extended to support 64-bit loads and stores using adjacent GPR pairs. This core design is known as a PPE. It's the first thing that runs on the CPU die. What I’m curious about is how does the ‘Self Boot Engine’ initialize itself in the first few miliseconds? Maybe, a motherboard chip does the actual work, but that just invites the same question, at some point something must be initializing itself, how?

tux3 3y ago
A CPU can contain hardcoded memory, it can detect the power-on event. That we have an incredibly fancy chain of smaller CPUs booting bigger CPU is more surprising! Things should be initializing themselves. Those are 'just' voltage levels in the chip. I would 'simply' have the main core of the main CPU come out of power-on-reset in a vaguely usable state, and then let regular firmware on a NAND spin up the RAM and setup everything else! It's still not clear to me why is a second core needed to setup the first core the right way, instead of coming out of reset the right way
- thrtythreeforty 3y ago
  Frequently (I work on large SOCs) it's less risk and more flexible to let software do boot. For example, there is some initialization step in the way between the big core and the instructions, and it is deemed easier/less risky to let software perform this initialization rather than do it in hardware and not have a fallback in case of hardware bugs. Or, if you have the fallback, then you've just reinvented the simple-core boot mechanism so you might as well ditch the hardware work. I hate this trade-off because it means so much irritating work for the software engineer and frequently has downstream impact on other parts of the system (e.g. does PCIe link come up in time?) but the risk of silicon problems drives this design decision.
thrtythreeforty 3y ago
Eventually some core boots from a ROM on the die or (less frequently) off-chip flash storage. As to how that is accomplished, well, the core's logic is designed so that it reads the instruction from the reset vector in ROM on the first clock cycle after reset is released. Software has control after that.
- MichaelZuo 3y ago
  I'm still a bit confused about the exact mechanism, how does the core logic 'read' anything before it's initialized? What is doing the reading?
  - pixelfarmer 3y ago
    Instruction fetcher + address lines and how these things are wired together. If you are really interested, there are various people having big sets of videos about creating an 8 bit CPU from scratch with TTL logic chips. Alternatively, or on top of that, you can dig up reference / hardware manuals about relatively simple CPUs / SoCs and look at the initialization parts and pieces.
    - MichaelZuo 3y ago
      Thanks for the info, though I think the point where I'm curious about is how exactly the 'TTL logic chip' itself starts-up. Are there any 'how-to-make TTL logic chips from scratch' resources that you know of? For example, the die images and schematic of the most basic TTL chip, the SN7400: https://en.wikipedia.org/wiki/File:SN7400_1965.jpg https://en.wikipedia.org/wiki/File:SN7400_1965.jpg https://en.wikipedia.org/wiki/File:TTL-00-die-schema.jpg https://en.wikipedia.org/wiki/File:TTL-00-die-schema.jpg Show odd patterns and I can't figure out how pushing electricity across it will make it do anything other than heat up.
      - pixelfarmer 3y ago
        Sorry for the late reply! What you see on these dies can also be recreated manually with transistors, diodes, resistors, and maybe the odd capacitor. https://en.wikipedia.org/wiki/7400-series_integrated_circuits https://en.wikipedia.org/wiki/7400-series_integrated_circuit... shows further down the on die circuit. There is https://en.wikipedia.org/wiki/NAND_gate https://en.wikipedia.org/wiki/NAND_gate to give you further examples, plus there are links to TTL logic in general and all sorts of other gates. It should be easy to find pages that show how to recreate this stuff with actual transistors and resistors (or youtube videos for that matter). The startup is pretty much just powering these chips and maybe having the input lines defaulting to for example "low" with resistors for a defined input and then output state (if the chips don't offer that themselves already -> chip manual). However, all that doesn't happen in zero time, same as signal propagation which needs time, but this is also part of such chip manuals to tell you about the timings, max frequencies etc. they can be operated at.
hlandau 3y ago
Since all POWER9 firmware is open, I can actually answer this: The very first instructions executed by the SBE are from OTPROM (one-time programmable ROM, set at the factory). This is just a few instructions: https://github.com/open-power/sbe/blob/master/src/boot/otprom_init.S https://github.com/open-power/sbe/blob/master/src/boot/otpro... The first SBE code embodied in a mutable nonvolatile storage device executed is here: https://github.com/open-power/sbe/blob/master/src/boot/loader_l1.S https://github.com/open-power/sbe/blob/master/src/boot/loade... Interestingly the SBE code is actually stored on a serial I2C EEPROM on the CPU module itself, not on the host. This is quite unusual from an x86 perspective where there is usually not any mutable nonvolatile state on the CPU itself. POWER9 is also a little unusual in that just powering on the CPU doesn't cause it to do anything by itself. You have to talk to the CPU over FSI (Flexible Service Interface), a custom IBM clock+data control protocol which is mastered by the BMC and used by the BMC to send a 'Go' signal after powering on the system's power rails. This FSI interface is basically the CPU's "front port" - think of an operator front panel on an old mainframe. It's kind of fascinating how IBM's system designs still reflect this kind of past. Indeed, until very recently (I think this only changed with POWER8, but it may have been POWER7) IBM's own POWER server CPUs weren't designed to boot, in the sense of being brought up in a self-sufficient way - they were externally IPLed by the FSP (IBM's version of a BMC). Basically, what we think of as boot firmware would actually execute on the FSP - which is a PPC476 running Linux - so you have the boot firmware executing as a userspace Linux program on the FSP, doing register pokes to get the CPU initialised, do memory training, etc. all remotely via the FSI master, rather than it being done on the CPU itself. It would even load the hypervisor, PowerVM. The CPU itself traditionally was essentially held in reset until the service processor had completed all this init. So the POWER CPUs have traditionally all been initialised externally, rather than via 'bootstrapping' in the literal sense. Even memory training was done this way, all via the CPU's front port. What's really amazing is that when IBM wanted to switch to a self-boot model around the time of POWER8 (I think), they looked at their boot/IPL code, which was designed under the assumption it would be running on an external processor and initialise the CPU by probing it. Their response was to write a C++-level emulation environment for all the internal API calls this firmware uses to access the hardware. Basically, you have hardware initialisation procedures written as though running on a separate machine, and then a C++ framework all of this is hosted in which pretends that this is still the case, but allows all of these hardware initialisation procedures to be reused without having to rewrite them all. Amazingly it works. Though the size of this boot firmware is so large, it even has to have a paging mechanism for its own code while running in CAR mode(!). http://github.com/open-power/hostboot http://github.com/open-power/hostboot With POWER8/9 since IBM moved to a self-boot model, the FSI interface is often just used to send the 'Go' signal, and I believe by the BMC to query temperature information after boot. But the CPU won't do anything until you send that signal. It's kind of cute, really. Sending the 'Go' signal starts the SBE running. And yes, this means that all of the hardware initialisation procedures which were written to be used from a separate service processor are now never used that way. This kind of 'front port' based design to system 'IPL' in which a service processor just pokes the CPU remotely to initialise it is pretty fascinating. It's like the idea of automating what a human operator would have done to initialise a mainframe long, long, ago (makes me think of the autopilot in Airplane). Though of course the amount of stuff you'd have to do flipping switches on an operator panel to initialise one of these CPUs may take a lifetime if done manually... So this is an example of how IBM's technical history very definitely lives on and is reflected in their systems engineering today. Worth noting an advantage of this FSI interface is that it's also effectively a hardware debugger interface. In fact I believe the Talos/Blackbird systems ship with the 'pdbg' hardware debugger tool accessible via BMC SSH shell. So these systems effectively have a hardware debugger built in, which is just a Linux userspace tool which pokes the CPU's debug registers over FSI. https://github.com/open-power/pdbg https://github.com/open-power/pdbg The idea of 'front ports' seems pretty rare nowadays in most SoCs. Or rather, it actually does exist: it's called JTAG. In that regard IBM's traditional boot-via-FSI model isn't so different from the idea of initialising a chip at boot using JTAG. ...I'm fairly sure I've heard some horror stories in which some systems actually do this.

tux3 3y ago

A CPU can contain hardcoded memory, it can detect the power-on event. That we have an incredibly fancy chain of smaller CPUs booting bigger CPU is more surprising! Things should be initializing themselves. Those are 'just' voltage levels in the chip. I would 'simply' have the main core of the main CPU come out of power-on-reset in a vaguely usable state, and then let regular firmware on a NAND spin up the RAM and setup everything else! It's still not clear to me why is a second core needed to setup the first core the right way, instead of coming out of reset the right way

thrtythreeforty 3y ago
Frequently (I work on large SOCs) it's less risk and more flexible to let software do boot. For example, there is some initialization step in the way between the big core and the instructions, and it is deemed easier/less risky to let software perform this initialization rather than do it in hardware and not have a fallback in case of hardware bugs. Or, if you have the fallback, then you've just reinvented the simple-core boot mechanism so you might as well ditch the hardware work. I hate this trade-off because it means so much irritating work for the software engineer and frequently has downstream impact on other parts of the system (e.g. does PCIe link come up in time?) but the risk of silicon problems drives this design decision.

thrtythreeforty 3y ago

Eventually some core boots from a ROM on the die or (less frequently) off-chip flash storage. As to how that is accomplished, well, the core's logic is designed so that it reads the instruction from the reset vector in ROM on the first clock cycle after reset is released. Software has control after that.

MichaelZuo 3y ago
I'm still a bit confused about the exact mechanism, how does the core logic 'read' anything before it's initialized? What is doing the reading?
- pixelfarmer 3y ago
  Instruction fetcher + address lines and how these things are wired together. If you are really interested, there are various people having big sets of videos about creating an 8 bit CPU from scratch with TTL logic chips. Alternatively, or on top of that, you can dig up reference / hardware manuals about relatively simple CPUs / SoCs and look at the initialization parts and pieces.
  - MichaelZuo 3y ago
    Thanks for the info, though I think the point where I'm curious about is how exactly the 'TTL logic chip' itself starts-up. Are there any 'how-to-make TTL logic chips from scratch' resources that you know of? For example, the die images and schematic of the most basic TTL chip, the SN7400: https://en.wikipedia.org/wiki/File:SN7400_1965.jpg https://en.wikipedia.org/wiki/File:SN7400_1965.jpg https://en.wikipedia.org/wiki/File:TTL-00-die-schema.jpg https://en.wikipedia.org/wiki/File:TTL-00-die-schema.jpg Show odd patterns and I can't figure out how pushing electricity across it will make it do anything other than heat up.
    - pixelfarmer 3y ago
      Sorry for the late reply! What you see on these dies can also be recreated manually with transistors, diodes, resistors, and maybe the odd capacitor. https://en.wikipedia.org/wiki/7400-series_integrated_circuits https://en.wikipedia.org/wiki/7400-series_integrated_circuit... shows further down the on die circuit. There is https://en.wikipedia.org/wiki/NAND_gate https://en.wikipedia.org/wiki/NAND_gate to give you further examples, plus there are links to TTL logic in general and all sorts of other gates. It should be easy to find pages that show how to recreate this stuff with actual transistors and resistors (or youtube videos for that matter). The startup is pretty much just powering these chips and maybe having the input lines defaulting to for example "low" with resistors for a defined input and then output state (if the chips don't offer that themselves already -> chip manual). However, all that doesn't happen in zero time, same as signal propagation which needs time, but this is also part of such chip manuals to tell you about the timings, max frequencies etc. they can be operated at.

hlandau 3y ago

Since all POWER9 firmware is open, I can actually answer this: The very first instructions executed by the SBE are from OTPROM (one-time programmable ROM, set at the factory). This is just a few instructions: https://github.com/open-power/sbe/blob/master/src/boot/otprom_init.S https://github.com/open-power/sbe/blob/master/src/boot/otpro... The first SBE code embodied in a mutable nonvolatile storage device executed is here: https://github.com/open-power/sbe/blob/master/src/boot/loader_l1.S https://github.com/open-power/sbe/blob/master/src/boot/loade... Interestingly the SBE code is actually stored on a serial I2C EEPROM on the CPU module itself, not on the host. This is quite unusual from an x86 perspective where there is usually not any mutable nonvolatile state on the CPU itself. POWER9 is also a little unusual in that just powering on the CPU doesn't cause it to do anything by itself. You have to talk to the CPU over FSI (Flexible Service Interface), a custom IBM clock+data control protocol which is mastered by the BMC and used by the BMC to send a 'Go' signal after powering on the system's power rails. This FSI interface is basically the CPU's "front port" - think of an operator front panel on an old mainframe. It's kind of fascinating how IBM's system designs still reflect this kind of past. Indeed, until very recently (I think this only changed with POWER8, but it may have been POWER7) IBM's own POWER server CPUs weren't designed to boot, in the sense of being brought up in a self-sufficient way - they were externally IPLed by the FSP (IBM's version of a BMC). Basically, what we think of as boot firmware would actually execute on the FSP - which is a PPC476 running Linux - so you have the boot firmware executing as a userspace Linux program on the FSP, doing register pokes to get the CPU initialised, do memory training, etc. all remotely via the FSI master, rather than it being done on the CPU itself. It would even load the hypervisor, PowerVM. The CPU itself traditionally was essentially held in reset until the service processor had completed all this init. So the POWER CPUs have traditionally all been initialised externally, rather than via 'bootstrapping' in the literal sense. Even memory training was done this way, all via the CPU's front port. What's really amazing is that when IBM wanted to switch to a self-boot model around the time of POWER8 (I think), they looked at their boot/IPL code, which was designed under the assumption it would be running on an external processor and initialise the CPU by probing it. Their response was to write a C++-level emulation environment for all the internal API calls this firmware uses to access the hardware. Basically, you have hardware initialisation procedures written as though running on a separate machine, and then a C++ framework all of this is hosted in which pretends that this is still the case, but allows all of these hardware initialisation procedures to be reused without having to rewrite them all. Amazingly it works. Though the size of this boot firmware is so large, it even has to have a paging mechanism for its own code while running in CAR mode(!). http://github.com/open-power/hostboot http://github.com/open-power/hostboot With POWER8/9 since IBM moved to a self-boot model, the FSI interface is often just used to send the 'Go' signal, and I believe by the BMC to query temperature information after boot. But the CPU won't do anything until you send that signal. It's kind of cute, really. Sending the 'Go' signal starts the SBE running. And yes, this means that all of the hardware initialisation procedures which were written to be used from a separate service processor are now never used that way. This kind of 'front port' based design to system 'IPL' in which a service processor just pokes the CPU remotely to initialise it is pretty fascinating. It's like the idea of automating what a human operator would have done to initialise a mainframe long, long, ago (makes me think of the autopilot in Airplane). Though of course the amount of stuff you'd have to do flipping switches on an operator panel to initialise one of these CPUs may take a lifetime if done manually... So this is an example of how IBM's technical history very definitely lives on and is reflected in their systems engineering today. Worth noting an advantage of this FSI interface is that it's also effectively a hardware debugger interface. In fact I believe the Talos/Blackbird systems ship with the 'pdbg' hardware debugger tool accessible via BMC SSH shell. So these systems effectively have a hardware debugger built in, which is just a Linux userspace tool which pokes the CPU's debug registers over FSI. https://github.com/open-power/pdbg https://github.com/open-power/pdbg The idea of 'front ports' seems pretty rare nowadays in most SoCs. Or rather, it actually does exist: it's called JTAG. In that regard IBM's traditional boot-via-FSI model isn't so different from the idea of initialising a chip at boot using JTAG. ...I'm fairly sure I've heard some horror stories in which some systems actually do this.