5 ms·

> It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode

by kccqzy 3y ago

> It's responsible for initialising the chip and getting it out of bed enough to the point where at least one of the main cores can run using cache-as-RAM mode The somewhat surprising but true implication is that on boot, the CPU is initialized before the RAM is initialized. So there is a window of time during boot when the main core on the CPU is running instructions that cannot access the RAM. Even on register-starved x86 it is possible to write code without using RAM, but it certainly seems more convenient to me to treat the cache as RAM. Documentation for a special compiler that compiles to code that doesn't use RAM: https://github.com/wt/coreboot/blob/master/util/romcc/romcc.1 https://github.com/wt/coreboot/blob/master/util/romcc/romcc....

intelVISA 3y ago
CAR is a gem. It's great for lite OSes in hostile envs.
- derefr 3y ago
  Funny enough, a modern CPU doing CAR still has more memory than a PC from the 1980s. Presuming you statically recompiled them, you could run entire SNES games from a modern CPU’s cache! (And that being said, now I’m wondering whether you could force eviction and retainment into L3 cache on demand, to achieve something like memory bank switching…)
  - JonathonW 3y ago
    SNES games? You could comfortably run Windows 95 within the L3 cache on many recent Intel processors (the one in my 2019-era MBP has 16 MB of L3 onboard; current generation processors go even bigger and Windows 95 only needs 4 MB). It's not really clear to me from the limited bits of info that I've read whether or not L3 is guaranteed to be accessible when doing CAR, but, if it is, you've got enough memory available to do a lot of stuff. (And even the L2 cache is starting to get pretty big on the higher-end current-gen chips.)
    - hlandau 3y ago
      In fact, the largest POWER9 CPUs have up to 110MB of L3... and Zen 4's L3 apparently maxes out at 384MB(!!).
      - Dylan16807 3y ago
        > Zen 4's L3 apparently maxes out at 384MB(!!). That's with them holding back and not adding any v-cache. If you stacked an extra 64MB on each of the 12 compute dies, you'd have 1152MB.
    - derefr 3y ago
      Well, keep in mind that in the sort of state the computer is in when doing CAR, you don't get to talk to storage devices; nor do you get the benefit of having some kind of ROM on the bus. I know Windows 95 is happy to run from 4MB RAM with access to a hard drive; but how much memory would W95 need for a "bootable live-CD environment" where the disk image must be resident (if compressed) in memory along with all work RAM? (This is why I compared to the SNES: if you have to map the SNES's RAM and [every bank of] the game's ROM, then you're looking at 4–16MB depending on the game. The SNES is pretty much the newest console whose games would entirely fit, I think.)
      - pezezin 3y ago
        The Ryzen 5800X3D has 96 MB of L3 cache, enough to fit the biggest NeoGeo or N64 games. The Zen 4-based Genoa-X will have up to 1 GB of L3 cache! You could comfortably fit Windows 95 there...
      - lproven 3y ago
        For a benchmark test for a British computer magazine in about 1996, I hand stripped down an installation of Windows 95 so that it could fit on a mid-1990s SSD: which is to say, a PCI slot device with 16 MB of DRAM which appeared to the computer as a disk drive. So, effectively, a 16 MB SSD. Not gigabytes, megabytes. So I can say with some authority that it is possible to have a complete running installation of Windows 95 in 16 megs of disk space; however I had to trim it pretty brutally to fit, so, for example, I think it had 2 fonts left, one of which was used for the widgets that display window-close boxes and so on. The problem was that the object of the exercise was the benchmark how much quicker it was — but there wasn't enough space left to install any applications, and our benchmark tool used real applications. So, although I did several days work and I did get paid for it, the effort was at heart in vain. But in principle yes you could run Windows 95 entirely out of 16 MB of cache memory, and if you had say 32 MB of cache memory it would be no problem at all.
- andai 3y ago
  What does hostile mean in this context?
  - sweetjuly 3y ago
    It might be desirable to not trust RAM as other agents who have access to physical memory could be tampering with it.
  - intelVISA 3y ago
    Any machine you may not have True Control over e.g. outside of JTAG range, in the cloud, runs proprietary software.
notacoward 3y ago
I got some exposure to this at SiCortex, where we had our own MIPS-based processors and so had to do many of these things in software. There was one ColdFire (embedded 68K) processor running μClinux per board, plus 27 of our own. This "Module Service Processor" would boot first, fetch a boot image from the one-per-system "System Service Processor" (pretty generic PC), load that via JTAG into each node's cache, then finally set each one loose to do things like memory registration and interconnect setup. This all set the stage for the actual Linux boot, which itself involved two stages with a switch_root in between. My very first assignment was to work on some of that MSP-to-node stuff, then later I had to dive into memory registration at least twice, even though both were pretty far from my real specialty. Small company, y'know. This kind of low-level work is significantly more complicated than even most kernel developers realize - hence the need for articles like OP. Ditto for anything on large (more than single-board) systems. The intersection of the two was, frankly, a bit exhausting. Just keeping track of all the moving parts and their respective states induced a cognitive load that made debugging other already-hard problems that much more difficult. My hat's off for anyone who has kept on doing that stuff longer than I did, or who has to do it in an environment where vendors are keeping so many secrets.
- lgg 3y ago
  This stuff is certainly pretty rarified these days. I remember when the PPC970 came out people were shocked how difficult it was to bootstrap. IBM didn't really care as POWER4 (from which it was derived) was not a merchant chip and they had management processors (and very high margins) in all their machines to handle it. Apple was the launch partner and even back then had a lot of in house expertise doing this sort of work. Everyone else who tried to use it was in for some real pain and most of them gave up. The guys doing the eval boards with support from IBM literally posted this: https://web.archive.org/web/20060715134515/http://www.970eval.com/970FAQ.html#anchor#4 https://web.archive.org/web/20060715134515/http://www.970eva... TL;DR, the last line is "Once all of the above is completed, the processor will be able to successfully fetch instructions from a boot source. You are now effectively at the same point you would have been 5 months ago, had this been a standard 750 bringup... Board bringup from this point should be very straightforward and follow established methods."

by kccqzy 3y ago

intelVISA 3y ago

CAR is a gem. It's great for lite OSes in hostile envs.

derefr 3y ago
Funny enough, a modern CPU doing CAR still has more memory than a PC from the 1980s. Presuming you statically recompiled them, you could run entire SNES games from a modern CPU’s cache! (And that being said, now I’m wondering whether you could force eviction and retainment into L3 cache on demand, to achieve something like memory bank switching…)
- JonathonW 3y ago
  SNES games? You could comfortably run Windows 95 within the L3 cache on many recent Intel processors (the one in my 2019-era MBP has 16 MB of L3 onboard; current generation processors go even bigger and Windows 95 only needs 4 MB). It's not really clear to me from the limited bits of info that I've read whether or not L3 is guaranteed to be accessible when doing CAR, but, if it is, you've got enough memory available to do a lot of stuff. (And even the L2 cache is starting to get pretty big on the higher-end current-gen chips.)
  - hlandau 3y ago
    In fact, the largest POWER9 CPUs have up to 110MB of L3... and Zen 4's L3 apparently maxes out at 384MB(!!).
    - Dylan16807 3y ago
      > Zen 4's L3 apparently maxes out at 384MB(!!). That's with them holding back and not adding any v-cache. If you stacked an extra 64MB on each of the 12 compute dies, you'd have 1152MB.
  - derefr 3y ago
    Well, keep in mind that in the sort of state the computer is in when doing CAR, you don't get to talk to storage devices; nor do you get the benefit of having some kind of ROM on the bus. I know Windows 95 is happy to run from 4MB RAM with access to a hard drive; but how much memory would W95 need for a "bootable live-CD environment" where the disk image must be resident (if compressed) in memory along with all work RAM? (This is why I compared to the SNES: if you have to map the SNES's RAM and [every bank of] the game's ROM, then you're looking at 4–16MB depending on the game. The SNES is pretty much the newest console whose games would entirely fit, I think.)
    - pezezin 3y ago
      The Ryzen 5800X3D has 96 MB of L3 cache, enough to fit the biggest NeoGeo or N64 games. The Zen 4-based Genoa-X will have up to 1 GB of L3 cache! You could comfortably fit Windows 95 there...
    - lproven 3y ago
      For a benchmark test for a British computer magazine in about 1996, I hand stripped down an installation of Windows 95 so that it could fit on a mid-1990s SSD: which is to say, a PCI slot device with 16 MB of DRAM which appeared to the computer as a disk drive. So, effectively, a 16 MB SSD. Not gigabytes, megabytes. So I can say with some authority that it is possible to have a complete running installation of Windows 95 in 16 megs of disk space; however I had to trim it pretty brutally to fit, so, for example, I think it had 2 fonts left, one of which was used for the widgets that display window-close boxes and so on. The problem was that the object of the exercise was the benchmark how much quicker it was — but there wasn't enough space left to install any applications, and our benchmark tool used real applications. So, although I did several days work and I did get paid for it, the effort was at heart in vain. But in principle yes you could run Windows 95 entirely out of 16 MB of cache memory, and if you had say 32 MB of cache memory it would be no problem at all.
andai 3y ago
What does hostile mean in this context?
- sweetjuly 3y ago
  It might be desirable to not trust RAM as other agents who have access to physical memory could be tampering with it.
- intelVISA 3y ago
  Any machine you may not have True Control over e.g. outside of JTAG range, in the cloud, runs proprietary software.

notacoward 3y ago

I got some exposure to this at SiCortex, where we had our own MIPS-based processors and so had to do many of these things in software. There was one ColdFire (embedded 68K) processor running μClinux per board, plus 27 of our own. This "Module Service Processor" would boot first, fetch a boot image from the one-per-system "System Service Processor" (pretty generic PC), load that via JTAG into each node's cache, then finally set each one loose to do things like memory registration and interconnect setup. This all set the stage for the actual Linux boot, which itself involved two stages with a switch_root in between. My very first assignment was to work on some of that MSP-to-node stuff, then later I had to dive into memory registration at least twice, even though both were pretty far from my real specialty. Small company, y'know. This kind of low-level work is significantly more complicated than even most kernel developers realize - hence the need for articles like OP. Ditto for anything on large (more than single-board) systems. The intersection of the two was, frankly, a bit exhausting. Just keeping track of all the moving parts and their respective states induced a cognitive load that made debugging other already-hard problems that much more difficult. My hat's off for anyone who has kept on doing that stuff longer than I did, or who has to do it in an environment where vendors are keeping so many secrets.

lgg 3y ago
This stuff is certainly pretty rarified these days. I remember when the PPC970 came out people were shocked how difficult it was to bootstrap. IBM didn't really care as POWER4 (from which it was derived) was not a merchant chip and they had management processors (and very high margins) in all their machines to handle it. Apple was the launch partner and even back then had a lot of in house expertise doing this sort of work. Everyone else who tried to use it was in for some real pain and most of them gave up. The guys doing the eval boards with support from IBM literally posted this: https://web.archive.org/web/20060715134515/http://www.970eval.com/970FAQ.html#anchor#4 https://web.archive.org/web/20060715134515/http://www.970eva... TL;DR, the last line is "Once all of the above is completed, the processor will be able to successfully fetch instructions from a boot source. You are now effectively at the same point you would have been 5 months ago, had this been a standard 750 bringup... Board bringup from this point should be very straightforward and follow established methods."