6 ms·

In the future even your RAM will have firmware; and the subject of POWER10 blobs

62 points by hlandau 6y ago

KirillPanov 6y ago
Everything old is new again. They tried this with FB-DIMMs in the early 2000s and it didn't go well: https://en.wikipedia.org/wiki/Fully_Buffered_DIMM https://en.wikipedia.org/wiki/Fully_Buffered_DIMM That SERDES chip unavoidably increases latency, roughly doubling it. This isn't likely to catch on as the only kind of DRAM in the system; latency is too important. Maybe we'll see successful systems with two levels of DRAM: small low-latency parallel DRAM and a large heap of higher-latency serial DRAMs. That's plausible.
- sitkack 6y ago
  Main memory will move to an HBM, "regular" ram beyond the first 32,64 ... GB will be DDRX, then it will be pmem (Optane, etc). We will just keep getting more layers in the hierarchy.
  - wahern 6y ago
    Yep. Because memory access fundamentally scales in O(sqrt(N)) time, not O(1). http://www.ilikebigbits.com/2014_04_21_myth_of_ram_1.html http://www.ilikebigbits.com/2014_04_21_myth_of_ram_1.html
- PhantomGremlin 6y ago
  latency is too important Yeah, you would think so. But it really hasn't improved all that much over the years. (Of course, that's partially because it's hard to do!). Roughly speaking, when I designed with DRAM in 1983, the access time from RAS was 150 ns for the most popular parts. Now, with DDR4, you're talking CAS latency of about 10 ns, an improvement of 15x. In the same timeframe CPU clock rate increased from about 10 MHz to about 3000 MHz, an improvement of 300x. I grok your comment about perhaps having two levels of DRAM, but the whole memory hierarchy thing is already crazy. Most computers already have 3 levels of cache ahead of the DRAM.
  - justaguy88 6y ago
    Latency is very important, but cost is even more important. If there was cheaper lower latency DRAM they'd use it
  - datameta 6y ago
    With Storage Class Memory aka Persistent memory becoming a talked about thing in the industry they're not too far off base. It sits right between capacitive memory and nand flash; may end up being the DRAM's page file and the SSD's cache.
- ardy42 6y ago
  > Everything old is new again. They tried this with FB-DIMMs in the early 2000s and it didn't go well: IIRC, Rambus RDRAM was also serial, or at least less parallel than conventional DRAM.
londons_explore 6y ago
I'm surprised we don't have "variable latency DRAM" yet. Ie. DRAM which can either return data quickly, or raise a flag saying it needs more time. That would enable error detection circuitry to redirect some reads and writes around defects in the silicon without incurring a latency penalty for the majority of reads and writes.
- Hypx 6y ago
  We sorta have that already in the form of virtual memory. The next step might be SCM (Storage Class Memory) where we use very fast NVM in lieu of SSDs or HDDs. Higher latency RAM would just be another tier in the memory hierarchy if it catches on.
- wiml 6y ago
  It used to be pretty common for RAM or peripherals to have wait states, or REQ/ACK handshakes, or similar mechanisms that would allow the peripheral to request a variable amount of time to respond. I suspect it's not a more popular scheme because it makes it hard to pipeline your activity if there's unpredictable timing somewhere.
hakfoo 6y ago
In the era of "home computers", the serial connections were usually for low-speed peripherals like modems, and a you had a parallel bus for things like hard drives. This seemed to carry well into the modern PC/Mac era. Then in rapid succession starting in the early 2000s, we saw USB and SATA/SAS replace most IDE and SCSI use cases, and PCI Express replace PCI and AGP, along with Rambus taking a swing at conventional SDRAM. Was there some fundamental breakthrough that made it possible to do high-speed serial busses, or was it that the costs of managing them finally undercut the hassles of signaling difficulties on a parallel bus? The other thing I wonder about is what happened to SRAM? I know it always used a price concern-- "64K of DRAM is $10, and 64K of SRAM is $50", but given that DRAM is so cheap now, I'm surprised there's no SRAM product for enthusiasts/cost-no-object performance seekers. They'll already pay 5-10x the price of commodity DDR4 for the highest-binned overclockable stuff.
- freeone3000 6y ago
  Intel sells Optane, which is SRAM for the modern era. It's kind of tricky to get anything out of it (edit: besides using it as a really fast disk, which is what it's currently treated like) without bespoke software, as it's neither disk (byte-addressable!) nor RAM (persistent!). Linux kernel support is experimental and ongoing. It's coming, but it's not here yet. It's also $3000 per TB, triple the price of DDR4, and not much faster.
  - selimthegrim 6y ago
    https://www.tomshardware.com/news/intel-kills-off-all-optane-only-ssds-for-consumers-no-replacements-planned https://www.tomshardware.com/news/intel-kills-off-all-optane...
    - freeone3000 6y ago
      They're killing the "consumer" line, not the product entirely. That same announcement unveils the P5800X, which hooks up to a PCI-e x4 slot just the same. There are resellers available for single-unit quantities.
  - 6y ago
moonchild 6y ago
> contrary to the README above, the repository contains no source code It does! It's not in the mainline source tree, but is tagged. The most recent version appears to be https://github.com/open-power/ocmb-explorer-fw/tree/ea3fae3c696a1f035599b4a8ea82dfcbd278c6a6 https://github.com/open-power/ocmb-explorer-fw/tree/ea3fae3c...
mswann 6y ago
> If you are familiar with work done in the FOSS community to procure devices with fully open firmware, the name Synopsys may ring a bell here. Synopsys is a company which sells IP blocks[...]. One of their IP block products is a DDR4 memory controller and PHY solution, which appears popular and which is found in a number of SoCs. It is however a known fact that Synopsys's DDR4 PHY requires a Synopsys-issued firmware blob to operate; therefore, chips which use the Synopsys DDR4 PHY inevitably have firmware blobs in their boot path > Probably the most notable SoC containing a Synopsys DDR4 memory controller and PHY is the NXP i.MX8M series, which was for example selected by Purism to form the heart of their Librem 5 phone. Is this (still) true? I just tried to find any binary blobs in U-Boot source tree [0], but didn't find any. At least for some Rockchip SoCs (RK3399), which I assume also use Synopsys IP DDR PHYs, I know that it's not necessary to provide anything else apart from upstream U-Boot and Arm Trusted Firmware (ATF) [1]. And AFAICT current mainline U-Boot can initialize NXP i.MX8M's DDR PHY, too. [0] https://gitlab.denx.de/u-boot/u-boot https://gitlab.denx.de/u-boot/u-boot [1] https://stikonas.eu/wordpress/2019/09/15/blobless-boot-with-rockpro64/ https://stikonas.eu/wordpress/2019/09/15/blobless-boot-with-...
- mswann 6y ago
  After digging through some resources on the web I now believe that this is indeed still true - at least I couldn't find anything to the contrary (apparently you have to provide the firmware blob outside of U-Boot, which then loads it at runtime). This probably means the Rockchip SoCs I was talking about don't use Synopsys IP for the DDR PHY (although they tend to use a lot of Synopsys IP for all kinds of other components). I wonder how I could have missed that issue with the Librem 5 till now.
  - hlandau 6y ago
    Author here. This is correct. There are other DDR PHY IP providers such as Cadence, which did not elect to require firmware blobs in their design. You could say that the Synopsys DDR4 PHY design is a little overengineered. I've confirmed the presence of these blobs in i.MX8M BSPs myself. See also Purism's blogpost discussing this very subject.
- 6y ago
peter_d_sherman 6y ago
>"Industry trends: serial memory attachment. One of the most interesting things which will happen in the near future in computing is the adoption of serial memory interfaces. For pretty much the entire history of modern computing, RAM has been attached to a system via a high-speed parallel interface. Making parallel interfaces fast is hard and requires extremely rigorous control of timing skew between pins, therefore the routing of PCB traces between a CPU and RAM slots must be done with great precision. At the speeds of modern parallel RAM interfaces like DDR4, what is theoretically a digital interface in practice must be viewed as practically analog (to the point that part of a CPU DDR4 controller is called the “PHY”). Moreover, the maximum distance between a CPU and its RAM slots is extremely tight. The positioning of RAM slots on a motherboard is largely constrained by these physics considerations. For these reasons you have never seen anything like the flexibility with RAM attachment that you can get with, for example, PCIe or SAS. PCIe and SAS are serial architectures which support cabling and even switching, allowing entire additional chassis of PCIe and SAS devices to be attached to a system via cables. Parallel RAM attachment methods like DDR4 and DDR5, by comparison, are both inflexible and pushing the physics to the limit (it is unclear to me whether there will even be a DDR6). Due to the complexity of the analog concerns when running parallel interfaces at such high speeds, the size of a “PHY” IP block gets larger and larger with each successive iteration of DDR, taking up more room on a CPU's silicon die. For a CPU with eight memory channels, the amount of space taken up by DRAM controllers and PHYs is now substantial. For these and other reasons, the move to serial DRAM attachment is being considered by industry, most notably by IBM. The idea is that a multi-lane serial interface, not unlike e.g. PCIe would be used to attach DIMMs rather than a parallel interface." PDS: I've foreseen the coming of serial memory for a long time now. It just makes sense. What I would love to see is a whole ecosystem of open-source, open-hardware high-speed (RAM speed) serial bus, serial connection, serial protocol, etc., etc. It should be one-size fits all. Sort of like you have a CPU, and now you want to attach either perhipherals or RAM to it. Well, use the same high-speed open-source/open-hardware serial interface and protocol for all of them! It would be a thing of beauty, should this ecosystem exist in the future...
- pbalcer 6y ago
  Isn't that CXL [1]? Serial-attached memory is coming very soon. [1] - https://www.computeexpresslink.org/post/the-benefits-of-serial-attached-memory-with-compute-express-link https://www.computeexpresslink.org/post/the-benefits-of-seri...

62 points by hlandau 6y ago

Everything old is new again. They tried this with FB-DIMMs in the early 2000s and it didn't go well: https://en.wikipedia.org/wiki/Fully_Buffered_DIMM https://en.wikipedia.org/wiki/Fully_Buffered_DIMM That SERDES chip unavoidably increases latency, roughly doubling it. This isn't likely to catch on as the only kind of DRAM in the system; latency is too important. Maybe we'll see successful systems with two levels of DRAM: small low-latency parallel DRAM and a large heap of higher-latency serial DRAMs. That's plausible.

sitkack 6y ago
Main memory will move to an HBM, "regular" ram beyond the first 32,64 ... GB will be DDRX, then it will be pmem (Optane, etc). We will just keep getting more layers in the hierarchy.
- wahern 6y ago
  Yep. Because memory access fundamentally scales in O(sqrt(N)) time, not O(1). http://www.ilikebigbits.com/2014_04_21_myth_of_ram_1.html http://www.ilikebigbits.com/2014_04_21_myth_of_ram_1.html
PhantomGremlin 6y ago
latency is too important Yeah, you would think so. But it really hasn't improved all that much over the years. (Of course, that's partially because it's hard to do!). Roughly speaking, when I designed with DRAM in 1983, the access time from RAS was 150 ns for the most popular parts. Now, with DDR4, you're talking CAS latency of about 10 ns, an improvement of 15x. In the same timeframe CPU clock rate increased from about 10 MHz to about 3000 MHz, an improvement of 300x. I grok your comment about perhaps having two levels of DRAM, but the whole memory hierarchy thing is already crazy. Most computers already have 3 levels of cache ahead of the DRAM.
- justaguy88 6y ago
  Latency is very important, but cost is even more important. If there was cheaper lower latency DRAM they'd use it
- datameta 6y ago
  With Storage Class Memory aka Persistent memory becoming a talked about thing in the industry they're not too far off base. It sits right between capacitive memory and nand flash; may end up being the DRAM's page file and the SSD's cache.
ardy42 6y ago
> Everything old is new again. They tried this with FB-DIMMs in the early 2000s and it didn't go well: IIRC, Rambus RDRAM was also serial, or at least less parallel than conventional DRAM.

londons_explore 6y ago

I'm surprised we don't have "variable latency DRAM" yet. Ie. DRAM which can either return data quickly, or raise a flag saying it needs more time. That would enable error detection circuitry to redirect some reads and writes around defects in the silicon without incurring a latency penalty for the majority of reads and writes.

Hypx 6y ago
We sorta have that already in the form of virtual memory. The next step might be SCM (Storage Class Memory) where we use very fast NVM in lieu of SSDs or HDDs. Higher latency RAM would just be another tier in the memory hierarchy if it catches on.
wiml 6y ago
It used to be pretty common for RAM or peripherals to have wait states, or REQ/ACK handshakes, or similar mechanisms that would allow the peripheral to request a variable amount of time to respond. I suspect it's not a more popular scheme because it makes it hard to pipeline your activity if there's unpredictable timing somewhere.

hakfoo 6y ago

In the era of "home computers", the serial connections were usually for low-speed peripherals like modems, and a you had a parallel bus for things like hard drives. This seemed to carry well into the modern PC/Mac era. Then in rapid succession starting in the early 2000s, we saw USB and SATA/SAS replace most IDE and SCSI use cases, and PCI Express replace PCI and AGP, along with Rambus taking a swing at conventional SDRAM. Was there some fundamental breakthrough that made it possible to do high-speed serial busses, or was it that the costs of managing them finally undercut the hassles of signaling difficulties on a parallel bus? The other thing I wonder about is what happened to SRAM? I know it always used a price concern-- "64K of DRAM is $10, and 64K of SRAM is $50", but given that DRAM is so cheap now, I'm surprised there's no SRAM product for enthusiasts/cost-no-object performance seekers. They'll already pay 5-10x the price of commodity DDR4 for the highest-binned overclockable stuff.

freeone3000 6y ago
Intel sells Optane, which is SRAM for the modern era. It's kind of tricky to get anything out of it (edit: besides using it as a really fast disk, which is what it's currently treated like) without bespoke software, as it's neither disk (byte-addressable!) nor RAM (persistent!). Linux kernel support is experimental and ongoing. It's coming, but it's not here yet. It's also $3000 per TB, triple the price of DDR4, and not much faster.
- selimthegrim 6y ago
  https://www.tomshardware.com/news/intel-kills-off-all-optane-only-ssds-for-consumers-no-replacements-planned https://www.tomshardware.com/news/intel-kills-off-all-optane...
  - freeone3000 6y ago
    They're killing the "consumer" line, not the product entirely. That same announcement unveils the P5800X, which hooks up to a PCI-e x4 slot just the same. There are resellers available for single-unit quantities.
- 6y ago

moonchild 6y ago

> contrary to the README above, the repository contains no source code It does! It's not in the mainline source tree, but is tagged. The most recent version appears to be https://github.com/open-power/ocmb-explorer-fw/tree/ea3fae3c696a1f035599b4a8ea82dfcbd278c6a6 https://github.com/open-power/ocmb-explorer-fw/tree/ea3fae3c...

mswann 6y ago

> If you are familiar with work done in the FOSS community to procure devices with fully open firmware, the name Synopsys may ring a bell here. Synopsys is a company which sells IP blocks[...]. One of their IP block products is a DDR4 memory controller and PHY solution, which appears popular and which is found in a number of SoCs. It is however a known fact that Synopsys's DDR4 PHY requires a Synopsys-issued firmware blob to operate; therefore, chips which use the Synopsys DDR4 PHY inevitably have firmware blobs in their boot path > Probably the most notable SoC containing a Synopsys DDR4 memory controller and PHY is the NXP i.MX8M series, which was for example selected by Purism to form the heart of their Librem 5 phone. Is this (still) true? I just tried to find any binary blobs in U-Boot source tree [0], but didn't find any. At least for some Rockchip SoCs (RK3399), which I assume also use Synopsys IP DDR PHYs, I know that it's not necessary to provide anything else apart from upstream U-Boot and Arm Trusted Firmware (ATF) [1]. And AFAICT current mainline U-Boot can initialize NXP i.MX8M's DDR PHY, too. [0] https://gitlab.denx.de/u-boot/u-boot https://gitlab.denx.de/u-boot/u-boot [1] https://stikonas.eu/wordpress/2019/09/15/blobless-boot-with-rockpro64/ https://stikonas.eu/wordpress/2019/09/15/blobless-boot-with-...

mswann 6y ago
After digging through some resources on the web I now believe that this is indeed still true - at least I couldn't find anything to the contrary (apparently you have to provide the firmware blob outside of U-Boot, which then loads it at runtime). This probably means the Rockchip SoCs I was talking about don't use Synopsys IP for the DDR PHY (although they tend to use a lot of Synopsys IP for all kinds of other components). I wonder how I could have missed that issue with the Librem 5 till now.
- hlandau 6y ago
  Author here. This is correct. There are other DDR PHY IP providers such as Cadence, which did not elect to require firmware blobs in their design. You could say that the Synopsys DDR4 PHY design is a little overengineered. I've confirmed the presence of these blobs in i.MX8M BSPs myself. See also Purism's blogpost discussing this very subject.
6y ago

peter_d_sherman 6y ago

>"Industry trends: serial memory attachment. One of the most interesting things which will happen in the near future in computing is the adoption of serial memory interfaces. For pretty much the entire history of modern computing, RAM has been attached to a system via a high-speed parallel interface. Making parallel interfaces fast is hard and requires extremely rigorous control of timing skew between pins, therefore the routing of PCB traces between a CPU and RAM slots must be done with great precision. At the speeds of modern parallel RAM interfaces like DDR4, what is theoretically a digital interface in practice must be viewed as practically analog (to the point that part of a CPU DDR4 controller is called the “PHY”). Moreover, the maximum distance between a CPU and its RAM slots is extremely tight. The positioning of RAM slots on a motherboard is largely constrained by these physics considerations. For these reasons you have never seen anything like the flexibility with RAM attachment that you can get with, for example, PCIe or SAS. PCIe and SAS are serial architectures which support cabling and even switching, allowing entire additional chassis of PCIe and SAS devices to be attached to a system via cables. Parallel RAM attachment methods like DDR4 and DDR5, by comparison, are both inflexible and pushing the physics to the limit (it is unclear to me whether there will even be a DDR6). Due to the complexity of the analog concerns when running parallel interfaces at such high speeds, the size of a “PHY” IP block gets larger and larger with each successive iteration of DDR, taking up more room on a CPU's silicon die. For a CPU with eight memory channels, the amount of space taken up by DRAM controllers and PHYs is now substantial. For these and other reasons, the move to serial DRAM attachment is being considered by industry, most notably by IBM. The idea is that a multi-lane serial interface, not unlike e.g. PCIe would be used to attach DIMMs rather than a parallel interface." PDS: I've foreseen the coming of serial memory for a long time now. It just makes sense. What I would love to see is a whole ecosystem of open-source, open-hardware high-speed (RAM speed) serial bus, serial connection, serial protocol, etc., etc. It should be one-size fits all. Sort of like you have a CPU, and now you want to attach either perhipherals or RAM to it. Well, use the same high-speed open-source/open-hardware serial interface and protocol for all of them! It would be a thing of beauty, should this ecosystem exist in the future...

pbalcer 6y ago
Isn't that CXL [1]? Serial-attached memory is coming very soon. [1] - https://www.computeexpresslink.org/post/the-benefits-of-serial-attached-memory-with-compute-express-link https://www.computeexpresslink.org/post/the-benefits-of-seri...