7 ms·

I think there are a lot of misconceptions here in other sub comments. For starters, the DRAM manufacturers may possibly be the biggest penny pinchers in the se

I think there are a lot of misconceptions here in other sub comments. For starters, the DRAM manufacturers may possibly be the biggest penny pinchers in the semiconductor business. They're too cheap to use differential signaling even on GDDR memory. I'm not saying that PCIe RAM is never going to happen, but as somebody who works with DRAM controllers and DRAM vendors, I can not fathom a world where DRAM manufacturers put a PCIe controller on each chip, especially one capable of Gen 6.0 speeds! The extra die area, complexity, and bringup testing necessary for a PCIe controller is insane Second of all, the latency would really destroy system performance. The DDR protocol is designed to minimize latency (well kind of, there are still speed targets that need to be met.) The whole point of using a parallel bus with unscrambled data is to eliminate encoding/decoding latency. PCIe uses a Serializer/Deserializer (SerDes) that's scrambled on one end and then de-scrambled on the other end using a LFSR. It also goes through an adaptive tap filter. After that, the ordered sets from each lane are assembled and sent through a FIFO for clock domain crossing/timing recovery. All of those steps add latency to the system, and that's before we get to the Data Link Layer and Transaction layer. While I'm not well versed in CXL, I know that it shares the same Physical layer (a SerDes Phy.) Now you might imagine a chip with one PCIe end point that fans out to an array of DRAM chips on the back end, and you don't care that much about latency. But then that makes PCIe RC and EP in the system redundant because you could put the DRAM controller where you would otherwise put the PCIe controller. Now it might make packaging easier and allow for a cheaper version of DDR to be implemented, but is there a net savings? and does it justify the latency hit? Maybe. I'm not saying it's never going to happen, but I wouldn't bet on that tech.

was_a_dev 4y ago
Is it possible one manufacturer "takes the leap" and provides ram modules that are faster than competition. Therefore driving the transition purely on competition alone?
- SSLy 4y ago
  No, because they're in an unspoken, totally not illegal oligopoly not to dump prices.
- Lramseyer 4y ago
  Doubtful. PCIe Gen 5.0 x16 over DDR5 is a huge cost increase, a huge latency increase, and only a minor bandwidth increase (about double after accounting for protocol overhead.) At that point, you might as well widen your memory bus and use 2 DRAMs. The memory controller doesn't get much more complicated because both chips share a CA bus, and the DRAMs are cheaper. I don't work for a DRAM vendor, but I would imagine that the front end logic in the DRAM (the DDR controller inside the DRAM) uses way less than half the die area that a PCIe Gen 5.0 x16 EP would use.
pbalcer 4y ago
There's a significant momentum behind CXL [1]. Samsung already announced a CXL DRAM module [2]. From where I sit, it's all but inevitable that this technology will get broader adoption, at least with large CSPs. Its benefits are just too good to pass up. The extra latency is estimated to be no worse than a NUMA hop [3]. Many companies are already figuring out how to best deal with the more complex memory topologies that CXL will enable [4, 5]. But I'm biased (I work for Intel), so take what I say with a grain of salt. [1] - https://www.computeexpresslink.org/members https://www.computeexpresslink.org/members [2] - https://news.samsung.com/global/samsung-electronics-introduces-industrys-first-512gb-cxl-memory-module https://news.samsung.com/global/samsung-electronics-introduc... [3] - https://www.lenovoxperience.com/newsDetail/283yi044hzgcdv7snkrmmx9ovpq6aesmy9u9k7ai2648j7or https://www.lenovoxperience.com/newsDetail/283yi044hzgcdv7sn... [4] - https://arxiv.org/pdf/2203.00241.pdf https://arxiv.org/pdf/2203.00241.pdf [5] - https://arxiv.org/pdf/2206.02878.pdf https://arxiv.org/pdf/2206.02878.pdf
- Lramseyer 4y ago
  Well I'll be... Though the renderings indicate that this is a CXL controller connected to DRAMs on the back end. It makes sense, since the front end logic for CXL is die area intensive. This is basically an SSD without all of the complexity of wear leveling. > The extra latency is estimated to be no worse than a NUMA hop I was almost going to mention that in my previous comment and point out that this might be a thing in enterprise applications where SRAM is plentiful, but not likely to outright replace DDR in most consumer computing devices as the parent comment seemed to suggest.
  - pbalcer 4y ago
    > This is basically an SSD without all of the complexity of wear leveling. With all the benefits of an enterprise SSD (EDSFF in most cases) form factor. And that's what data center folks want. > not likely to outright replace DDR in most consumer computing devices Right. I expect that CXL will initially be deployed as an internal transparent optimization for cloud providers and large data centers, with the latency and complexity (to an extent) hidden at hypervisor/orchestration layers. Some sophisticated software might have special support. Eventually we might get S3-like disaggregated memory services for cloud instances enabled by pooling (pure speculation on my part).
formerly_proven 4y ago
> They're too cheap to use differential signaling even on GDDR memory. Makes it quite impressive that they're able to push 20+ Gigabit/s/pin even in dual-sided / T-layouts (though that also uses PAM4). Of course, comparing historical GPU layouts it's pretty obvious how the RAM chips are creeping ever more closely to the GPU die - weird how that is.
tails4e 4y ago
The DRAM signaling is very strange, I worked briefly on a hardware DDR controller and the timing reference was over pins called dqs. These pins sent a signal to represent the bit transition so they could be safely captured by the receiver. All good, except the pins were bidirectional so when transmitting you sent a reference on the same pins, so lost the reference from the other side. The issue then was when you just switch from trasmitting to receiving, you have to quickly re-lock to the sending reference, which is a pain in the ass, and really complicates things. DDR phys could be simplified if dqs was unique per TX and RX directions, but nope, penny pinching on the signal count.
- rowanG077 4y ago
  I don't think that is big of an issue. I have designed a memory controller and since the SDRAM protocol is really unambiguous which party owns the dqs lanes at what point I never had a problem with this. I'm curious what problems you encountered.
  - tails4e 4y ago
    The main issue is not who owned it, but the fact a pll (or similar) was needed to create a local clock locked to dqs, but dqs from the sender would dissappear for an unknown number of cycles depending on the rx vs TX schedule. So that local pll could drift and would need to be re locked once dqs from the sender was presented again
throwawaylinux 4y ago
There is a long tradition of serial memory interfaces. RDRAM/XDR DRAM, FB-DIMM/AMB, Intel SMB, OMI DIMM/DDIMM, etc. They seem to have most success in high end server CPUs where memory capacity and throughput are important. They have cost, power, latency disadvantages, so their forays into low end servers, PCs, etc has been more sporadic.
cptnapalm 4y ago
Forgive me if this is an ignorant question, but couldn't you use PCIe RAM like cache in the same way a CPU's cache levels work? It's not as fast, but could feed the regular RAM faster than from another source.

was_a_dev 4y ago

Is it possible one manufacturer "takes the leap" and provides ram modules that are faster than competition. Therefore driving the transition purely on competition alone?

SSLy 4y ago
No, because they're in an unspoken, totally not illegal oligopoly not to dump prices.
Lramseyer 4y ago
Doubtful. PCIe Gen 5.0 x16 over DDR5 is a huge cost increase, a huge latency increase, and only a minor bandwidth increase (about double after accounting for protocol overhead.) At that point, you might as well widen your memory bus and use 2 DRAMs. The memory controller doesn't get much more complicated because both chips share a CA bus, and the DRAMs are cheaper. I don't work for a DRAM vendor, but I would imagine that the front end logic in the DRAM (the DDR controller inside the DRAM) uses way less than half the die area that a PCIe Gen 5.0 x16 EP would use.

pbalcer 4y ago

There's a significant momentum behind CXL [1]. Samsung already announced a CXL DRAM module [2]. From where I sit, it's all but inevitable that this technology will get broader adoption, at least with large CSPs. Its benefits are just too good to pass up. The extra latency is estimated to be no worse than a NUMA hop [3]. Many companies are already figuring out how to best deal with the more complex memory topologies that CXL will enable [4, 5]. But I'm biased (I work for Intel), so take what I say with a grain of salt. [1] - https://www.computeexpresslink.org/members https://www.computeexpresslink.org/members [2] - https://news.samsung.com/global/samsung-electronics-introduces-industrys-first-512gb-cxl-memory-module https://news.samsung.com/global/samsung-electronics-introduc... [3] - https://www.lenovoxperience.com/newsDetail/283yi044hzgcdv7snkrmmx9ovpq6aesmy9u9k7ai2648j7or https://www.lenovoxperience.com/newsDetail/283yi044hzgcdv7sn... [4] - https://arxiv.org/pdf/2203.00241.pdf https://arxiv.org/pdf/2203.00241.pdf [5] - https://arxiv.org/pdf/2206.02878.pdf https://arxiv.org/pdf/2206.02878.pdf

Lramseyer 4y ago
Well I'll be... Though the renderings indicate that this is a CXL controller connected to DRAMs on the back end. It makes sense, since the front end logic for CXL is die area intensive. This is basically an SSD without all of the complexity of wear leveling. > The extra latency is estimated to be no worse than a NUMA hop I was almost going to mention that in my previous comment and point out that this might be a thing in enterprise applications where SRAM is plentiful, but not likely to outright replace DDR in most consumer computing devices as the parent comment seemed to suggest.
- pbalcer 4y ago
  > This is basically an SSD without all of the complexity of wear leveling. With all the benefits of an enterprise SSD (EDSFF in most cases) form factor. And that's what data center folks want. > not likely to outright replace DDR in most consumer computing devices Right. I expect that CXL will initially be deployed as an internal transparent optimization for cloud providers and large data centers, with the latency and complexity (to an extent) hidden at hypervisor/orchestration layers. Some sophisticated software might have special support. Eventually we might get S3-like disaggregated memory services for cloud instances enabled by pooling (pure speculation on my part).

formerly_proven 4y ago

> They're too cheap to use differential signaling even on GDDR memory. Makes it quite impressive that they're able to push 20+ Gigabit/s/pin even in dual-sided / T-layouts (though that also uses PAM4). Of course, comparing historical GPU layouts it's pretty obvious how the RAM chips are creeping ever more closely to the GPU die - weird how that is.

tails4e 4y ago

The DRAM signaling is very strange, I worked briefly on a hardware DDR controller and the timing reference was over pins called dqs. These pins sent a signal to represent the bit transition so they could be safely captured by the receiver. All good, except the pins were bidirectional so when transmitting you sent a reference on the same pins, so lost the reference from the other side. The issue then was when you just switch from trasmitting to receiving, you have to quickly re-lock to the sending reference, which is a pain in the ass, and really complicates things. DDR phys could be simplified if dqs was unique per TX and RX directions, but nope, penny pinching on the signal count.

rowanG077 4y ago
I don't think that is big of an issue. I have designed a memory controller and since the SDRAM protocol is really unambiguous which party owns the dqs lanes at what point I never had a problem with this. I'm curious what problems you encountered.
- tails4e 4y ago
  The main issue is not who owned it, but the fact a pll (or similar) was needed to create a local clock locked to dqs, but dqs from the sender would dissappear for an unknown number of cycles depending on the rx vs TX schedule. So that local pll could drift and would need to be re locked once dqs from the sender was presented again

throwawaylinux 4y ago

There is a long tradition of serial memory interfaces. RDRAM/XDR DRAM, FB-DIMM/AMB, Intel SMB, OMI DIMM/DDIMM, etc. They seem to have most success in high end server CPUs where memory capacity and throughput are important. They have cost, power, latency disadvantages, so their forays into low end servers, PCs, etc has been more sporadic.

cptnapalm 4y ago

Forgive me if this is an ignorant question, but couldn't you use PCIe RAM like cache in the same way a CPU's cache levels work? It's not as fast, but could feed the regular RAM faster than from another source.