8 ms·

PCIe vs. CXL for Memory and Storage

82 points by teleforce 3y ago

jeffreygoesto 3y ago
Afaik, CXL 3.0 will run on top of UCIe as well. Let's hope for an open standard that will enable SoCs to be composed from many compatible chiplets. Cache coherency and small granularity below page size is a key ingredient.
superjan 3y ago
Again an article trying to sell a product/technology without bothering to explain upfront what it is, or what the acronym stands for. I can gather bits and pieces from how CXL compares to existing tech, but why keep me guessing?
- loeg 3y ago
  It's happily unambiguous: https://www.google.com/search?q=cxl https://www.google.com/search?q=cxl
- photonbeam 3y ago
  They’re selling to an audience that already knows what it is.
iamtedd 3y ago
How old is this article? The "new types of persistent memory like Intel’s Optane Technology" is dead, as of July 2022.
- jleahy 3y ago
  Well it says "Limited adoption so far; CXL expected to accelerate through 2022 and 2023" (which didn't happen), so presumably 2021.
- loeg 3y ago
  First copy in the Wayback Machine is Nov 2022: https://web.archive.org/web/20221130194031/https://www.synopsys.com/designware-ip/technical-bulletin/cxl2-3-storage-memory-applications.html https://web.archive.org/web/20221130194031/https://www.synop...
ksec 3y ago
Well for one thing CXL 3.0 is built on top of PCI-Express 6.0. I also think it is pretty much targeting HPC or Server use case. So it may not ever arriving at consumer PC or smartphone. And in case anyone is wondering, PCIe 7.0 is estimate to complete in 2025 and possibly arriving on market in 2027. At that point we are looking at SSD with 4x pci-e lane with up to 64GB/s.
- dist-epoch 3y ago
  CXL seems to be good for attaching large number of memory modules. Doesn't seem like a match for consumer PC/smartphone, which typically have 1-2 memory sticks, or at most 4.
- dvas 3y ago
  I think the real world latency numbers will be interesting to see with PCIe 6.0/7.0 since adopting PAM4 modulation allows the greater bandwidth. [0] Why Did PCIe 6.0 Adopt PAM4? There Are Many Reasons. https://blog.samtec.com/post/why-did-pcie-6-0-adopt-pam4-there-are-many-reasons/ https://blog.samtec.com/post/why-did-pcie-6-0-adopt-pam4-the...
- doctorpangloss 3y ago
  Manufacturers could have never anticipated that they could charge $1,500 for an accelerator on the one hand, and $15,000 for the same accelerator on the other. They’re the same thing! CXL doesn’t really make sense. Does even NVLink? And yet, we have no more 2 slot GPUs, we lost NVLink on 4090s. There are no AM5 motherboards with 4 2 spaced PCIe 16x slots. Why are the consumer chips even limited for IO? It’s just a bunch of bullshit to segregate a market and extract more rents. We’re going backwards. That said end users are also to blame, for every 1 person building a PC to do something innovative there are 19 who buy stuff with the brightest RGBs. So it’s up to the market really. Individual consumers might spend an amortized $12k/y on their cars. It’s not outside the realm of reason that hardware manufacturers can take way more of the surplus they’ve been giving to end users. Whether you think this is good or bad is a subjective thing. I personally don’t think much innovation comes out of NVIDIA, they have somehow tricked the world that thousands of very simple processors, tiled neatly, is as sophisticated as a thing as a general purpose CPU, or that CUDA is especially innovative on its own and not riding on a huge amount of inertia. You should transfer all the surplus to the people taking the most risks, basically the people writing the software with CUDA, which once was rendering for CG and physics simulation and now is people trying to make ML models more efficient.
dvas 3y ago
CXL and its coherency mechanisms will be interesting to watch as the requirements for LLMs and related applications requiring large memory pools continue to grow. This includes some HPC related workloads also. One of the use cases I have seen recently was driving down the total cost of DRAM in a larger-scale deployment of systems at AWS, Azure, Meta etc. Pond [1] which is a memory pooling system, claims to achieve the desired performance and at the same time lowering costs as one example. I think a look at the overall, bigger picture is important. For example considering how a system will be combined together with multiple GPU's, memory systems and other accelerators to meet the demands of applications, consider interconnects like NVLink [3] too. For those interested, I have left a previous comment about experimenting with CXL on a local setup [0] [0] https://news.ycombinator.com/item?id=37944691#37948761 https://news.ycombinator.com/item?id=37944691#37948761 [1] Pond: CXL-Based Memory Pooling Systems for Cloud Platforms https://arxiv.org/abs/2203.00241 https://arxiv.org/abs/2203.00241 [2] Intel Reveals the "What" and "Why" of CXL Interconnect, its Answer to NVLink https://www.techpowerup.com/254462/intel-reveals-the-what-and-why-of-cxl-interconnect-its-answer-to-nvlink https://www.techpowerup.com/254462/intel-reveals-the-what-an... [3] NVLink and NVSwitch; The building blocks of advanced multi-GPU communication—within and between servers. https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/
peter_d_sherman 3y ago
When I originally (many moons ago!) read about CXL -- I conceptualized it solely as a means to attach additional memory -- to a single CPU... Now that I read this article, I now also conceptualize CXL -- as a bus, that is, as a potential future PCIe bus replacement... That is, assuming that PCIe is patent-encumbered, and CXL is not -- and I don't know the current legal/patent statuses of either... Anyway, a very interesting article! There's definitely more tiers in the memory/storage (latency) hierarchy these days, and potentially room for even more... Also... >"With CXL 2.0 and CXL 3.0 which include switching, a host can access memory from one or more devices that form a pool. It’s important to note that in this kind of pooled configuration, only the resources themselves and not the contents of the memory are shared among the hosts: each region of memory can only belong to a single coherency domain. Memory sharing, which has been added to the CXL 3.0 specification, actually allows individual memory regions within pooled resources to be shared between multiple hosts." Historically, this reminds me of some of the ideas present in the the Cray-1: https://en.wikipedia.org/wiki/Cray-1 https://en.wikipedia.org/wiki/Cray-1 I'm guessing (but not knowing!) that in the future, "homebrew" supercomputers using CXL to link multiple CPUs (and GPUs/TPUs/NPUs...) to multiple nearby banks of gargantuan shared memory (HAL 9000: "I'm sorry Dave, I can't let you do that..." <g>) -- will be able to be created by people with less-than-Supercomputing-Center-budgets -- but all of that is highly speculative for the time being!

82 points by teleforce 3y ago

Afaik, CXL 3.0 will run on top of UCIe as well. Let's hope for an open standard that will enable SoCs to be composed from many compatible chiplets. Cache coherency and small granularity below page size is a key ingredient.

superjan 3y ago

Again an article trying to sell a product/technology without bothering to explain upfront what it is, or what the acronym stands for. I can gather bits and pieces from how CXL compares to existing tech, but why keep me guessing?

loeg 3y ago
It's happily unambiguous: https://www.google.com/search?q=cxl https://www.google.com/search?q=cxl
photonbeam 3y ago
They’re selling to an audience that already knows what it is.

iamtedd 3y ago

How old is this article? The "new types of persistent memory like Intel’s Optane Technology" is dead, as of July 2022.

jleahy 3y ago
Well it says "Limited adoption so far; CXL expected to accelerate through 2022 and 2023" (which didn't happen), so presumably 2021.
loeg 3y ago
First copy in the Wayback Machine is Nov 2022: https://web.archive.org/web/20221130194031/https://www.synopsys.com/designware-ip/technical-bulletin/cxl2-3-storage-memory-applications.html https://web.archive.org/web/20221130194031/https://www.synop...

ksec 3y ago

Well for one thing CXL 3.0 is built on top of PCI-Express 6.0. I also think it is pretty much targeting HPC or Server use case. So it may not ever arriving at consumer PC or smartphone. And in case anyone is wondering, PCIe 7.0 is estimate to complete in 2025 and possibly arriving on market in 2027. At that point we are looking at SSD with 4x pci-e lane with up to 64GB/s.

dist-epoch 3y ago
CXL seems to be good for attaching large number of memory modules. Doesn't seem like a match for consumer PC/smartphone, which typically have 1-2 memory sticks, or at most 4.
dvas 3y ago
I think the real world latency numbers will be interesting to see with PCIe 6.0/7.0 since adopting PAM4 modulation allows the greater bandwidth. [0] Why Did PCIe 6.0 Adopt PAM4? There Are Many Reasons. https://blog.samtec.com/post/why-did-pcie-6-0-adopt-pam4-there-are-many-reasons/ https://blog.samtec.com/post/why-did-pcie-6-0-adopt-pam4-the...
doctorpangloss 3y ago
Manufacturers could have never anticipated that they could charge $1,500 for an accelerator on the one hand, and $15,000 for the same accelerator on the other. They’re the same thing! CXL doesn’t really make sense. Does even NVLink? And yet, we have no more 2 slot GPUs, we lost NVLink on 4090s. There are no AM5 motherboards with 4 2 spaced PCIe 16x slots. Why are the consumer chips even limited for IO? It’s just a bunch of bullshit to segregate a market and extract more rents. We’re going backwards. That said end users are also to blame, for every 1 person building a PC to do something innovative there are 19 who buy stuff with the brightest RGBs. So it’s up to the market really. Individual consumers might spend an amortized $12k/y on their cars. It’s not outside the realm of reason that hardware manufacturers can take way more of the surplus they’ve been giving to end users. Whether you think this is good or bad is a subjective thing. I personally don’t think much innovation comes out of NVIDIA, they have somehow tricked the world that thousands of very simple processors, tiled neatly, is as sophisticated as a thing as a general purpose CPU, or that CUDA is especially innovative on its own and not riding on a huge amount of inertia. You should transfer all the surplus to the people taking the most risks, basically the people writing the software with CUDA, which once was rendering for CG and physics simulation and now is people trying to make ML models more efficient.

dvas 3y ago

CXL and its coherency mechanisms will be interesting to watch as the requirements for LLMs and related applications requiring large memory pools continue to grow. This includes some HPC related workloads also. One of the use cases I have seen recently was driving down the total cost of DRAM in a larger-scale deployment of systems at AWS, Azure, Meta etc. Pond [1] which is a memory pooling system, claims to achieve the desired performance and at the same time lowering costs as one example. I think a look at the overall, bigger picture is important. For example considering how a system will be combined together with multiple GPU's, memory systems and other accelerators to meet the demands of applications, consider interconnects like NVLink [3] too. For those interested, I have left a previous comment about experimenting with CXL on a local setup [0] [0] https://news.ycombinator.com/item?id=37944691#37948761 https://news.ycombinator.com/item?id=37944691#37948761 [1] Pond: CXL-Based Memory Pooling Systems for Cloud Platforms https://arxiv.org/abs/2203.00241 https://arxiv.org/abs/2203.00241 [2] Intel Reveals the "What" and "Why" of CXL Interconnect, its Answer to NVLink https://www.techpowerup.com/254462/intel-reveals-the-what-and-why-of-cxl-interconnect-its-answer-to-nvlink https://www.techpowerup.com/254462/intel-reveals-the-what-an... [3] NVLink and NVSwitch; The building blocks of advanced multi-GPU communication—within and between servers. https://www.nvidia.com/en-us/data-center/nvlink/ https://www.nvidia.com/en-us/data-center/nvlink/

peter_d_sherman 3y ago

When I originally (many moons ago!) read about CXL -- I conceptualized it solely as a means to attach additional memory -- to a single CPU... Now that I read this article, I now also conceptualize CXL -- as a bus, that is, as a potential future PCIe bus replacement... That is, assuming that PCIe is patent-encumbered, and CXL is not -- and I don't know the current legal/patent statuses of either... Anyway, a very interesting article! There's definitely more tiers in the memory/storage (latency) hierarchy these days, and potentially room for even more... Also... >"With CXL 2.0 and CXL 3.0 which include switching, a host can access memory from one or more devices that form a pool. It’s important to note that in this kind of pooled configuration, only the resources themselves and not the contents of the memory are shared among the hosts: each region of memory can only belong to a single coherency domain. Memory sharing, which has been added to the CXL 3.0 specification, actually allows individual memory regions within pooled resources to be shared between multiple hosts." Historically, this reminds me of some of the ideas present in the the Cray-1: https://en.wikipedia.org/wiki/Cray-1 https://en.wikipedia.org/wiki/Cray-1 I'm guessing (but not knowing!) that in the future, "homebrew" supercomputers using CXL to link multiple CPUs (and GPUs/TPUs/NPUs...) to multiple nearby banks of gargantuan shared memory (HAL 9000: "I'm sorry Dave, I can't let you do that..." <g>) -- will be able to be created by people with less-than-Supercomputing-Center-budgets -- but all of that is highly speculative for the time being!