4 ms·

CXL Is Dead in the AI Era

22 points by wmf 3y ago

esafak 3y ago
I'm a seasoned machine learning engineer who has never heard of CXL, so it's worth spelling it out: https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link
- rajnathani 3y ago
  You need to be a chip designer to have necessarily heard of it. For bandwidth, most ML engineers really just care about the bandwidth between nodes in their cluster (eg: Mellanox, or whichever networking solution).
- SideburnsOfDoom 3y ago
  And people take note, if you write an essay that uses a TLA 3 times in the first paragraph, it's worth expanding the Three Letter Acronym on the first of those in your essay.
- throwup238 3y ago
  You’re not the only one who hasn’t heard of it. Usually any time I see a tech related acronym on HN I at least have a vague idea of what it’s related to based on incidental exposure but not this time. I searched HN for the acronym and there are two posts with more than a few comments, with the most popular one posted five months ago. “Promised as the messiah” my behind.
  - wmf 3y ago
    HN doesn't really deal with hardware much so it's not surprising that it hasn't been discussed. In the server world CXL has been a big topic.
gpapilion 3y ago
This is a feature large csps we’re looking for to improve their unit economics. There are a lot of issues that the author points out, but first and foremost it’s not a clear TCO win.
jauntywundrkind 3y ago
Hoping this is shortsighted. It's an interesting & valuable comparison: low bandwidth low latency PCIe serdes vs high band high latency Ethernet-like serdes. But I'm not sure I'm on board with the half-conclusion we get from this half-paywalled analysis. AI is seemingly insatiable sure & there's a relentless push to higher bandwidth, yes. NVLink seems to be kicking ass & PCIe is super struggling to keep any kind of pace absolutely, but it still seems wild to me to write off CXL at such an early stage CXL has some really real things going for it. It's lower level than PCIe and reuses the same serdes that many systems already have, so it's a natural low cost extension of what we already have, for little intrinsic cost. There's 512GBps of PCIe on a modern Epyc (128 lanes of PCIe 5.0); the proposal of CXL is that simplifying the protocol can let us do more with those links easily. GPUs could use more efficienct protocols to talk to the cpu, with better coherency, that might let cpu and main memory work much better in tandem; not unified memory but no longer so very far apart memories. Large memory or storage pools that nodes can attach to on demand offers some interesting capabilities to disaggregate storage, which mirrors the huge success we've seen with modern disaggregated databases being able to do successfully scale. CXL 3.1 also introduces host to host capabilities. This makes CXL a potential competitor to Ethernet at some levels of scale. Ethernet is getting around to 1.6Tbps/200GBps but via enormous nics and switches. For sure the CXL switches will also be huge but there's no nic. Imagine a Epyc with PCIe 6.0 speed (8Gbps/1TBps) or 7.0 (2TBps) that doesn't need a nic to talk to fabric, that has much much lower latency than Ethernet, that is an rdma beast. It's insufficient as a full speed AI fabric that can satisfy AI needs, but its a huge speed bump, built right into the core. It's hard for me to reject our ability to find good uses for this in the future, just because it's now how AI works today. The article very briefly mentions Infinity Fabric XGMI, which is what AMD is floating as what they'll use to compete with NVLink. The article talks about this like it's the future, like it's what will crush CXL. But the same PCIe switch chips that run Infinity Fabric XGMI will also run CXL. Maybe perhaps CXL goes nowhere, IF-XGMI rises, but I tend to think there'll be more sympathetic growth & development among these standards. https://www.servethehome.com/next-gen-broadcom-pcie-switches-to-support-amd-infinity-fabric-xgmi-to-counter-nvidia-nvlink/ https://www.servethehome.com/next-gen-broadcom-pcie-switches...
- stefanha 3y ago
  What is the killer app for cache coherency? Current PCIe devices work without cache coherency to host memory. Drivers follow a handful of patterns for communicating with the device via PCI BARs and DMA. Cache coherency is not required. If we're not talking about devices but disaggregated memory, then an application running on a single host doesn't need cache coherency on the fabric because the host already ensures cache coherency between the CPUs. (Is that correct?) The only case I see where cache coherency might be interesting is for multiple nodes running applications on disaggregated memory. Kind of like a single system image cluster, but it's not a software architecture that has been successful in recent years. This is something that has puzzled me about CXL. Would be great to understand what use case cache coherency serves. Thanks!
anonymousDan 3y ago
Whatever about CXL being unsuitable for AI workloads, it's unclear why memory disaggregation wouldn't benefit from it. I don't have access to the second half of the article, can anyone give the gist of the argument?

22 points by wmf 3y ago

I'm a seasoned machine learning engineer who has never heard of CXL, so it's worth spelling it out: https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link

rajnathani 3y ago
You need to be a chip designer to have necessarily heard of it. For bandwidth, most ML engineers really just care about the bandwidth between nodes in their cluster (eg: Mellanox, or whichever networking solution).
SideburnsOfDoom 3y ago
And people take note, if you write an essay that uses a TLA 3 times in the first paragraph, it's worth expanding the Three Letter Acronym on the first of those in your essay.
throwup238 3y ago
You’re not the only one who hasn’t heard of it. Usually any time I see a tech related acronym on HN I at least have a vague idea of what it’s related to based on incidental exposure but not this time. I searched HN for the acronym and there are two posts with more than a few comments, with the most popular one posted five months ago. “Promised as the messiah” my behind.
- wmf 3y ago
  HN doesn't really deal with hardware much so it's not surprising that it hasn't been discussed. In the server world CXL has been a big topic.

gpapilion 3y ago

This is a feature large csps we’re looking for to improve their unit economics. There are a lot of issues that the author points out, but first and foremost it’s not a clear TCO win.

jauntywundrkind 3y ago

Hoping this is shortsighted. It's an interesting & valuable comparison: low bandwidth low latency PCIe serdes vs high band high latency Ethernet-like serdes. But I'm not sure I'm on board with the half-conclusion we get from this half-paywalled analysis. AI is seemingly insatiable sure & there's a relentless push to higher bandwidth, yes. NVLink seems to be kicking ass & PCIe is super struggling to keep any kind of pace absolutely, but it still seems wild to me to write off CXL at such an early stage CXL has some really real things going for it. It's lower level than PCIe and reuses the same serdes that many systems already have, so it's a natural low cost extension of what we already have, for little intrinsic cost. There's 512GBps of PCIe on a modern Epyc (128 lanes of PCIe 5.0); the proposal of CXL is that simplifying the protocol can let us do more with those links easily. GPUs could use more efficienct protocols to talk to the cpu, with better coherency, that might let cpu and main memory work much better in tandem; not unified memory but no longer so very far apart memories. Large memory or storage pools that nodes can attach to on demand offers some interesting capabilities to disaggregate storage, which mirrors the huge success we've seen with modern disaggregated databases being able to do successfully scale. CXL 3.1 also introduces host to host capabilities. This makes CXL a potential competitor to Ethernet at some levels of scale. Ethernet is getting around to 1.6Tbps/200GBps but via enormous nics and switches. For sure the CXL switches will also be huge but there's no nic. Imagine a Epyc with PCIe 6.0 speed (8Gbps/1TBps) or 7.0 (2TBps) that doesn't need a nic to talk to fabric, that has much much lower latency than Ethernet, that is an rdma beast. It's insufficient as a full speed AI fabric that can satisfy AI needs, but its a huge speed bump, built right into the core. It's hard for me to reject our ability to find good uses for this in the future, just because it's now how AI works today. The article very briefly mentions Infinity Fabric XGMI, which is what AMD is floating as what they'll use to compete with NVLink. The article talks about this like it's the future, like it's what will crush CXL. But the same PCIe switch chips that run Infinity Fabric XGMI will also run CXL. Maybe perhaps CXL goes nowhere, IF-XGMI rises, but I tend to think there'll be more sympathetic growth & development among these standards. https://www.servethehome.com/next-gen-broadcom-pcie-switches-to-support-amd-infinity-fabric-xgmi-to-counter-nvidia-nvlink/ https://www.servethehome.com/next-gen-broadcom-pcie-switches...

stefanha 3y ago
What is the killer app for cache coherency? Current PCIe devices work without cache coherency to host memory. Drivers follow a handful of patterns for communicating with the device via PCI BARs and DMA. Cache coherency is not required. If we're not talking about devices but disaggregated memory, then an application running on a single host doesn't need cache coherency on the fabric because the host already ensures cache coherency between the CPUs. (Is that correct?) The only case I see where cache coherency might be interesting is for multiple nodes running applications on disaggregated memory. Kind of like a single system image cluster, but it's not a software architecture that has been successful in recent years. This is something that has puzzled me about CXL. Would be great to understand what use case cache coherency serves. Thanks!

anonymousDan 3y ago

Whatever about CXL being unsuitable for AI workloads, it's unclear why memory disaggregation wouldn't benefit from it. I don't have access to the second half of the article, can anyone give the gist of the argument?