4 ms·

CXL is useful for many more things beyond just memory expansion (which is a "type 3 device"). The purpose of CXL is to allow for memory coherency between diffe

CXL is useful for many more things beyond just memory expansion (which is a "type 3 device"). The purpose of CXL is to allow for memory coherency between different CXL devices. To quote the spec on type 2 devices: > CXL Type 2 devices, in addition to fully coherent cache, also have memory, for example DDR, High-Bandwidth Memory (HBM), etc., attached to the device. These devices execute against memory, but their performance comes from having massive bandwidth between the accelerator and device-attached memory. The main goal for CXL is to provide a means for the Host to push operands into device-attached memory and for the Host to pull results out of device-attached memory such that it does not add software and hardware cost that offsets the benefit of the accelerator. There's some cool things you can do with CXL, like resurrecting the whole persistent memory idea with low-latency flash, making hardware offload devices more capable since you now get free cache coherency, and a whole bunch of other stuff. But yes, it's really not for consumer use-cases. The applications I've seen colleagues work on are mostly enterprise stuff like cool RDMA integrations, cache-coherent flash, and more I can't talk about here.

jnwatson 3y ago
CXL for CPU access to DRAM is overkill. Plenty of server mobos allow you to directly attach 2 TiB of DRAM.
- lazide 3y ago
  Some 4 and supposedly 6TB now too.
  - justinlloyd 3y ago
    Last one I put physical hands on was running 24TB, and was expandable to 32TB.
- vinay_ys 3y ago
  Rack level disaggregated compute-memory-storage-accelerator architectures allow for dynamic partitioning/aggregation of hardware to suite concurrent workloads and their evolution over time and make it easier to achieve both cost efficiency as well as incremental, continuous and non-disruptive hardware upgrades.
  - paulmd 3y ago
    Wendell Level1techs has been talking about his Liqid PCIe Fabric stuff lately, sounds pretty sick, this mention inspired me to go search for his other videos on the topic https://www.youtube.com/watch?v=x1NWdoXmVsE https://www.youtube.com/watch?v=x1NWdoXmVsE https://www.youtube.com/watch?v=Iwep9D_q4e0 https://www.youtube.com/watch?v=Iwep9D_q4e0 https://www.youtube.com/watch?v=zXlWeVrsk14 https://www.youtube.com/watch?v=zXlWeVrsk14
gavinray 3y ago
CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me. Persistent Memory databases are such a neat idea, it's a shame that the hardware for it isn't commonplace.
- e63f67dd-065b 3y ago
  Yeah optane seemed like it had so much potential -- with the right abstractions, it neatly dealt with the eternal problem of the separation between RAM and disk. Much of DB and storage system engineering comes down to decisions around when to persist what, and optane was very promising in allowing for new architectures that had much better performance at lower complexity. Alas, optane's dead now. I do know people actively working on resurrecting a lot of pmem work on low-latency flash, however, and it seems like this is one area with a low of momentum behind it.
  - paulmd 3y ago
    I'm utterly unqualified to discuss using it in practice but it always seemed like PMEM would have required a fundamental idiom shift away from the DRAM concept. If you treat it like memory it's always going to be slow memory. If you treat it like storage, it can be very fast storage. If that makes sense. Software that "treats storage like memory" would end up looking a lot like javacard imo. Or something like Samsung's in-memory key-value database stuff. But it wouldn't really look like a linux kernel allocating memory inside an all-pmem partition. Difference, in my head, being that essentially the filesystem is an unnecessary layer/abstraction in the middle. You want something that either looks like a garbage-collected runtime, or an in-memory database that vacuums, or LISP collections of object-trees, etc. There is no point to having a separation between memory and disk anymore, that is the point - "resource allocation is disk persistence" so to speak. It would have required a big-bang rewrite/second-system that is just not possible with the dominance of the existing RAM/storage dichotomy. Or at least a couple killer apps from large vendors/etc that really outperformed what was possible with RAM.
    - dvas 3y ago
helf 3y ago
[dead]
simcop2387 3y ago
I've got to imagine that at some point in the future when it's been common on server stuff for a while it'll end up as part of a next-gen "thunderbolt" style connection, simply because the silicon by then will be a commodity and it'll be cheaper to use the same thing everywhere. I'm imagining a docking station that doubles the ram, and gives it a powerful GPU/NPU/TPU for a workstation. Since the ram would be close to the accelerator over CXL they can talk directly and make full bandwidth use of it even if the laptop uses the remote ram as a "very very very fast swap" or something else to prioritize things. I'm not so sure it'd be useful for persistent storage at that level but it'd make for some really cool options for a portable workstation since you wouldn't need everything inside the device.
- jauntywundrkind 3y ago
  CXL 3.0 is where it gets interesting, where you can start to have switched fabric where many hosts can talk to many devices. Having that pool of ram & GPUs be on demand usable by whomever has some attractive possibilities. Also, one can just imagine having some pools of data in CXL memory that a host can just attach to & read, which seems like a cool possibility. It does kind of upset me a bit that CXL 3.0 still seems purely host-to-switch-to-device oriented. If you have your formerly PCIe slots on your cores speaking CXL and doing directory memory over fabric, I'd really really love to be able to talk to other hosts. Maybe that happens & is possible in 3.0, but it feels like CXL isnt paving that cowpath, isn't making is obvious, and that there will be a bunch of proprietary nasty ways to bridge computers & chat over CXL that are all non-standard, & I wish CXL had been more direct about making themselves & their upcoming switched fabric viable & interesting for host-to-host.
  - simcop2387 3y ago
    I think a large part of that is probably because the needs for doing that kind of host to host thing aren't well understood yet for the general cases. Right now I think that's being fleshed out in the market using the whole DPU model still. I think sadly you're going to be right about how it turns out for a while and it won't be until something like CXL 6.0 or whatever comes after it that starts to really address that. I'm wondering when we'll see the first CXL DPU style device that uses some of the fancy new 800Gbps or even 1.6Tbps networking stuff that's being developed right now. That'd be enough bandwidth to put things on the other end of a data-center with very little other than the latency penalty.

jnwatson 3y ago

CXL for CPU access to DRAM is overkill. Plenty of server mobos allow you to directly attach 2 TiB of DRAM.

lazide 3y ago
Some 4 and supposedly 6TB now too.
- justinlloyd 3y ago
  Last one I put physical hands on was running 24TB, and was expandable to 32TB.
vinay_ys 3y ago
Rack level disaggregated compute-memory-storage-accelerator architectures allow for dynamic partitioning/aggregation of hardware to suite concurrent workloads and their evolution over time and make it easier to achieve both cost efficiency as well as incremental, continuous and non-disruptive hardware upgrades.
- paulmd 3y ago
  Wendell Level1techs has been talking about his Liqid PCIe Fabric stuff lately, sounds pretty sick, this mention inspired me to go search for his other videos on the topic https://www.youtube.com/watch?v=x1NWdoXmVsE https://www.youtube.com/watch?v=x1NWdoXmVsE https://www.youtube.com/watch?v=Iwep9D_q4e0 https://www.youtube.com/watch?v=Iwep9D_q4e0 https://www.youtube.com/watch?v=zXlWeVrsk14 https://www.youtube.com/watch?v=zXlWeVrsk14

gavinray 3y ago

CXL for Persistent Memory, as the Optane successor (or rather, sole-survivor if you will) is what excites me. Persistent Memory databases are such a neat idea, it's a shame that the hardware for it isn't commonplace.

e63f67dd-065b 3y ago
Yeah optane seemed like it had so much potential -- with the right abstractions, it neatly dealt with the eternal problem of the separation between RAM and disk. Much of DB and storage system engineering comes down to decisions around when to persist what, and optane was very promising in allowing for new architectures that had much better performance at lower complexity. Alas, optane's dead now. I do know people actively working on resurrecting a lot of pmem work on low-latency flash, however, and it seems like this is one area with a low of momentum behind it.
- paulmd 3y ago
  I'm utterly unqualified to discuss using it in practice but it always seemed like PMEM would have required a fundamental idiom shift away from the DRAM concept. If you treat it like memory it's always going to be slow memory. If you treat it like storage, it can be very fast storage. If that makes sense. Software that "treats storage like memory" would end up looking a lot like javacard imo. Or something like Samsung's in-memory key-value database stuff. But it wouldn't really look like a linux kernel allocating memory inside an all-pmem partition. Difference, in my head, being that essentially the filesystem is an unnecessary layer/abstraction in the middle. You want something that either looks like a garbage-collected runtime, or an in-memory database that vacuums, or LISP collections of object-trees, etc. There is no point to having a separation between memory and disk anymore, that is the point - "resource allocation is disk persistence" so to speak. It would have required a big-bang rewrite/second-system that is just not possible with the dominance of the existing RAM/storage dichotomy. Or at least a couple killer apps from large vendors/etc that really outperformed what was possible with RAM.
  - dvas 3y ago

helf 3y ago

[dead]

simcop2387 3y ago

I've got to imagine that at some point in the future when it's been common on server stuff for a while it'll end up as part of a next-gen "thunderbolt" style connection, simply because the silicon by then will be a commodity and it'll be cheaper to use the same thing everywhere. I'm imagining a docking station that doubles the ram, and gives it a powerful GPU/NPU/TPU for a workstation. Since the ram would be close to the accelerator over CXL they can talk directly and make full bandwidth use of it even if the laptop uses the remote ram as a "very very very fast swap" or something else to prioritize things. I'm not so sure it'd be useful for persistent storage at that level but it'd make for some really cool options for a portable workstation since you wouldn't need everything inside the device.

jauntywundrkind 3y ago
CXL 3.0 is where it gets interesting, where you can start to have switched fabric where many hosts can talk to many devices. Having that pool of ram & GPUs be on demand usable by whomever has some attractive possibilities. Also, one can just imagine having some pools of data in CXL memory that a host can just attach to & read, which seems like a cool possibility. It does kind of upset me a bit that CXL 3.0 still seems purely host-to-switch-to-device oriented. If you have your formerly PCIe slots on your cores speaking CXL and doing directory memory over fabric, I'd really really love to be able to talk to other hosts. Maybe that happens & is possible in 3.0, but it feels like CXL isnt paving that cowpath, isn't making is obvious, and that there will be a bunch of proprietary nasty ways to bridge computers & chat over CXL that are all non-standard, & I wish CXL had been more direct about making themselves & their upcoming switched fabric viable & interesting for host-to-host.
- simcop2387 3y ago
  I think a large part of that is probably because the needs for doing that kind of host to host thing aren't well understood yet for the general cases. Right now I think that's being fleshed out in the market using the whole DPU model still. I think sadly you're going to be right about how it turns out for a while and it won't be until something like CXL 6.0 or whatever comes after it that starts to really address that. I'm wondering when we'll see the first CXL DPU style device that uses some of the fancy new 800Gbps or even 1.6Tbps networking stuff that's being developed right now. That'd be enough bandwidth to put things on the other end of a data-center with very little other than the latency penalty.