7 ms·
That is significantly more bandwidth than a DDR5 memory bus. Is there something about PCIe that makes it infeasible to use as a memory bus? Otherwise that just
by Veserv 3y ago
That is significantly more bandwidth than a DDR5 memory bus. Is there something about PCIe that makes it infeasible to use as a memory bus? Otherwise that just seems like a ton of free memory bandwidth just being left on the table.
- O5vYtytb 3y agoIsn't that what CXL is?
- crest 3y agoCXL supports cache-coherent memories and even the numbers claimed in marketing materials are ~5x when it comes than DDR5 access times measured on desktop CPUs in microbenchmarks. It's still a lot better than having software handle the coherency protocol and could be really useful to allow either very large ccNUMA systems or memory capacity and bandwidth hungry accelerators to access the host DRAM as long they can tolerate the latency e.g. something along the lines of a several SX-Aurora cards hooked into a big host with a few TiB of DRAM spread over ≥16 DDR5 or later channels.
- redleader55 3y agoPCIe is a pretty complicated protocol based on messages. Using this for RAM would mean the CPU and everything will be required to speak this protocol. Using PCIe for RAM will also have impact on latency, which is undesired. Speed is not always the only thing.
- Quarrel 3y ago> Speed is not always the only thing. or perhaps more accurately, bandwidth is not the only goal.
- alexgartrell 3y agoOr perhaps more accurately, throughput is not the only goal.
- deleted 3y ago[deleted]
- cogman10 3y agoThroughput, latency, cost. Pick 2.
- deleted 3y ago[deleted]
- tyler569 3y agoWithout being a domain expert, my intuition would be that PCIe is optimized for throughput over latency and there's probably a throughput compromise when you want low-latency access.
- smallmancontrov 3y agoYeah but DDR is starting to move to more complex modulations, CXL is bringing down PCIe latency, and PCIe is starting from a position of elevated competence when compared to other standards. For example, you might expect that PCIe obtains parallelism by sending different packets down different lanes but in fact it shreds packets across lanes specifically because of latency. When PCIe eats another standard, the average quality of the standards ecosystem generally goes up. That said, memory latency is so important that even small sacrifices should be heavily scrutinized.
- Retr0id 3y agoI'm speculating here but I think the answer has more to do with latency than bandwidth.
- practicemaths 3y agoWhat's the latency comparison between the memory bus and PCIe?
- malfist 3y agoI dug into this for a hn comment months ago, but I think it's 2-3 orders of magnitude difference in latency. RAM is measured in nanoseconds, PCIe is measured in microseconds
- pclmulqdq 3y agoThe difference is smaller than that if you optimize it. The memory bus is about 50 ns in latency, and PCIe you can get down to sub 500 ns.
- malfist 3y agoI don't think you've checked those numbers. SSD access is in the order of 10-20 microseconds (10,000 - 20,000 ns) and memory bus access is ~10-15 nanoseconds. Here's the comment I made a couple months ago when I looked up the numbers: I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible in 4 clock cycles, in 4.8 ghz cpu like the i7-10700, L1 cache latency is sub 1ns.
- pclmulqdq 3y agoI have absolutely checked those numbers, and I have written PCIe hardware cores and drivers before, as well as microbenchmarking CPUs pretty extensively. I think you're mixing up a few things: CAS latency and total access latency of DRAM are not the same, and SSDs and generic PCIe devices are not the same. Most of SSD latency is in the SSD's firmware and accesses to the backing flash memory, not in the PCIe protocol itself - hence why the Intel Optane SSDs were super fast. Many NICs will advertise sub-microsecond round-trip time for example, and those are PCIe devices. Most of DRAM access latency (and a decent chunk of access latency to low-latency PCIe devices) comes from the CPU's cache coherency network, queueing in the DRAM controllers, and opening of new rows. If you're thinking only of CAS latency, you are actually missing the vast majority of the latency involved in DRAM operations - it's the best-case scenario - you will only get the CAS latency if you are hitting an open row on an idle bus with a bank that is ready to accept an access.
- avar 3y agoWouldn't PCI as a "memory bus" just be an SSD connected to the PCI bus setup as a swap space, which you can already do?
- nolist_policy 3y agoNo, NVMe SSDs are not memory mapped. While they max out at 14GB/s - giving you 56GB/s per PCIe 5.0 x16 slot - emulating memory/swapping is expensive.
- p1esk 3y agoLatency
- qwertox 3y agoIt would be nicer to see the ability to extend GPUs with off-the-shelf RAM, think SO-DIMM clipped to the back of the GPU. Then being able to choose to buy 2 x 64 GB DIMMs or just a 8GB one, maybe also have some GBs of good RAM soldered onto the GPU like it is now. One can dream.
- temp0826 3y agoWhy does this make me picture a bunch of GPU company execs in big leather chairs cackling around a conference table with a dead goat in the middle?
- briffle 3y agoBut then the vendors would not be able to differentiate (ie, markup) their products. They currently charge thousands more for adding $75 of memory.
- crote 3y agoWasn't this a thing in the past - like the 1990s? I know the Matrox G200 supported it, but there might've been others too.
- beauzero 3y agoI wonder if this is to help support a newish arch. like the one Microsoft and Intel are proposing (Taipei conf.) with each laptop/pc having a dedicated NPU, CPU, GPU to bring some of the Copilot button back on PC? Everything I see today I inadvertently read in "oh this will help me as a hobbyist". It generally doesn't. Probably just technology doing what it does and moving forward.
- anonymousDan 3y agoI've seen some very recent research proposing just that: https://arxiv.org/pdf/2305.05033.pdf https://arxiv.org/pdf/2305.05033.pdf . Of course still far from production, but interesting that in some cases it actually also improves latency.
- crest 3y agoEven the fastest (as in lowest latency) CXL memories claim a ~200ns latency (~4x worse than DDR5) and it wouldn't be free.
- imtringued 3y ago200ns latency for communicating with another CPU or GPU device is pretty good. Just imagine how many microseconds it would take to do all of this over ethernet.
- nxobject 3y agoOr, we could make it a general purpose unified system bus... call it the Unibus, or something like that.
- numpad0 3y agoSerDes overhead and high pin count? PCIe is not a real "bus" architecture but parallelized hub architecture system. I don't know enough to explain why but these leads to higher power consumption than bus at scale. M.2 SSDs with 3.0 x4 links already had issues with controllers thermal throttling often explained as heat from on-chip serializer/deserializer.