3 ms·
> You would still be limited by the CPU memory bandwidth, and you will share it with it. The word I take issue with here is "still", in that today GPUs accessi
by mlyle 3y ago
> You would still be limited by the CPU memory bandwidth, and you will share it with it.
The word I take issue with here is "still", in that today GPUs accessing main memory are not limited by CPU memory bandwidth but instead by the PCIe link bandwidth.
I'm assuming that we will, in practice, still be limited by PCIe link bandwidth; but perhaps much less so.
- numpad0 3y agoThat problem is solved by CXL that allow PCIe RAM card on the bus, not faster RAM on CPU through bus
- mlyle 3y agoCXL is nice. It doesn't change my belief that accessing RAM over PCIe will not be a win for very long.
- imtringued 3y agoI don't know what you're on about. It is significantly faster to access memory directly via CXL than to use message passing with JSON or gRPC. Also CXL allows seamless pointer sharing between GPU and CPU in a fine grained way. These are obvious wins independent of memory bandwidth.
- mlyle 3y ago> It is significantly faster to access memory directly via CXL than to use message passing with JSON or gRPC. I'm not talking about supercomputing. We're in a thread talking about the feasibility of running larger models on "smaller GPUs". Do you think that GPUs access host memory using JSON or gRPC? CXL is nice because it allows coherent, low-latency access to memories. This is not going to move the needle a whole lot on running large models on GPUs, because they are more bandwidth sensitive than latency sensitive, and coherence is not a huge concern. I also think that RAM interfaces are going to scale much quicker than IO interfaces, so it's at best a temporary win over local GPU memory for inference.