This TB/sec of memory bandwidth is only when the data is on the GPU. The real problem is that GPU memory is expensive and models are getting bigger. What we wo

This TB/sec of memory bandwidth is only when the data is on the GPU. The real problem is that GPU memory is expensive and models are getting bigger. What we would like to do is what has usually been done with the cache hierarchy where you bring the data as needed to the GPU. As this would allow to only need a lot less memory, as we could stream parameters from RAM or disk as needed. But there is a bottleneck at the pcie interface ( 32GB/s for pcie 4.0x16 ), and you can't feed the GPU fast enough. Which means you can't leverage cheap memory directly.