And both 3090 and 4090 can do 32 GB/s host-device. Not far from CPU-RAM. You only load the matrix once. The bandwidth for the matmul is orders of magnitude larg

And both 3090 and 4090 can do 32 GB/s host-device. Not far from CPU-RAM. You only load the matrix once. The bandwidth for the matmul is orders of magnitude larger and happens all in device and mostly in cache.