Yes, NVIDIA GPUs can indeed read directly from main memory. It's called Heterogeneous Memory Management. https://developer.nvidia.com/blog/simplifying-gpu-appl

Yes, NVIDIA GPUs can indeed read directly from main memory. It's called Heterogeneous Memory Management. https://developer.nvidia.com/blog/simplifying-gpu-application-development-with-heterogeneous-memory-management/ https://developer.nvidia.com/blog/simplifying-gpu-applicatio... This would allow you to mmap() the weights file into CPU memory, and pass that CPU pointer to the GPU and allow it to fault the data in on-demand. The GPU will have less bandwidth accessing this memory than if you copied it onto the GPU though, as it will need to take a trip over PCIe to read it. But it would eliminate the need to manually upload data to the GPU and the synchronisation involved with that.