> I am wondering if for example you could have a PC with 128GB RAM and allocate 80GB to the GPU and use that for AI. Kinda. Model tiling is pretty common from

> I am wondering if for example you could have a PC with 128GB RAM and allocate 80GB to the GPU and use that for AI. Kinda. Model tiling is pretty common from what I've seen, where you can keep the full model loaded in DRAM and swap it to the GPU as-needed. Even that isn't really necessary if you have a good SSD though, in my experience it's extraordinarily fast to just load from NVMe to DRAM. > Despite searching quite far I can't find a clear answer to this. I doubt many people have ran into it as an issue. If you have a unified memory system, your bottleneck is still the SSD or storage medium that the model lives on. Edit: also note that this is a feature of CUDA. Page streaming from the CPU to the GPU has been working for a while now: https://developer.nvidia.com/blog/maximizing-unified-memory-performance-cuda/ https://developer.nvidia.com/blog/maximizing-unified-memory-...