3 ms·
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole dam
by dheera 3y ago
Stupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD?
Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"
- coolspot 3y agoThat’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow. For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
- anonymousDan 3y agoIs it memory/disk bandwidth bound or latency bound?
- mike_hearn 3y agoMemory bandwidth bound.
- sebzim4500 3y agoI haven't run the numbers, but I would expect that doing that would make it slower than just running on the CPU.