3 ms·
In the era of huge ML models, wouldn't it make more sense to do the opposite, i.e. use main RAM or SSD as swap voor VRAM? So that we can run e.g. ChatGPT on con
by FooBarWidget 4y ago
In the era of huge ML models, wouldn't it make more sense to do the opposite, i.e. use main RAM or SSD as swap voor VRAM? So that we can run e.g. ChatGPT on consumer hardware.
- rbanffy 4y agoA lot of low-end GPUs do that and, when you don't have enough memory to fit the working dataset in the GPU's very fast RAM, the CPU will end up swapping in and out of the GPU via the PCIe bus.
- brucethemoose2 4y agoThis is easier said than done, though certain frameworks (like Microsoft's Deepspeed) try to do this "automatically"