4 ms·
Maybe a stupid question, I'm not familiar with how video ports work, but does the concept of memory swapping exist for video cards? Or is that physically imposs
by CodeCompost 3y ago
Maybe a stupid question, I'm not familiar with how video ports work, but does the concept of memory swapping exist for video cards? Or is that physically impossible?
Obviously that would be useless for games, but for LLMs it may be an option?
- rwmj 3y agoIf you need to use all the memory then swapping breaks down, best you can do is try to divide the work so half can be done, then you completely swap all the memory and do the other half. Also PCIe is (relatively) slow for moving gigabytes of data back and forth.
- rahimnathwani 3y agoSwapping would be too slow. Llama.cpp allows you to specify that n layers load into the GPU's VRAM, and the rest load into main memory.
- brucethemoose2 3y agoSome frameworks have tried to do this automatically, but its extremely slow. You are better off just running whatever won't fit on the CPU (which is what llama.cpp does).