3 ms·
> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. No… that’s not such a great thing. Helpful in a pinch, but if you’
by coder543 3y ago
> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM.
No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model on the GPU is not useful.
> Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.
I agree with this, if someone has a desktop that can fit two GPUs.
- deleted 3y ago[deleted]
- sp332 3y agoThe main benefit of a GPU in that case is much faster prompt reading. Could be useful for Code Llama cases where you want the model to read a lot of code and then write a line or part of a line.
- zten 3y agoMulti-GPU in desktop chassis gets crazy pretty quickly. If you don't care about aesthetics and can figure out both the power delivery and PCI-E lane situation, https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... has an example that will make a Bitcoin mining rig look clean. Water cooling can get you down to 2x slot height, with all of the trouble involved in water cooling. NVIDIA really segmented the market quite well. Gamers hate blower cards, but they are the right physical dimensions to make multi-GPU work well, and they are exclusively on the workstation cards.