4 ms·
Yes. I am definitely using both GPUs, I can run the quant 4 65b models entirely in VRAM (they use about 40GB). If I push everything into VRAM - I get 12.2 tok
by horsawlarway 3y ago
Yes. I am definitely using both GPUs, I can run the quant 4 65b models entirely in VRAM (they use about 40GB).
If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b.
If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU.
Ex - here's my nvidia-smi output while 65b is running
https://imgur.com/a/JnaieKg https://imgur.com/a/JnaieKg