4 ms·
Are you using the second GPU at all? My experience is only a few systems will share load across GPUs. I didn't bother with dual GPUs for that reason. 4-5 toke
by frognumber 3y ago
Are you using the second GPU at all?
My experience is only a few systems will share load across GPUs. I didn't bother with dual GPUs for that reason.
4-5 tokens per second is slower than my system. I'm getting in the teens. I'm a little surprised since yours is newer, faster, and has way more RAM.
- horsawlarway 3y agoYes. I am definitely using both GPUs, I can run the quant 4 65b models entirely in VRAM (they use about 40GB). If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b. If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU. Ex - here's my nvidia-smi output while 65b is running https://imgur.com/a/JnaieKg https://imgur.com/a/JnaieKg