2 ms·
How much VRAM do you need to run this on GPU?
by davikr 3y ago
How much VRAM do you need to run this on GPU?
- api 3y agoAt 8-bit about 34G plus a little overhead. You can go as low as 4-bit with some loss of accuracy and halve that.
- Havoc 3y agoVaries by quantitation but Q4 needs about 30 gigs
- nulld3v 3y agoThe TheBloke Q4 is running on my 24GB 3090 with double context length (8192 tokens) even.
- Havoc 3y agoContext length goes into sys ram not vram from what I can tell. Yep that was a conservative call on my part. I've got a 3090 and a 2070 (8gb) so can push Q5 if needed but I find with a billion browser tabs and some chat history it tends to bomb so Q4 it is.
- tildef 3y agoLoading it with 4-bit quantization takes a total of 22G vram including the rest of Xorg for me (not really trying to eek out any extra megs here). As such, the 8 bit version is probably not for people on consumer GPUs. Inference speed on a 4090 is about the same as the web version of ChatGPT with GPT4. The generated output at 4 bit is good so far (no worse than llama2 at least), though I haven't really put it through its paces.