2 ms·
Varies by quantitation but Q4 needs about 30 gigs
by Havoc 3y ago
Varies by quantitation but Q4 needs about 30 gigs
- nulld3v 3y agoThe TheBloke Q4 is running on my 24GB 3090 with double context length (8192 tokens) even.
- Havoc 3y agoContext length goes into sys ram not vram from what I can tell. Yep that was a conservative call on my part. I've got a 3090 and a 2070 (8gb) so can push Q5 if needed but I find with a billion browser tabs and some chat history it tends to bomb so Q4 it is.