3 ms·
The TheBloke Q4 is running on my 24GB 3090 with double context length (8192 tokens) even.
by nulld3v 3y ago
The TheBloke Q4 is running on my 24GB 3090 with double context length (8192 tokens) even.
- Havoc 3y agoContext length goes into sys ram not vram from what I can tell. Yep that was a conservative call on my part. I've got a 3090 and a 2070 (8gb) so can push Q5 if needed but I find with a billion browser tabs and some chat history it tends to bomb so Q4 it is.