3 ms·
I get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that. So nowhere close to the 4090, but plenty fast anyway.
by MacsHeadroom 4y ago
I get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that.
So nowhere close to the 4090, but plenty fast anyway.