3 ms·
This is faster than running it on an RTX 4090 I think.
by lxe 4y ago
This is faster than running it on an RTX 4090 I think.
- deleted 4y ago[deleted]
- jokethrowaway 4y agoNope a 4090 can do the 30b-4bit model at 20 tokens/s
- MacsHeadroom 4y agoI get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that. So nowhere close to the 4090, but plenty fast anyway.