3 ms·
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments. (20 tokens/second on a
by MacsHeadroom 4y ago
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments.
(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)
- lxe 4y agoHow are you getting 20 tokens/second? I'm getting 2.6 tokens/s on 3090 with int4 prequantized model. Is 4090 so much faster?