3 ms·
It's a lot closer these days with 30B 4bit quantized GPTQ fitting in 1 RTX 3090. Goes from 20 tokens per second to 15 tokens per second nearing the ~3k token c
by stevenhuang 3y ago
It's a lot closer these days with 30B 4bit quantized GPTQ fitting in 1 RTX 3090.
Goes from 20 tokens per second to 15 tokens per second nearing the ~3k token context length, with similar quality in output to chatgpt 3.5.