3 ms·
The 13b and 30b run quite well on a 4090 at 4-bit quantization.
by 0xbadc0de5 4y ago
The 13b and 30b run quite well on a 4090 at 4-bit quantization.
- thewataccount 4y agoAh dang I missed that I was still using the 8bit mode, I'll look into that thanks!