4 ms·
It's crazy how good open weight models are already. With a 7900 XTX I can run a roughly GPT-3.5 quality q6 model at 70 TPS. FP16 at 30 TPS. I can swap out the m
by Zambyte 2y ago
It's crazy how good open weight models are already. With a 7900 XTX I can run a roughly GPT-3.5 quality q6 model at 70 TPS. FP16 at 30 TPS. I can swap out the model to cater to specific tasks.
Sure, it's nowhere near the amount of knowledge as Claude Sonnet / Opus, or the suite of GPT 4s, but when you hook it up to external sources of knowledge, it's still freaking useful.