4 ms·
Yeah, quite good actually, especially if you quantize it on text close to what you are trying to output. Llama 70B is a huge compromise at 2.65bpw... This does
by brucethemoose2 3y ago
Yeah, quite good actually, especially if you quantize it on text close to what you are trying to output.
Llama 70B is a huge compromise at 2.65bpw... This does make the much "dumber." Yi 34B is much better, as you can quantize it at ~4bpw and still have a huge context.
- lossolo 3y agoHow would you compare mistral-7b-instruct 16fp (or similar 7b/13b model like llama2 etc) to Yi-34b quantized?
- brucethemoose2 3y ago34B is better. Quantization hurts some, especially in "pro" non chat use cases like RAG, but the increased parameter count makes models so much smarter in comparison. The perplexity graph here is a pretty good illustration: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 YMMV, as Mistral and Yi are not necessarily comparable like different sizes of llama, and it depends on the task.