34B is better. Quantization hurts some, especially in "pro" non chat use cases like RAG, but the increased parameter count makes models so much smarter in compa

34B is better. Quantization hurts some, especially in "pro" non chat use cases like RAG, but the increased parameter count makes models so much smarter in comparison. The perplexity graph here is a pretty good illustration: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 YMMV, as Mistral and Yi are not necessarily comparable like different sizes of llama, and it depends on the task.