3 ms·

The Q8 model is totally different?

• manmal 2y ago
  My experience with quantizations is that anything below 6 is noticeably worse. Coherence suffers. I’ve rarely gotten anything really useful out of a Q4 model, code wise. For transformations they are great though, eg convert JSON to Markdown and vice versa.
  • yieldcrv 2y ago
    I like Q5 The sweet spot for me
  • ltbarcly3 2y ago
    No I mean the quantized versions of this model in particular have less parameters as well. They are almost different models.