4 ms·

Better to run the Q8 model on an epyc pair with 768GB, you'll get the same performance

• ltbarcly3 2y ago
  The Q8 model is totally different?
  • manmal 2y ago
    My experience with quantizations is that anything below 6 is noticeably worse. Coherence suffers. I’ve rarely gotten anything really useful out of a Q4 model, code wise. For transformations they are great though, eg convert JSON to Markdown and vice versa.
    • yieldcrv 2y ago
      I like Q5 The sweet spot for me
    • ltbarcly3 2y ago
      No I mean the quantized versions of this model in particular have less parameters as well. They are almost different models.