3 ms·

Would anyone be willing to compress this down to maybe 14b-20b for us on peasant 16gb rigs?

• orangepanda 2y ago
  Or maybe even middle class plebeian 24gb rigs?
  • griomnib 2y ago
    At that point just run 8b.
    • pulse7 2y ago
      Or wait for the IQ2_M quantization of 70b which you can run very fast on 24GB VRAM with context size of 4096...
      • griomnib 2y ago
        At some point there’s so much degradation with quantizing I think 8b is going to be better for many tasks.