5 ms·

Do you have a ballpark idea of how much RAM would be necessary to run llama 3.1 8b and 70b on 8-quant?

• karolist 2y ago
  Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.
  • ekianjo 2y ago
    You mean 8, not 3?
    • karolist 2y ago
      Yes, apologies, can't edit now