3 ms·

Llama-2-70b with 16-bit precision = 2 bytes * 70 billion = 140 GB of memory Assuming zero copy, it should be loadable on around 170GB. Reference: https://www.

• stevenhuang 3y ago
  No one runs inference at full FP16 precision. 6 bit quant gives ~90% of original quality at more than half memory savings.
• nullc 3y ago
  Sigh. My comment gives a measurement of the memory usage of the model in question, Falcon 180B, (in mostly-q8 form, which is how someone would likely use it if not q4). Perhaps it would have been more obvious that I actually ran the model and measured it if people hadn't flagged the self-reply with the model output.