3 ms·

Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.

• polygamous_bat 3y ago
  "Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.
  • throeaaysn 3y ago
    [dead]