5 ms·

to clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit

• beacon294 3y ago
  It is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.
  • skavi 3y ago
    So your experience isn’t representative of the work presented by this post? Or does llama.cpp use the same technique for quantization?
    • beacon294 3y ago
      I don't know, hopefully it will help set general expectations.