4 ms·

Is there a Llama2 65b quantized version for Mac M2?

• eurekin 3y ago
  I'd look for llama2 thebloke 70b GGML on hugginface. Update, this might work: https://huggingface.co/TheBloke/Llama-2-70B-GGML https://huggingface.co/TheBloke/Llama-2-70B-GGML Example command mentions the 4bit variant: ./main -m llama-2-70b.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "Llamas are"
  • ionwake 3y ago
    I also found this link ( https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-apple-silicon-mac-c3f46143f327 https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-a...) but havent verified if it helps, but thanks for your reply I am taking a look now.
    • eurekin 3y ago
      Just checked on a m1 w 32GB ram. 7b and 13b work as a charm. 70b doesn't fit, even the 2 bit version
• deleted 3y ago
  [deleted]
• deleted 3y ago
  [deleted]