3 ms·

love that, I’ve been using Mistral 7B on my M1 and I thought it was tolerable but turned out I wasnt utilizing Metal and now its amazing 8x7B nowadays As long

love that, I’ve been using Mistral 7B on my M1 and I thought it was tolerable but turned out I wasnt utilizing Metal and now its amazing 8x7B nowadays As long as metal is used on an iphone I could see it worked well too. I use quantize 5 on my laptop but quantize 4 seems very practical

zakki 3y ago
How do you use 8x7B?
- yieldcrv 3y ago
  running it with LM Studio. No command line and has a huggingface browser, very user friendly.
- localhost 3y ago
  Use ollama [1] $ ollama run mixtral [1] https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama
alberth 3y ago
What did you do to enable Metal?
- yieldcrv 3y ago
  I use LM Studio and 0.2.9 now has a checkmark for it
  - addandsubtract 3y ago
    How much RAM do you have? Ollama comes with a note: "this model requires 48GB of RAM".
moneywoes 3y ago
could you please eli5 how metal works
- yieldcrv 3y ago
  its just a co-processor included in Apple devices that has relevant components of a GPU, can accelerate graphics and general computation tasks Apple’s chip architecture has a variety of co-processors alongside the CPU that make it the most perfect for transformers, in a convenient laptop form factor with fanless heat dissipation and low energy footprint