4 ms·
Would you recommend the 64 over 128? What kind of models will 128 open up over 64? Can you get most of them out of the 64?
by cced 3y ago
Would you recommend the 64 over 128? What kind of models will 128 open up over 64? Can you get most of them out of the 64?
- tomduncalf 3y agoI’m not sure, not really an expert in this field, just casually interested as an outsider. When I bought my Mac, 64 was as high as you could go.
- SparkyMcUnicorn 3y agoFor LLM models, TheBloke has provided memory requirements for all of their quantisations. For Apple Silicon you want to look at the GGUF models. Here's Mixtral-8x7B-Instruct: https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF#provided-files https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-G... 64GB of memory will struggle on 65-70B (or bigger) models, and you'll be limited to running 70B on Q3 or Q4 if you want to use it somewhat comfortably.