4 ms·

Maybe this is the right thread to ask. If you were in the market for a new Mac (eg MacBook Pro), would you go for the 100+GB RAM option for running LLMs locally

by manmal 3y ago

Maybe this is the right thread to ask. If you were in the market for a new Mac (eg MacBook Pro), would you go for the 100+GB RAM option for running LLMs locally? Or is the difference between heavily quantized models and their unquantized versions so small, and progress so fast, that it wouldn’t be worth it?

givinguflac 3y ago
All I know is I’ve been disappointed at the limitations of 32GB, lots of inaccessible models with that amount, but also some that are useable. I wish I’d gotten more.
coder543 3y ago
An unquantized Qwen1.5-110B model would require some ~220GB of RAM, so 100+GB would not be "enough" for that, unless we put a big emphasis on the "+". I consider "heavily" quantized to be anything below 4-bit quantization. At 4-bit, you could run a 110B model on around 55GB to 60GB of memory. Right now, Llama-3-70B-Instruct is the highest ranked model you can download[0], and you should be able to fit the 6-bit quantization into 64GB of RAM. Historically, 4-bit quantization represents very little quality loss compared to the full 16-bit models for LLMs, but I have heard rumors that Llama 3 might be so well trained that the quality loss starts to occur earlier, so 6-bit quantization seems like a safe bet for good quality. If you had 128GB of RAM, you still couldn't run the unquantized 70B model, but you could run the 8-bit quantization in a little over 70GB of RAM. Which could feel unsatisfying, since you would have so much unused RAM sitting around, and Apple charges a shocking amount of money for RAM. [0]: https://leaderboard.lmsys.org/ https://leaderboard.lmsys.org/
- wongarsu 3y ago
  However if you want to use the LLM in your workflow instead of just experimenting with it on its own you also need RAM to run everything else comfortably. 96GB RAM might be a good compromise for now. 64GB is cutting it close, 128GB leaves more breathing room but is expensive.
  - coder543 3y ago
    Yep, I agree with that.
- andai 3y ago
  Phi 3 Q4 spazzes out on some inputs (emits a stream of garbage), while the FP16 version doesn't (at least for the cases I could find). Maybe they just botched the quantization (I have good results with other Q4 models), but it is an interesting data point.
  - coder543 3y ago
    Phi 3 in particular had some issues with the end-of-text token not being handled correctly at launch, as I recall, but I could be remembering incorrectly.
zitterbewegung 3y ago
This isn’t the right thread for this but you can look at the model and the difference in parameters. A MacBook Pro will be cheap but have slow inference whereas the using cards will be more expensive , faster for inference (usage) and refining. If you have the money go for the MacBook Pro since it seems that networks under 100B that are trained for a very long time have improved performance and the amount of parameters you need would fit in the MacBook Pro
jprd 3y ago
If running LLMs is your intention, a Mac is probably that absolute worse way to try achieve that. Soldered RAM, no real upgrade path - M2/M3 is cool, but not for this.
jmorgan 3y ago
I think it's worth it, although it might be best to wait for the next iteration: there's rumors the M4 Macs will support up to 512GB of memory [1]. The current 128GB (e.g. M3 Max) and 192GB (e.g. M2 Ultra) Macs run these large models. For example on the M2 Ultra, the Qwen 110B model, 4-bit quantized, gets almost 10 t/s using Ollama [2] and other tools built with llama.cpp. There's also the benefit of being able to load different models simultaneously which is becoming important for RAG and agent-related workflows. [1] https://www.macrumors.com/2024/04/11/m4-ai-chips-late-2024/ https://www.macrumors.com/2024/04/11/m4-ai-chips-late-2024/ [2] https://ollama.com/library/qwen:110b https://ollama.com/library/qwen:110b
hnfong 3y ago
There was some speculations in a Reddit thread: https://www.reddit.com/r/LocalLLaMA/comments/1b5uv86/perplexity_is_not_a_good_measurement_of_how_well/ https://www.reddit.com/r/LocalLLaMA/comments/1b5uv86/perplex... As far as quantifiable results in terms of perplexity go, q4+ quants are generally considered OK. (eg. https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720 )
lhl 3y ago
These numbers are a bit old but will give you a good ballpark for scaling: https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167 You can basically just divide by the multiple as you scale up parameters. Since this is all with a 7B model, just multiply memory by 10X and divide speed by 10X . For batch size=1 (single user interactive inference) if you can fit the model you're basically going to be memory bandwidth limited, but pay attention to the "PP" (prompt generation) number - this is the speed for how long it will take to process any existing conversation. If you're 4000 tokens in, and you are prompt processing at 100 tokens/s, that means you will wait for 40 seconds before any text even starts generating for the next turn. If you're not in a rush, I'd wait for the M4, it's rumored to have much better AI processing (the M3 actually reduced memory bandwidth vs the M2...)
- deleted 3y ago
  [deleted]
loudmax 3y ago
I understand the preference to run models locally rather than in a rented space, but given the speed of development and the amount of money involved, you should have a clear reason why you need to run these models locally. As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G.* The base price for new one with that much unified RAM is around $4000. The Mac Pro towers (not MacBook) have up to 192G unified RAM. The base price for that configuration is around $8600. The smaller LLMs are getting quite good. A lightly quantized Llama-8B should comfortably run on a MacBook Pro with with 16G of RAM which you can get for around $2000. The money you save on a cheaper machine will go a very long way renting compute from a datacenter. If you need to run locally, then high end Macs are excellent machines. Though at those prices you might get better value buying a second hand crypto-mining rig with multiple Nvidia 4090's. EDIT: I was wrong about the MBP unified RAM. You can get an M3 Max with 128GB for around $4700.
- Terretta 3y ago
  > As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G. My Macbook Pro has 2.5x that (128GB), and I run models that use 2x that RAM (96GB) with no impact to my IDE, browser, or other apps running at the time, they act like they're on a 32GB machine. LM Studio makes it easy for newcomers to on-device LLMs to dip your toe into this, both for turning on Metal and helping suggest which models will fit entirely in RAM.
  - loudmax 3y ago
    You're right, you can get an M3 Max MBP with 128GB, starting at $4700. My main point is that if your objective is dipping your toe into this, you can do it with smaller models for far less. That is a really sweet machine, but for the amount of money involved you should be clear about what your needs are.
- coder543 3y ago
  > As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G Huh? They have options up to 128GB… https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space-black-apple-m3-max-with-14-core-cpu-and-30-core-gpu-36gb-memory-1tb https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

by manmal 3y ago

givinguflac 3y ago

All I know is I’ve been disappointed at the limitations of 32GB, lots of inaccessible models with that amount, but also some that are useable. I wish I’d gotten more.

coder543 3y ago

An unquantized Qwen1.5-110B model would require some ~220GB of RAM, so 100+GB would not be "enough" for that, unless we put a big emphasis on the "+". I consider "heavily" quantized to be anything below 4-bit quantization. At 4-bit, you could run a 110B model on around 55GB to 60GB of memory. Right now, Llama-3-70B-Instruct is the highest ranked model you can download[0], and you should be able to fit the 6-bit quantization into 64GB of RAM. Historically, 4-bit quantization represents very little quality loss compared to the full 16-bit models for LLMs, but I have heard rumors that Llama 3 might be so well trained that the quality loss starts to occur earlier, so 6-bit quantization seems like a safe bet for good quality. If you had 128GB of RAM, you still couldn't run the unquantized 70B model, but you could run the 8-bit quantization in a little over 70GB of RAM. Which could feel unsatisfying, since you would have so much unused RAM sitting around, and Apple charges a shocking amount of money for RAM. [0]: https://leaderboard.lmsys.org/ https://leaderboard.lmsys.org/

wongarsu 3y ago
However if you want to use the LLM in your workflow instead of just experimenting with it on its own you also need RAM to run everything else comfortably. 96GB RAM might be a good compromise for now. 64GB is cutting it close, 128GB leaves more breathing room but is expensive.
- coder543 3y ago
  Yep, I agree with that.
andai 3y ago
Phi 3 Q4 spazzes out on some inputs (emits a stream of garbage), while the FP16 version doesn't (at least for the cases I could find). Maybe they just botched the quantization (I have good results with other Q4 models), but it is an interesting data point.
- coder543 3y ago
  Phi 3 in particular had some issues with the end-of-text token not being handled correctly at launch, as I recall, but I could be remembering incorrectly.

zitterbewegung 3y ago

This isn’t the right thread for this but you can look at the model and the difference in parameters. A MacBook Pro will be cheap but have slow inference whereas the using cards will be more expensive , faster for inference (usage) and refining. If you have the money go for the MacBook Pro since it seems that networks under 100B that are trained for a very long time have improved performance and the amount of parameters you need would fit in the MacBook Pro

jprd 3y ago

If running LLMs is your intention, a Mac is probably that absolute worse way to try achieve that. Soldered RAM, no real upgrade path - M2/M3 is cool, but not for this.

jmorgan 3y ago

I think it's worth it, although it might be best to wait for the next iteration: there's rumors the M4 Macs will support up to 512GB of memory [1]. The current 128GB (e.g. M3 Max) and 192GB (e.g. M2 Ultra) Macs run these large models. For example on the M2 Ultra, the Qwen 110B model, 4-bit quantized, gets almost 10 t/s using Ollama [2] and other tools built with llama.cpp. There's also the benefit of being able to load different models simultaneously which is becoming important for RAG and agent-related workflows. [1] https://www.macrumors.com/2024/04/11/m4-ai-chips-late-2024/ https://www.macrumors.com/2024/04/11/m4-ai-chips-late-2024/ [2] https://ollama.com/library/qwen:110b https://ollama.com/library/qwen:110b

hnfong 3y ago

There was some speculations in a Reddit thread: https://www.reddit.com/r/LocalLLaMA/comments/1b5uv86/perplexity_is_not_a_good_measurement_of_how_well/ https://www.reddit.com/r/LocalLLaMA/comments/1b5uv86/perplex... As far as quantifiable results in terms of perplexity go, q4+ quants are generally considered OK. (eg. https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720 )

lhl 3y ago

These numbers are a bit old but will give you a good ballpark for scaling: https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167 You can basically just divide by the multiple as you scale up parameters. Since this is all with a 7B model, just multiply memory by 10X and divide speed by 10X . For batch size=1 (single user interactive inference) if you can fit the model you're basically going to be memory bandwidth limited, but pay attention to the "PP" (prompt generation) number - this is the speed for how long it will take to process any existing conversation. If you're 4000 tokens in, and you are prompt processing at 100 tokens/s, that means you will wait for 40 seconds before any text even starts generating for the next turn. If you're not in a rush, I'd wait for the M4, it's rumored to have much better AI processing (the M3 actually reduced memory bandwidth vs the M2...)

deleted 3y ago
[deleted]

loudmax 3y ago

I understand the preference to run models locally rather than in a rented space, but given the speed of development and the amount of money involved, you should have a clear reason why you need to run these models locally. As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G.* The base price for new one with that much unified RAM is around $4000. The Mac Pro towers (not MacBook) have up to 192G unified RAM. The base price for that configuration is around $8600. The smaller LLMs are getting quite good. A lightly quantized Llama-8B should comfortably run on a MacBook Pro with with 16G of RAM which you can get for around $2000. The money you save on a cheaper machine will go a very long way renting compute from a datacenter. If you need to run locally, then high end Macs are excellent machines. Though at those prices you might get better value buying a second hand crypto-mining rig with multiple Nvidia 4090's. EDIT: I was wrong about the MBP unified RAM. You can get an M3 Max with 128GB for around $4700.

Terretta 3y ago
> As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G. My Macbook Pro has 2.5x that (128GB), and I run models that use 2x that RAM (96GB) with no impact to my IDE, browser, or other apps running at the time, they act like they're on a 32GB machine. LM Studio makes it easy for newcomers to on-device LLMs to dip your toe into this, both for turning on Metal and helping suggest which models will fit entirely in RAM.
- loudmax 3y ago
  You're right, you can get an M3 Max MBP with 128GB, starting at $4700. My main point is that if your objective is dipping your toe into this, you can do it with smaller models for far less. That is a really sweet machine, but for the amount of money involved you should be clear about what your needs are.
coder543 3y ago
> As far as I know, the most RAM you can get in a MacBook Pro (which is what you said you're shopping for) is 48G Huh? They have options up to 128GB… https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space-black-apple-m3-max-with-14-core-cpu-and-30-core-gpu-36gb-memory-1tb https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...