14 ms·

I’m deliberating on how much RAM to get on my new MBP. Is 32gb going to stand me in good stead?

mchiang 3y ago
Local memory management will definitely get better in the future. For now: You should have at least 8 GB of RAM to run the 3B models, 16 GB to run the 7B models, and 32 GB to run the 13B models. My personal recommendation is to get as much memory as you can if you want to work with local models [including VRAM if you are planning to be executing on GPU]
- bdavbdav 3y ago
  Thanks - the issue I’m facing is the CTO lead times on Macs here!
- sourcecodeplz 3y ago
  I run the original llama 7b model just fine on 8GB of Ram. It is best to give advice from experience not only what you read from others.
  - mchiang 3y ago
    This is from us manually testing it on macbooks that we have available. It might run, but it's probably using swap.
rootusrootus 3y ago
32GB should be fine. I went a little overboard and got a new MBP with M2 MAX and 96GB, but the hardware is really best suited at this point to a 30B model. I can and do play around with 65B models, but at that point you're making a fairly big tradeoff in generation speed for an incremental increase in quality. As a datapoint, I have a 30B model [0] loaded right now and it's using 23.44GB of RAM. Getting around 9 tokens/sec, which is very usable. I also have the 65B version of the same model [1] and it's good for around 3.6 tokens/second, but it uses 44GB of RAM. Not unusably slow, but more often than not I opt for the 30B because it's good enough and a lot faster. Haven't tried the llama2 70B yet. [0] https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-GGML https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2... [1] https://huggingface.co/TheBloke/Upstage-Llama1-65B-Instruct-GGML https://huggingface.co/TheBloke/Upstage-Llama1-65B-Instruct-...
- swader999 3y ago
  What's your use case for local if you don't mind?
- bdavbdav 3y ago
  Thankyou that’s really helpful! The CTO lead times on Mac are huge here so it’s either the pro with 16 or the max with 32. Ideally I’d go pro with 64.

Local memory management will definitely get better in the future. For now: You should have at least 8 GB of RAM to run the 3B models, 16 GB to run the 7B models, and 32 GB to run the 13B models. My personal recommendation is to get as much memory as you can if you want to work with local models [including VRAM if you are planning to be executing on GPU]

bdavbdav 3y ago
Thanks - the issue I’m facing is the CTO lead times on Macs here!
sourcecodeplz 3y ago
I run the original llama 7b model just fine on 8GB of Ram. It is best to give advice from experience not only what you read from others.
- mchiang 3y ago
  This is from us manually testing it on macbooks that we have available. It might run, but it's probably using swap.

rootusrootus 3y ago

32GB should be fine. I went a little overboard and got a new MBP with M2 MAX and 96GB, but the hardware is really best suited at this point to a 30B model. I can and do play around with 65B models, but at that point you're making a fairly big tradeoff in generation speed for an incremental increase in quality. As a datapoint, I have a 30B model [0] loaded right now and it's using 23.44GB of RAM. Getting around 9 tokens/sec, which is very usable. I also have the 65B version of the same model [1] and it's good for around 3.6 tokens/second, but it uses 44GB of RAM. Not unusably slow, but more often than not I opt for the 30B because it's good enough and a lot faster. Haven't tried the llama2 70B yet. [0] https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2048-GGML https://huggingface.co/TheBloke/upstage-llama-30b-instruct-2... [1] https://huggingface.co/TheBloke/Upstage-Llama1-65B-Instruct-GGML https://huggingface.co/TheBloke/Upstage-Llama1-65B-Instruct-...

swader999 3y ago
What's your use case for local if you don't mind?
bdavbdav 3y ago
Thankyou that’s really helpful! The CTO lead times on Mac are huge here so it’s either the pro with 16 or the max with 32. Ideally I’d go pro with 64.