5 ms·

LLAMA.cpp with 65B parameters runs on a MacBook M1 Max with 64GB of RAM. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gis

LLAMA.cpp with 65B parameters runs on a MacBook M1 Max with 64GB of RAM. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...

notpushkin 4y ago
What's the correlation between parameter count and RAM usage? Will LLaMA-13B fit on my MacBook Air with 8 GB of RAM or am I stuck with 7B?
- maxxk 4y ago
  13B uses about 9GB on my MacBook Air. If you have another machine (x86) with enough RAM to convert the original LLaMA representation to GGML, you can give it a try. But quantization step must be done on MacBook. Maybe it is more feasible for you to use 7B with larger context. For some "autocompletion" experiments with Python code I had to extend context to 2048 tokens (+1-1.5GB).
cfn 4y ago
I have also seen it working on a Mac Studio with 64Gb of RAM. It is quite slow, not unbearably so, but slow.
aortega 4y ago
That is still a 4000 usd computer. You can get 2 RTX3900 used for ~1000 usd and run 65B much faster. I have a discord server up serving almost 500 users with 65B. https://twitter.com/ortegaalfredo/status/1635402627327590400 https://twitter.com/ortegaalfredo/status/1635402627327590400 For some things is better than GPT3, for other even Alpaca is better.
- chesneyc 4y ago
  Thanks for taking the time to set this up. I will definitely give it a go later today. I don't have access to hardware that I can run LLaMA on and I'm really curious to see what the 65B model has to offer.
- zitterbewegung 4y ago
  How do you make it load on two GPUs or does llama.cpp does it automatically? I have a setup with a threadrippper and a RTX3090 and a Titan RTX. I haven't had the time to set it up so that's why I have been using my Mac.
  - kkielhofner 4y ago
    llama.cpp doesn't use the GPU at all. The genius *.cpp (whisper.cpp, llama.cpp) projects are specifically intended to optimize/democratize otherwise GPU only models to run on CPU/non-GPU (CUDA, ROCm). Technically speaking the released models are capable of running on GPU via standard framework (PyTorch, TensorFlow) support for CPU but in practice without a lot of optimization they are incredibly slow to the point of useless, hence *.cpp. You want something along these lines (warning: unnecessarily potentially offensive): https://rentry.org/llama-tard-v2 https://rentry.org/llama-tard-v2
    - aortega 4y ago
      Llama.cpp takes advantage that LLaMa 7B is a tiny, very optimized model. It would run in anything, and very fast. I really doubt you can run the 30B or 65B models at acceptable speed on a CPU at least for a couple years. (I'm ready to eat my words in a couple weeks)
    - zitterbewegung 4y ago
      Okay my thread ripper can handle it because it has a 128GB of Ram.

notpushkin 4y ago

What's the correlation between parameter count and RAM usage? Will LLaMA-13B fit on my MacBook Air with 8 GB of RAM or am I stuck with 7B?

maxxk 4y ago
13B uses about 9GB on my MacBook Air. If you have another machine (x86) with enough RAM to convert the original LLaMA representation to GGML, you can give it a try. But quantization step must be done on MacBook. Maybe it is more feasible for you to use 7B with larger context. For some "autocompletion" experiments with Python code I had to extend context to 2048 tokens (+1-1.5GB).

cfn 4y ago

I have also seen it working on a Mac Studio with 64Gb of RAM. It is quite slow, not unbearably so, but slow.

aortega 4y ago

That is still a 4000 usd computer. You can get 2 RTX3900 used for ~1000 usd and run 65B much faster. I have a discord server up serving almost 500 users with 65B. https://twitter.com/ortegaalfredo/status/1635402627327590400 https://twitter.com/ortegaalfredo/status/1635402627327590400 For some things is better than GPT3, for other even Alpaca is better.

chesneyc 4y ago
Thanks for taking the time to set this up. I will definitely give it a go later today. I don't have access to hardware that I can run LLaMA on and I'm really curious to see what the 65B model has to offer.
zitterbewegung 4y ago
How do you make it load on two GPUs or does llama.cpp does it automatically? I have a setup with a threadrippper and a RTX3090 and a Titan RTX. I haven't had the time to set it up so that's why I have been using my Mac.
- kkielhofner 4y ago
  llama.cpp doesn't use the GPU at all. The genius *.cpp (whisper.cpp, llama.cpp) projects are specifically intended to optimize/democratize otherwise GPU only models to run on CPU/non-GPU (CUDA, ROCm). Technically speaking the released models are capable of running on GPU via standard framework (PyTorch, TensorFlow) support for CPU but in practice without a lot of optimization they are incredibly slow to the point of useless, hence *.cpp. You want something along these lines (warning: unnecessarily potentially offensive): https://rentry.org/llama-tard-v2 https://rentry.org/llama-tard-v2
  - aortega 4y ago
    Llama.cpp takes advantage that LLaMa 7B is a tiny, very optimized model. It would run in anything, and very fast. I really doubt you can run the 30B or 65B models at acceptable speed on a CPU at least for a couple years. (I'm ready to eat my words in a couple weeks)
  - zitterbewegung 4y ago
    Okay my thread ripper can handle it because it has a 128GB of Ram.