4 ms·

192GB of memory on the Mac Studio is enough to run Llama 65B in full FP16. And at 800GB/s bandwidth, it will do so pretty quickly. I think my M1 Pro memory ba

192GB of memory on the Mac Studio is enough to run Llama 65B in full FP16. And at 800GB/s bandwidth, it will do so pretty quickly. I think my M1 Pro memory bandwidth is 200GB/s and I was running quantized 13B Alpaca relatively quickly, I'd say useable for a personal chatbot, and I think it was swapping every now and then causing pauses. So having 4x the memory bandwidth should allow large models to run pretty damn fast. Maybe not H100 GPGPU speeds but enough for people to do some development on.

jyu 3y ago
What do you think the odds are we can get H100s or equivalent in Mac Studio?
- kristianp 3y ago
  Nvidia gpus haven't been supported by Macs for a long time. Apple and Nvidia relations are not good for some reason.
  - evilduck 3y ago
    https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-what-happened.html https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w... Nvidia had a knack for putting Apple into difficult situations.
    - smoldesu 3y ago
      I think Apple is afraid of the AI people switching to CUDA and neglecting Apple's own accelerators. Whatever the case is, it's entirely petty - Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers). If it was as simple as "Nvidia was an unreliable hardware partner", they wouldn't go out of their way to arbitrarily limit the drivers you can run.
      - throw0101b 3y ago
        > […] Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers). Heck, Nvidia still ships Solaris drivers: * https://www.nvidia.com/en-us/drivers/unix/ https://www.nvidia.com/en-us/drivers/unix/ I guess now that they have the infrastructure for these more niches OSes, it's not too hard to just keep going (especially if the kernel APIs are relatively stable).
      - threeseed 3y ago
        Apple doesn't arbitrarily limit the drivers you can run. And up until now there hasn't been a Mac Nvidia could even sell graphics cards for. Even now Mac Pro users will be such a tiny segment it's probably not worth the effort.
        ssijak 3y ago
        you can connect external graphic cards via thunderbolt and external casings
      - evilduck 3y ago
        Apple’s icy relationship with Nvidia started right around when CUDA came into existence and before Metal ever existed. Nvidia created tensions between Apple and Intel and later handed Apple the headache of a class action lawsuit by knowingly shipping faulty GPUs. I think Apple Silicon and Metal vs CUDA might have been a strategic reason for not repairing that relationship but that came years later.
        andrekandre 3y ago
        > knowingly shipping faulty GPUs. wow, how did i miss this... https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-what-happened.html https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w...
elorant 3y ago
How many tokens per second are you getting from Alpaca?
- selectodude 3y ago
  When llamaCPP came out, I was running 13B at 100ms/token on a base model MacBook Pro 14". Edit: apparently llama.cpp supports running on GPU, so I imagine it's gonna be a bit faster. Maybe a fun evening project for me to get going.
  - deleted 3y ago
    [deleted]
duskwuff 3y ago
> I think it was swapping every now and then causing pauses. What you're seeing is probably "context swapping", not swapping memory to disk. The model can't keep the entire history of its output in context at all times, so LLaMA periodically resets the context and re-prompts it with a portion of its recent output. https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b101bc974cbbf0e0dbbc32d5a3/examples/main/main.cpp#L333-L357 https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b1...

jyu 3y ago

What do you think the odds are we can get H100s or equivalent in Mac Studio?

kristianp 3y ago
Nvidia gpus haven't been supported by Macs for a long time. Apple and Nvidia relations are not good for some reason.
- evilduck 3y ago
  https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-what-happened.html https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w... Nvidia had a knack for putting Apple into difficult situations.
  - smoldesu 3y ago
    I think Apple is afraid of the AI people switching to CUDA and neglecting Apple's own accelerators. Whatever the case is, it's entirely petty - Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers). If it was as simple as "Nvidia was an unreliable hardware partner", they wouldn't go out of their way to arbitrarily limit the drivers you can run.
    - throw0101b 3y ago
      > […] Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers). Heck, Nvidia still ships Solaris drivers: * https://www.nvidia.com/en-us/drivers/unix/ https://www.nvidia.com/en-us/drivers/unix/ I guess now that they have the infrastructure for these more niches OSes, it's not too hard to just keep going (especially if the kernel APIs are relatively stable).
    - threeseed 3y ago
      Apple doesn't arbitrarily limit the drivers you can run. And up until now there hasn't been a Mac Nvidia could even sell graphics cards for. Even now Mac Pro users will be such a tiny segment it's probably not worth the effort.
      - ssijak 3y ago
        you can connect external graphic cards via thunderbolt and external casings
    - evilduck 3y ago
      Apple’s icy relationship with Nvidia started right around when CUDA came into existence and before Metal ever existed. Nvidia created tensions between Apple and Intel and later handed Apple the headache of a class action lawsuit by knowingly shipping faulty GPUs. I think Apple Silicon and Metal vs CUDA might have been a strategic reason for not repairing that relationship but that came years later.
      - andrekandre 3y ago
        > knowingly shipping faulty GPUs. wow, how did i miss this... https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-what-happened.html https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w...

elorant 3y ago

How many tokens per second are you getting from Alpaca?

selectodude 3y ago
When llamaCPP came out, I was running 13B at 100ms/token on a base model MacBook Pro 14". Edit: apparently llama.cpp supports running on GPU, so I imagine it's gonna be a bit faster. Maybe a fun evening project for me to get going.
- deleted 3y ago
  [deleted]

duskwuff 3y ago

> I think it was swapping every now and then causing pauses. What you're seeing is probably "context swapping", not swapping memory to disk. The model can't keep the entire history of its output in context at all times, so LLaMA periodically resets the context and re-prompts it with a portion of its recent output. https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b101bc974cbbf0e0dbbc32d5a3/examples/main/main.cpp#L333-L357 https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b1...