4 ms·

A 192gb Mac Studio should be able to run an unquantized 70B and I think would cost less than running a multi gpu setup made up of nvidia cards. I haven’t actual

A 192gb Mac Studio should be able to run an unquantized 70B and I think would cost less than running a multi gpu setup made up of nvidia cards. I haven’t actually done the math, though. If you factor in electricity costs over a certain time period it might make the Mac even cheaper!

oceanplexian 3y ago
A Mac studio will "run" the model as a glorified chat bot but it'll be unusable for anything interesting at 5-6t/s. With a couple of high end consumer GPUs you're going to get closer to 20t/s. You also be able to realistically fine tune models, and run other interesting things besides an LLM.
- vczf 3y ago
  I have a $5000 128GB M2 Ultra Mac Studio that I got for LLMs due to speculation like GP here on HN. I get 7.7 tok/s with LLaMA2 70B q6_K ggml (llama.cpp). It has some upsides in that I can run quantizations larger than 48GB with extended context, or run multiple models at once, but overall I wouldn't strongly recommend it for LLMs over an Intel+2x4090 setup. It's competitive, but has significant tradeoffs.
- kolleykibber 3y ago
  Could you give some examples of these interesting things?
throwawayadvsec 3y ago
your idea of cheapest is using a mac to run a LLM? are you out of your mind?
- gorbypark 3y ago
  Well, to be fair, to run an unquantized 70B model is going to take somewhere in the area of 160gb of VRAM (if my quick back of the napkin math is ok). I'm not quite sure of the state of GPUs these days, but getting a 2x a100 80gb (or 4x 40gb) setup is probably going to cost more than a Mac Studio with maxed out RAM. If we are talking quantized, I am currently running LLaMA v1 30B at 4 bits on a MacBook Air 24GB ram, which is only a little bit more expensive than what a 24GB 4090 retails for. The 4090 would crush the MacBook Air in tokens/sec, I am sure. It is however completely usable on my MacBook (4 tokens/second, IIRC? I might be off on that). A 4 bit 70B model should take about 36GB-40GB of RAM so a 64GB MacStudio might still be price competitive with a dual 4090 or 4090 / 3090 split setup. The cheapest Studio with 64GB of RAM is 2,399.00 (USD).
smoldesu 3y ago
Inferencing would probably be ~10x slower than tiling the model across equivalently priced Nvidia hardware. The highest-end M2 Mac chip you can purchase today struggles to compete with last-gen laptop cards from Nvidia. Once you factor in the value of CUDA in this space and the quality of ML driver Nvidia offers, I don't see why Macs are even considered in the "cheapest hardware" discussion. > If you factor in electricity costs over a certain time period it might make the Mac even cheaper! I dunno about that. The M2 Max will happily pull over 200w in GPU-heavy tasks, if we're comparing a 40-series card with CUDA optimizations to Pytorch with Metal Performance Shaders, my performance-per-watt money is on Nvidia's hardware.

oceanplexian 3y ago

A Mac studio will "run" the model as a glorified chat bot but it'll be unusable for anything interesting at 5-6t/s. With a couple of high end consumer GPUs you're going to get closer to 20t/s. You also be able to realistically fine tune models, and run other interesting things besides an LLM.

vczf 3y ago
I have a $5000 128GB M2 Ultra Mac Studio that I got for LLMs due to speculation like GP here on HN. I get 7.7 tok/s with LLaMA2 70B q6_K ggml (llama.cpp). It has some upsides in that I can run quantizations larger than 48GB with extended context, or run multiple models at once, but overall I wouldn't strongly recommend it for LLMs over an Intel+2x4090 setup. It's competitive, but has significant tradeoffs.
kolleykibber 3y ago
Could you give some examples of these interesting things?

throwawayadvsec 3y ago

your idea of cheapest is using a mac to run a LLM? are you out of your mind?

gorbypark 3y ago
Well, to be fair, to run an unquantized 70B model is going to take somewhere in the area of 160gb of VRAM (if my quick back of the napkin math is ok). I'm not quite sure of the state of GPUs these days, but getting a 2x a100 80gb (or 4x 40gb) setup is probably going to cost more than a Mac Studio with maxed out RAM. If we are talking quantized, I am currently running LLaMA v1 30B at 4 bits on a MacBook Air 24GB ram, which is only a little bit more expensive than what a 24GB 4090 retails for. The 4090 would crush the MacBook Air in tokens/sec, I am sure. It is however completely usable on my MacBook (4 tokens/second, IIRC? I might be off on that). A 4 bit 70B model should take about 36GB-40GB of RAM so a 64GB MacStudio might still be price competitive with a dual 4090 or 4090 / 3090 split setup. The cheapest Studio with 64GB of RAM is 2,399.00 (USD).

smoldesu 3y ago

Inferencing would probably be ~10x slower than tiling the model across equivalently priced Nvidia hardware. The highest-end M2 Mac chip you can purchase today struggles to compete with last-gen laptop cards from Nvidia. Once you factor in the value of CUDA in this space and the quality of ML driver Nvidia offers, I don't see why Macs are even considered in the "cheapest hardware" discussion. > If you factor in electricity costs over a certain time period it might make the Mac even cheaper! I dunno about that. The M2 Max will happily pull over 200w in GPU-heavy tasks, if we're comparing a 40-series card with CUDA optimizations to Pytorch with Metal Performance Shaders, my performance-per-watt money is on Nvidia's hardware.