3 ms·

70B+ models typically run great with my MacBook's 96GB of (V)RAM. I want a Mac Studio to run e.g. llama-405B, but I can't justify the marginal model quality ROI

70B+ models typically run great with my MacBook's 96GB of (V)RAM. I want a Mac Studio to run e.g. llama-405B, but I can't justify the marginal model quality ROI for like $7k or whatever. (But I waaant iiit!)

tarruda 2y ago
You can get refurbished Mac Studio M1 Ultra with 128GB VRAM for ~ $3k on ebay. M1 ultra has 800GB/s memory bandwidth, same as the M2 ultra. Not sure if 128GB VRAM is enough for running 405b (maybe at 3-bit quant?), but it seems to offer great value for running 70B models at 8-bit.
- a_wild_dandan 2y ago
  Yeah, I would want the 192GB Mac for attempting such hefty models. But I have such basic bitch needs that 405B is overkill haha.
  - tarruda 2y ago
    405b even at low quants would have very low tokens generation speed, so even if you got the 192GB it would probably not be a good experience. I think 405b is the kind of model that only makes sense to run in clusters of A100/H100. IMO it is not worth it, 70b models at q8 are already pretty darn good, and 128gb is more than enough for those.
    - a_wild_dandan 2y ago
      Exactly! Have you tried the Phi models? To me, they indicate that we can get much more efficient models. In a few years, 70b on gold standard synthetic data + RL might run circles around SotA. It's such an exciting time to be alive.
BaculumMeumEst 2y ago
what quant are you running for that rig? i've been running q4, not sure if I can bump that up to q5 across the board (or if it's worth it in general)
- a_wild_dandan 2y ago
  I run q5s usually, since it's a 40% haircut on model size, with nearly no PPL loss. (Presuming an 8b native model like Qwen.)
diggan 2y ago
> run great How many tokens/second is that approx? For reference, Qwen 2.5 32B on CPU (5950X) with GPU offloading (to RTX 3090ti) gets about 8.5 token/s, while 14B (fully on GPU) gets about ~64 tokens/s.
- a_wild_dandan 2y ago
  For 70B models, I usually get 15-25 t/s on my laptop. Obviously that heavily depends on which quant, context length, etc. I usually roll with q5s, since the loss is so minuscule.

tarruda 2y ago

You can get refurbished Mac Studio M1 Ultra with 128GB VRAM for ~ $3k on ebay. M1 ultra has 800GB/s memory bandwidth, same as the M2 ultra. Not sure if 128GB VRAM is enough for running 405b (maybe at 3-bit quant?), but it seems to offer great value for running 70B models at 8-bit.

a_wild_dandan 2y ago
Yeah, I would want the 192GB Mac for attempting such hefty models. But I have such basic bitch needs that 405B is overkill haha.
- tarruda 2y ago
  405b even at low quants would have very low tokens generation speed, so even if you got the 192GB it would probably not be a good experience. I think 405b is the kind of model that only makes sense to run in clusters of A100/H100. IMO it is not worth it, 70b models at q8 are already pretty darn good, and 128gb is more than enough for those.
  - a_wild_dandan 2y ago
    Exactly! Have you tried the Phi models? To me, they indicate that we can get much more efficient models. In a few years, 70b on gold standard synthetic data + RL might run circles around SotA. It's such an exciting time to be alive.

BaculumMeumEst 2y ago

what quant are you running for that rig? i've been running q4, not sure if I can bump that up to q5 across the board (or if it's worth it in general)

a_wild_dandan 2y ago
I run q5s usually, since it's a 40% haircut on model size, with nearly no PPL loss. (Presuming an 8b native model like Qwen.)

diggan 2y ago

> run great How many tokens/second is that approx? For reference, Qwen 2.5 32B on CPU (5950X) with GPU offloading (to RTX 3090ti) gets about 8.5 token/s, while 14B (fully on GPU) gets about ~64 tokens/s.

a_wild_dandan 2y ago
For 70B models, I usually get 15-25 t/s on my laptop. Obviously that heavily depends on which quant, context length, etc. I usually roll with q5s, since the loss is so minuscule.