5 ms·

People who know more than me: they’re talking a lot about RAM and not much about GPU. Do you expect this will be able to handle AI workloads well? All I’ve he

People who know more than me: they’re talking a lot about RAM and not much about GPU. Do you expect this will be able to handle AI workloads well? All I’ve heard for the past two years is how important a beefy GPU is. Curious if that holds true here too.

lynndotpy 2y ago
VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.
- dartos 2y ago
  You can say the same about GPU clock speed as well…
- vlovich123 2y ago
  No, with limited VRAM you could offload the model partially or split across CPU and GPU. And since CPU has swap, you could run the absolute largest model. It’s just really really slow.
  - jeffhuys 2y ago
    Really, really, really, really, really, REALLY REALLY slow.
  - Espressosaurus 2y ago
    The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2. The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing. The M4's prompt parsing took significantly longer, but token generation was significantly faster. Having the memory to the cores that matter makes a big difference.
    - vlovich123 2y ago
      You're adding detail that's not relevant to anything I said. I was saying this statement: > VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis. Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
      - Espressosaurus 2y ago
        The criteria for purchase for anybody trying to use it is "run slowly but acceptably" vs. "run so slow as to be unusable". My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes. edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s. edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation. Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.
        vlovich123 2y ago
        I don't disagree. I'm just taking OP at the literal statement they made.
      - lynndotpy 2y ago
        Sure, this is technically correct, but somewhere there's a line of practicality. Running off a CPU (especially with swap) will be past that line. Otherwise, you don't even need a computer. Pen and paper is plenty. For all practical purposes, VRAM is a limiting factor.
Retr0id 2y ago
When it comes to LLMs in particular, it comes down to memory size+bandwidth more than anything else.
simlevesque 2y ago
What's more important isn't how beefy it is, it's how much memory it has. These are unified memory. The M3 Ultra with 512gb has as much VRAM as sixteen 5090.
qwertox 2y ago
A beefy GPU which can't hold models in VRAM is of very limited use. You'll see 16 GB of VRAM on gamer Nvidia cards, the RTX 5090 being an exception with 32 GB VRAM. The professional cards have around 96 GB of VRAM. The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exist on those devices). And while Apple's integrated GPU isn't as powerful as an Nvidia GPU, it is powerful enough for non-professional workloads and has the huge benefit of access to lots of memory.
- _zoltan_ 2y ago
  which professional card has 96GB of VRAM?
  - qwertox 2y ago
    Like the NVIDIA H100 NVL 94GB HBM3 PCIe 5.0 Data Center GPU for 27.651,20 € https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-hbm3-pcie-5-0-data-center-gpu/ https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-...
    - _zoltan_ 2y ago
      that's not a professional line card, but a data center card.
gatienboquet 2y ago
LLMs are primarily "memory-bound" rather than "compute-bound" during normal use. The model weights (billions of parameters) must be loaded into memory before you can use them. Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible. Processing power still matters for speed once everything fits in memory, but it's secondary to having enough VRAM in the first place.
- whimsicalism 2y ago
  Transformers are typically memory-bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips. My guess is that these chips could be compute-bound though given how little compute capacity they have.
  - Gracana 2y ago
    It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of GPUs, chewing through long prompts, code, documents etc is going to be slower.
    - whimsicalism 2y ago
      nobody in industry is using a 4090, they are using H100s which have 3TB/s. Apple also doesn’t have any equivalent to nvlink. I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops
      - Gracana 2y ago
        I chose the 3090/4090 because it seems to me that this machine could be a replacement for a workstation or a homelab rig at a similar price point, but not a $100-250k server in a datacenter. It's not really surprising or interesting that the datacenter GPUs are superior. FWIW I went the route of "bunch of GPUs in a desktop case" because I felt having the compute oomph was worth it.
matwood 2y ago
I was able to run and use the DeepSeek distilled 24gb on an M1 Max with 64gb of ram. It wasn't speedy, but it was usable. I imagine the M3/4s are much faster, especially on smaller, more specific models.

lynndotpy 2y ago

VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.

dartos 2y ago
You can say the same about GPU clock speed as well…
vlovich123 2y ago
No, with limited VRAM you could offload the model partially or split across CPU and GPU. And since CPU has swap, you could run the absolute largest model. It’s just really really slow.
- jeffhuys 2y ago
  Really, really, really, really, really, REALLY REALLY slow.
- Espressosaurus 2y ago
  The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2. The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing. The M4's prompt parsing took significantly longer, but token generation was significantly faster. Having the memory to the cores that matter makes a big difference.
  - vlovich123 2y ago
    You're adding detail that's not relevant to anything I said. I was saying this statement: > VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis. Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
    - Espressosaurus 2y ago
      The criteria for purchase for anybody trying to use it is "run slowly but acceptably" vs. "run so slow as to be unusable". My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes. edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s. edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation. Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.
      - vlovich123 2y ago
        I don't disagree. I'm just taking OP at the literal statement they made.
    - lynndotpy 2y ago
      Sure, this is technically correct, but somewhere there's a line of practicality. Running off a CPU (especially with swap) will be past that line. Otherwise, you don't even need a computer. Pen and paper is plenty. For all practical purposes, VRAM is a limiting factor.

Retr0id 2y ago

When it comes to LLMs in particular, it comes down to memory size+bandwidth more than anything else.

simlevesque 2y ago

What's more important isn't how beefy it is, it's how much memory it has. These are unified memory. The M3 Ultra with 512gb has as much VRAM as sixteen 5090.

qwertox 2y ago

A beefy GPU which can't hold models in VRAM is of very limited use. You'll see 16 GB of VRAM on gamer Nvidia cards, the RTX 5090 being an exception with 32 GB VRAM. The professional cards have around 96 GB of VRAM. The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exist on those devices). And while Apple's integrated GPU isn't as powerful as an Nvidia GPU, it is powerful enough for non-professional workloads and has the huge benefit of access to lots of memory.

_zoltan_ 2y ago
which professional card has 96GB of VRAM?
- qwertox 2y ago
  Like the NVIDIA H100 NVL 94GB HBM3 PCIe 5.0 Data Center GPU for 27.651,20 € https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-hbm3-pcie-5-0-data-center-gpu/ https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-...
  - _zoltan_ 2y ago
    that's not a professional line card, but a data center card.

gatienboquet 2y ago

LLMs are primarily "memory-bound" rather than "compute-bound" during normal use. The model weights (billions of parameters) must be loaded into memory before you can use them. Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible. Processing power still matters for speed once everything fits in memory, but it's secondary to having enough VRAM in the first place.

whimsicalism 2y ago
Transformers are typically memory-bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips. My guess is that these chips could be compute-bound though given how little compute capacity they have.
- Gracana 2y ago
  It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of GPUs, chewing through long prompts, code, documents etc is going to be slower.
  - whimsicalism 2y ago
    nobody in industry is using a 4090, they are using H100s which have 3TB/s. Apple also doesn’t have any equivalent to nvlink. I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops
    - Gracana 2y ago
      I chose the 3090/4090 because it seems to me that this machine could be a replacement for a workstation or a homelab rig at a similar price point, but not a $100-250k server in a datacenter. It's not really surprising or interesting that the datacenter GPUs are superior. FWIW I went the route of "bunch of GPUs in a desktop case" because I felt having the compute oomph was worth it.

matwood 2y ago

I was able to run and use the DeepSeek distilled 24gb on an M1 Max with 64gb of ram. It wasn't speedy, but it was usable. I imagine the M3/4s are much faster, especially on smaller, more specific models.