5 ms·

> 10 grand for 256GB memory. A NVIDIA RTX 6000, 96 GB at 1.7 TB/s, is 13 grand. This 256 GB at 1.2 TB/s Mac is extremely competitive, it will be sold out ever

> 10 grand for 256GB memory. A NVIDIA RTX 6000, 96 GB at 1.7 TB/s, is 13 grand. This 256 GB at 1.2 TB/s Mac is extremely competitive, it will be sold out everywhere.

angoragoats 2mo ago
Except the RTX 6000 will run circles around the Mac studio in just about every way. Memory bandwidth is literally the only spec where Apple is competitive, and while high memory bandwidth is necessary for LLMs to perform well, many people strangely don't understand that memory bandwidth alone is not sufficient.
- F7F7F7 2mo ago
  It better because you’ll need a few of them to run some larger models (I’ll be just as vague citing which models).
  - angoragoats 2mo ago
    I was not speaking about specific models so I didn’t feel the need to cite any. Not sure why the backhanded insult was necessary. You need multiple Mac Studios to run the largest models as well, so neither is a one-size-fits-all device. If you can show me a model for which a Mac is faster than the RTX 6000 then I’ll be happy to update or retract my statement.
- speedgoose 2mo ago
  Energy efficiency is in another league with the Mac Studio for local workloads. I can run agents using deepseek v4 flash or Qwen 3.8 on my m3 ultra and it will be lukewarm and the fan will eventually start blowing softly.
  - angoragoats 2mo ago
    I’ve run the numbers on this, and an optimized nVidia build can meet or beat Apple platforms in terms of tokens per watt, which is the most important efficiency metric if what you care about is using the least amount of energy to generate a given response. Yes, the Mac might get lukewarm, but it will take 2-3+ times longer to do the same task.
    - speedgoose 2mo ago
      Yes I agree that at full speed with parallel workloads, the tokens per watt are better using Nvidia GPU servers. Also while the pre fill performance sucks, the Mac isn’t that slow and can use much better model compared to a similarly priced Nvidia workstation so it’s not really taking much longer in practice. It’s taking longer than in the cloud for sure. At least a local computer uses the local energy grid that is pretty clean and not fossil energy.
      - angoragoats 2mo ago
        Okay, so you said “Energy efficiency is in another league with the Mac Studio”; I assume you meant that it was more efficient than an nVidia GPU setup. But when pressed, you agreed that’s not actually the case. So it’s not actually that much different in terms of energy efficiency. > Also while the pre fill performance sucks, the Mac isn’t that slow and can use much better model compared to a similarly priced Nvidia workstation so it’s not really taking much longer in practice. This is exactly backwards; assuming you mean that the Mac has more RAM so you can use a larger model, the mac is going to be _even slower_ since the fastest Mac isn’t as fast as the average nVidia setup. My point is that comparing apples to apples (no pun intended), an nVidia setup is both faster and (possibly with some tweaking) more efficient than a Mac.
- aurareturn 2mo ago
  Mac studio wins in memory capacity, price, perf/watt and value. RTX 6000 wins in performance, if your model can fit into the VRAM. There are very obvious and clear advantages to a Mac Studio. It's an entire system for one and you're getting a world class CPU as well.
  - angoragoats 2mo ago
    > Mac studio wins in memory capacity, price, perf/watt and value. [citation needed]. I have personally specced out and built an nvidia GPU-based machine which after some optimization, handily beat the Mac Studio in terms of tokens/watt for LLM inference with most models. This was in the M2 Ultra era, and I haven't run the numbers for the later generations, but nvidia's cards have gotten faster just as Apple's CPUs/GPUs have, so I would guess that it's still possible to do. > RTX 6000 wins in performance, if your model can fit into the VRAM. "if your model can fit into the VRAM" can be true for the Mac as well. > There are very obvious and clear advantages to a Mac Studio. There are certain advantages for sure, depending on your use case. They may _seem_ to be obvious, but as evidenced above, I believe that many people overestimate the Mac's superiority on the metrics you cite when comparing a Mac vs. a dedicated GPU for LLM inference.
    - seanmcdirmid 2mo ago
      > "if your model can fit into the VRAM" can be true for the Mac as well. It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. Even going with two 5090s, you now have to shard your model and that is a PITA. But it turns out that MoE is the solution both for running models on macs of limited computer power means (not as fast as GPUs), and on multiple GPUs that require sharding the model.
      - angoragoats 2mo ago
        > It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. I bristle at general statements like this when it obviously depends on the specific Mac and GPU in question. But yes, comparing a maxed out M5 Ultra with an RTX 6000, the Mac has much more memory. > Even going with two 5090s, you now have to shard your model and that is a PITA. Every modern tool does this for you automatically. It is absolutely not a pain in the least (e.g. llama.cpp ships with pipeline parallelism enabled by default).
petercooper 2mo ago
How's the compute side now, I wonder? Because while the Ultras have impressive memory bandwidth for inference, processing prompts still takes a dog's age on my M3 Ultra. I heard the M5 makes some strides forward in this area, though, and the M7 in particular promises to go a lot further.
- deleted 2mo ago
  [deleted]
- dannyw 2mo ago
  M5 is excellent, they’ve finally gotten their own tensor cores. Good for inference; however if you like to train, data format support and effective performance is limited (M5 Pro). Some hardware features are not exposed or extremely slow. You’ll be fine for inference, but pales in comparison to what a RTX 6000 Pro can do for compute/matmuls/training.
- aurareturn 2mo ago
  4x faster prompt processing than M3 Ultra.
blints 2mo ago
The relevant comparison isn't one mac studio to one RTX 6000, it's a 24 channel DDR5 system, which also has ~1.2TB/s of memory bandwidth (or more when Xeon 6 compatible 8800mt/s memory becomes widely available), vastly higher prefill due to more CPU horsepower, orders of magnitude faster networking, can hook into GPU accelerators, can be upgraded etc. A baseline 384GB system from eg Puget is ~30K vs ~12K for the 256GB Mac Studio and you do get value for the money.
- ricardobeat 2mo ago
  So 3x more, plus the cost of a GPU (another 10k?). How is that value for money to get slightly better performance?
  - blints 2mo ago
    It can be more than "slightly", particularly if the model you're interested in (or will be interested in in 6 months) doesn't fit on the mac studio. You also need to account for eg storing 10TB of random checkpoints, load time when experimenting, and so on. When you start actually needing throughput these are all capability gaps in practical use, not just x% benchmark differences. If you just want to run Qwen 3.8 27B and Deepseek v4 Flash in perpetuity and that's it, there are a lot of solutions that will work and this is a fairly user friendly one.
    - kridsdale1 2mo ago
      You can find model variants to scale up to whatever capacity you have. Queen has models that just barely fit in 256gb, I ran them… okay… on my Studio.
    - freehorse 2mo ago
      A 3x price difference means you can get 3 256GB mac studios which you can connect through thunderbolt and with RDMA a total of 768GB ram with compute/memory bandwidth scaling basically linearly (with a small overhead cost).
    - aurareturn 2mo ago
ProllyInfamous 2mo ago
I've been using Macs as my primary machine for over three decades – but in the past year a Microcenter-Prebuilt (with a 5070Ti) has helped make Ubuntu my preferable computing environment (I have three Silicon Macs with similar access to 16GB vRAM – the 5070Ti "smokes" them in compute). But my next machine for LLMs will probably be the Mac Mini (with 64GB vRAM access) for low-$2300s – which is what I paid for my MC-Prebuilt (and am still very happy with – Ubuntu is great). After decades of installing and forgetting various shades of linux-distro, Ubuntu has kept me "using" the computer as a tool, instead of "just tinkering with it").

> 10 grand for 256GB memory. A NVIDIA RTX 6000, 96 GB at 1.7 TB/s, is 13 grand. This 256 GB at 1.2 TB/s Mac is extremely competitive, it will be sold out everywhere.

angoragoats 2mo ago

Except the RTX 6000 will run circles around the Mac studio in just about every way. Memory bandwidth is literally the only spec where Apple is competitive, and while high memory bandwidth is necessary for LLMs to perform well, many people strangely don't understand that memory bandwidth alone is not sufficient.

F7F7F7 2mo ago
It better because you’ll need a few of them to run some larger models (I’ll be just as vague citing which models).
- angoragoats 2mo ago
  I was not speaking about specific models so I didn’t feel the need to cite any. Not sure why the backhanded insult was necessary. You need multiple Mac Studios to run the largest models as well, so neither is a one-size-fits-all device. If you can show me a model for which a Mac is faster than the RTX 6000 then I’ll be happy to update or retract my statement.
speedgoose 2mo ago
Energy efficiency is in another league with the Mac Studio for local workloads. I can run agents using deepseek v4 flash or Qwen 3.8 on my m3 ultra and it will be lukewarm and the fan will eventually start blowing softly.
- angoragoats 2mo ago
  I’ve run the numbers on this, and an optimized nVidia build can meet or beat Apple platforms in terms of tokens per watt, which is the most important efficiency metric if what you care about is using the least amount of energy to generate a given response. Yes, the Mac might get lukewarm, but it will take 2-3+ times longer to do the same task.
  - speedgoose 2mo ago
    Yes I agree that at full speed with parallel workloads, the tokens per watt are better using Nvidia GPU servers. Also while the pre fill performance sucks, the Mac isn’t that slow and can use much better model compared to a similarly priced Nvidia workstation so it’s not really taking much longer in practice. It’s taking longer than in the cloud for sure. At least a local computer uses the local energy grid that is pretty clean and not fossil energy.
    - angoragoats 2mo ago
      Okay, so you said “Energy efficiency is in another league with the Mac Studio”; I assume you meant that it was more efficient than an nVidia GPU setup. But when pressed, you agreed that’s not actually the case. So it’s not actually that much different in terms of energy efficiency. > Also while the pre fill performance sucks, the Mac isn’t that slow and can use much better model compared to a similarly priced Nvidia workstation so it’s not really taking much longer in practice. This is exactly backwards; assuming you mean that the Mac has more RAM so you can use a larger model, the mac is going to be _even slower_ since the fastest Mac isn’t as fast as the average nVidia setup. My point is that comparing apples to apples (no pun intended), an nVidia setup is both faster and (possibly with some tweaking) more efficient than a Mac.
aurareturn 2mo ago
Mac studio wins in memory capacity, price, perf/watt and value. RTX 6000 wins in performance, if your model can fit into the VRAM. There are very obvious and clear advantages to a Mac Studio. It's an entire system for one and you're getting a world class CPU as well.
- angoragoats 2mo ago
  > Mac studio wins in memory capacity, price, perf/watt and value. [citation needed]. I have personally specced out and built an nvidia GPU-based machine which after some optimization, handily beat the Mac Studio in terms of tokens/watt for LLM inference with most models. This was in the M2 Ultra era, and I haven't run the numbers for the later generations, but nvidia's cards have gotten faster just as Apple's CPUs/GPUs have, so I would guess that it's still possible to do. > RTX 6000 wins in performance, if your model can fit into the VRAM. "if your model can fit into the VRAM" can be true for the Mac as well. > There are very obvious and clear advantages to a Mac Studio. There are certain advantages for sure, depending on your use case. They may _seem_ to be obvious, but as evidenced above, I believe that many people overestimate the Mac's superiority on the metrics you cite when comparing a Mac vs. a dedicated GPU for LLM inference.
  - seanmcdirmid 2mo ago
    > "if your model can fit into the VRAM" can be true for the Mac as well. It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. Even going with two 5090s, you now have to shard your model and that is a PITA. But it turns out that MoE is the solution both for running models on macs of limited computer power means (not as fast as GPUs), and on multiple GPUs that require sharding the model.
    - angoragoats 2mo ago
      > It is much more likely for your model to fit in large unified memory of a Mac than the smaller more limited memory of a GPU. I bristle at general statements like this when it obviously depends on the specific Mac and GPU in question. But yes, comparing a maxed out M5 Ultra with an RTX 6000, the Mac has much more memory. > Even going with two 5090s, you now have to shard your model and that is a PITA. Every modern tool does this for you automatically. It is absolutely not a pain in the least (e.g. llama.cpp ships with pipeline parallelism enabled by default).

petercooper 2mo ago

How's the compute side now, I wonder? Because while the Ultras have impressive memory bandwidth for inference, processing prompts still takes a dog's age on my M3 Ultra. I heard the M5 makes some strides forward in this area, though, and the M7 in particular promises to go a lot further.

deleted 2mo ago
[deleted]
dannyw 2mo ago
M5 is excellent, they’ve finally gotten their own tensor cores. Good for inference; however if you like to train, data format support and effective performance is limited (M5 Pro). Some hardware features are not exposed or extremely slow. You’ll be fine for inference, but pales in comparison to what a RTX 6000 Pro can do for compute/matmuls/training.
aurareturn 2mo ago
4x faster prompt processing than M3 Ultra.

blints 2mo ago

The relevant comparison isn't one mac studio to one RTX 6000, it's a 24 channel DDR5 system, which also has ~1.2TB/s of memory bandwidth (or more when Xeon 6 compatible 8800mt/s memory becomes widely available), vastly higher prefill due to more CPU horsepower, orders of magnitude faster networking, can hook into GPU accelerators, can be upgraded etc. A baseline 384GB system from eg Puget is ~30K vs ~12K for the 256GB Mac Studio and you do get value for the money.

ricardobeat 2mo ago
So 3x more, plus the cost of a GPU (another 10k?). How is that value for money to get slightly better performance?
- blints 2mo ago
  It can be more than "slightly", particularly if the model you're interested in (or will be interested in in 6 months) doesn't fit on the mac studio. You also need to account for eg storing 10TB of random checkpoints, load time when experimenting, and so on. When you start actually needing throughput these are all capability gaps in practical use, not just x% benchmark differences. If you just want to run Qwen 3.8 27B and Deepseek v4 Flash in perpetuity and that's it, there are a lot of solutions that will work and this is a fairly user friendly one.
  - kridsdale1 2mo ago
    You can find model variants to scale up to whatever capacity you have. Queen has models that just barely fit in 256gb, I ran them… okay… on my Studio.
  - freehorse 2mo ago
    A 3x price difference means you can get 3 256GB mac studios which you can connect through thunderbolt and with RDMA a total of 768GB ram with compute/memory bandwidth scaling basically linearly (with a small overhead cost).
  - aurareturn 2mo ago

ProllyInfamous 2mo ago

I've been using Macs as my primary machine for over three decades – but in the past year a Microcenter-Prebuilt (with a 5070Ti) has helped make Ubuntu my preferable computing environment (I have three Silicon Macs with similar access to 16GB vRAM – the 5070Ti "smokes" them in compute). But my next machine for LLMs will probably be the Mac Mini (with 64GB vRAM access) for low-$2300s – which is what I paid for my MC-Prebuilt (and am still very happy with – Ubuntu is great). After decades of installing and forgetting various shades of linux-distro, Ubuntu has kept me "using" the computer as a tool, instead of "just tinkering with it").