10 ms·

The memory amount is fantastic, memory bandwidth is half decent(~800 GB/s), and the compute capabilities are terrible(36 TOPS). For comparison, a single consum

The memory amount is fantastic, memory bandwidth is half decent(~800 GB/s), and the compute capabilities are terrible(36 TOPS). For comparison, a single consumer card like the RTX 5090 is only 32 GB of memory, has 1792 GB/s memory and 3593 TOPS of compute. The use cases will be limited. While you can't run a 600B model directly like Apple says(cause you need more memory for that), you can run a quantized version, but it will be very slow unless its a MoE architecture.

Havoc 2y ago
>36 Tops Thats going to be the NPU specifically. Pretty much nothing on llm front seems to use NPUs at this stage (copilot snapdragon laptops aside) so not sure the low number is a problem
BonoboIO 2y ago
A factor of 100 faster in compute … wow. It will be interesting when somebody will upgrade the ram ram of the 5090 like they did with 4090s
- bilbo0s 2y ago
  They’re a bit confused and not comparing the same compute. Pretty sure they’re comparing Nvidia’s gpu to Apple’s npu.
  - NorwegianDude 2y ago
    I'm not confused at all. It's the real numbers. Feel free to provide anything that suggests that the TOPS of the GPU in M chips are faster than the dedicated hardware for it. But you can't, cause it's not true. If you think Apple added the neural engine just for fun then I don't know what to tell you. You have a fundamental flaw in your understanding of how both chips work. Not using the tensor cores would be slower, and the same goes for apples neural engine. The numbers are both for the hardware both have implemented for maximum performance for this task.
llm_nerd 2y ago
I do think people are going a little overboard with all the commentary about AI in this discussion, and you rightly cite some of the empirical reasons. People are trying to rationalize convincing themselves to buy one of these, but they're deluding themselves. It's nice that these devices have loads of memory, but they don't have remotely the necessary level of compute to be competitive in the AI space. As a fun thing to run a local LLM as a hobbyist, sure, but this presents zero threat to nvidia. Apple hardware is irrelevant in the AI space, outside of making YouTube "I ran a quantized LLM on my 128GB Mac Mini" type content for clicks, and this release doesn't change that. Looks like a great desktop chip though. It would be nice if nvidia could start giving their less expensive offerings more memory, though they're currently in the realm Intel was 15 yearsago, thinking that their biggest competition is themselves.
dagmx 2y ago
You're comparing two different things. The compute level you’re talking about on the M3 Ultra is the neural engine. Not including the GPU. I expect the GPU here will be behind a 5090 for compute but not by the unrelated numbers you’re quoting. After all, the 5090 alone is multiple times the wattage of this SoC.
- bigyabai 2y ago
  > After all, the 5090 alone is multiple times the wattage of this SoC. FWIW, normalizing the wattages (or even underclocking the GPU) will still give you an Nvidia advantage most days. Apple's GPU designs are closer to AMD's designs than Nvidia's, which means they omit a lot of AI accelerators to focus on a less-LLM-relevent raster performance figure. Yes, the GPU is faster than the NPU. But Apple's GPU designs haven't traditionally put their competitors out of a job.
  - dagmx 2y ago
    M2 Ultra is ~250W (averaging various reports since Apple don’t publish) for the entire SoC. 5090 is 575W without the CPU. You’d have to cut the Nvidia to a quarter and then find a comparable CPU to normalize the wattage for an actual comparison. I agree that Apple GPUs aren’t putting the dedicated GPU companies in danger on the benchmarks, but they’re also not really targeting it? They’re in completely different zones on too many fronts to really compare.
    - bigyabai 2y ago
      Well, select your hardware of choice and see for yourself then: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks > but they’re also not really targeting it? That's fine, but it's not an excuse to ignore the power/performance ratio.
      - dagmx 2y ago
        But I’m not ignoring the power/performance ratio? If anything, you are doing that by handwaving away the difference. Give me a comparable system build where the NVIDIA GPU + any CPU of your choice is running at the same wattage as an M2 Ultra, and outperforms it on average. You’d get 150W for the GPU and 150W for the CPU. Again, you can’t really compare the two. They’re inherently different systems unless you only care about singular metrics.

Havoc 2y ago

>36 Tops Thats going to be the NPU specifically. Pretty much nothing on llm front seems to use NPUs at this stage (copilot snapdragon laptops aside) so not sure the low number is a problem

BonoboIO 2y ago

A factor of 100 faster in compute … wow. It will be interesting when somebody will upgrade the ram ram of the 5090 like they did with 4090s

bilbo0s 2y ago
They’re a bit confused and not comparing the same compute. Pretty sure they’re comparing Nvidia’s gpu to Apple’s npu.
- NorwegianDude 2y ago
  I'm not confused at all. It's the real numbers. Feel free to provide anything that suggests that the TOPS of the GPU in M chips are faster than the dedicated hardware for it. But you can't, cause it's not true. If you think Apple added the neural engine just for fun then I don't know what to tell you. You have a fundamental flaw in your understanding of how both chips work. Not using the tensor cores would be slower, and the same goes for apples neural engine. The numbers are both for the hardware both have implemented for maximum performance for this task.

llm_nerd 2y ago

I do think people are going a little overboard with all the commentary about AI in this discussion, and you rightly cite some of the empirical reasons. People are trying to rationalize convincing themselves to buy one of these, but they're deluding themselves. It's nice that these devices have loads of memory, but they don't have remotely the necessary level of compute to be competitive in the AI space. As a fun thing to run a local LLM as a hobbyist, sure, but this presents zero threat to nvidia. Apple hardware is irrelevant in the AI space, outside of making YouTube "I ran a quantized LLM on my 128GB Mac Mini" type content for clicks, and this release doesn't change that. Looks like a great desktop chip though. It would be nice if nvidia could start giving their less expensive offerings more memory, though they're currently in the realm Intel was 15 yearsago, thinking that their biggest competition is themselves.

dagmx 2y ago

You're comparing two different things. The compute level you’re talking about on the M3 Ultra is the neural engine. Not including the GPU. I expect the GPU here will be behind a 5090 for compute but not by the unrelated numbers you’re quoting. After all, the 5090 alone is multiple times the wattage of this SoC.

bigyabai 2y ago
> After all, the 5090 alone is multiple times the wattage of this SoC. FWIW, normalizing the wattages (or even underclocking the GPU) will still give you an Nvidia advantage most days. Apple's GPU designs are closer to AMD's designs than Nvidia's, which means they omit a lot of AI accelerators to focus on a less-LLM-relevent raster performance figure. Yes, the GPU is faster than the NPU. But Apple's GPU designs haven't traditionally put their competitors out of a job.
- dagmx 2y ago
  M2 Ultra is ~250W (averaging various reports since Apple don’t publish) for the entire SoC. 5090 is 575W without the CPU. You’d have to cut the Nvidia to a quarter and then find a comparable CPU to normalize the wattage for an actual comparison. I agree that Apple GPUs aren’t putting the dedicated GPU companies in danger on the benchmarks, but they’re also not really targeting it? They’re in completely different zones on too many fronts to really compare.
  - bigyabai 2y ago
    Well, select your hardware of choice and see for yourself then: https://browser.geekbench.com/opencl-benchmarks https://browser.geekbench.com/opencl-benchmarks > but they’re also not really targeting it? That's fine, but it's not an excuse to ignore the power/performance ratio.
    - dagmx 2y ago
      But I’m not ignoring the power/performance ratio? If anything, you are doing that by handwaving away the difference. Give me a comparable system build where the NVIDIA GPU + any CPU of your choice is running at the same wattage as an M2 Ultra, and outperforms it on average. You’d get 150W for the GPU and 150W for the CPU. Again, you can’t really compare the two. They’re inherently different systems unless you only care about singular metrics.