3 ms·

What is the answer?

by edf13 3y ago

What is the answer?

throwaway485 3y ago
I am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.
- less_less 3y ago
  Maybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.
zirgs 3y ago
CUDA.
sterlind 3y ago
Compute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090. And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff. tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away. seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.
- threeseed 3y ago
  There is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores. Either way quite a bit better than 30x slower. [1] https://www.youtube.com/watch?v=k_rmHRKc0JM https://www.youtube.com/watch?v=k_rmHRKc0JM
  - jorgemf 3y ago
    In that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).
- senko 3y ago
  > seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit. Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference. If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)
  - nightski 3y ago
    You can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.
    - senko 3y ago
      I'm not sure if it's cheaper when you compare the whole machine price, not just GPU, since you can't buy piecewise with the Mac.
      - nightski 3y ago
        The GPU is $4500. That gives you $2000 to build a machine (which you can build a top of the line consumer PC for cheaper than that, I just did recently w/128GB DDR5 and Zen 4 7950X3D).

What is the answer?

by edf13 3y ago

What is the answer?

I am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.

less_less 3y ago
Maybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.

zirgs 3y ago

CUDA.

sterlind 3y ago

Compute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090. And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff. tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away. seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.

threeseed 3y ago
There is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores. Either way quite a bit better than 30x slower. [1] https://www.youtube.com/watch?v=k_rmHRKc0JM https://www.youtube.com/watch?v=k_rmHRKc0JM
- jorgemf 3y ago
  In that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).
senko 3y ago
> seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit. Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference. If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)
- nightski 3y ago
  You can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.
  - senko 3y ago
    I'm not sure if it's cheaper when you compare the whole machine price, not just GPU, since you can't buy piecewise with the Mac.
    - nightski 3y ago
      The GPU is $4500. That gives you $2000 to build a machine (which you can build a top of the line consumer PC for cheaper than that, I just did recently w/128GB DDR5 and Zen 4 7950X3D).