9 ms·

Go ahead and ask datacenters why they all use overpriced Nvidia chips for AI training instead of shoving cheaper Mac Studios in there. Their answer might blow

Go ahead and ask datacenters why they all use overpriced Nvidia chips for AI training instead of shoving cheaper Mac Studios in there. Their answer might blow your mind. Spoiler alert: CUDA ecosystem, Linux suport, and most importantly for data centers, Mellanox high speed interfacing with virtually infinite scalability and great virtualization support so they can rent out slices of their HW to customers in exchange for money.

edf13 3y ago
What is the answer?
- throwaway485 3y ago
  I am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.
  - less_less 3y ago
    Maybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.
- zirgs 3y ago
  CUDA.
- sterlind 3y ago
  Compute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090. And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff. tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away. seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.
  - threeseed 3y ago
    There is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores. Either way quite a bit better than 30x slower. [1] https://www.youtube.com/watch?v=k_rmHRKc0JM https://www.youtube.com/watch?v=k_rmHRKc0JM
    - jorgemf 3y ago
      In that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).
  - senko 3y ago
    > seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit. Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference. If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)
    - nightski 3y ago
      You can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.
kkielhofner 3y ago
A 15 year head start in a category they essentially defined plus an entire generation of executives, developers, and users doesn’t hurt either. People complain about the “Nvidia tax” but the hardware is superior (untouchable at datacenter scale) and the “tax” turns into a dividend as soon as your (very expensive) team spends hour after hour (week after week) dealing with issues on other platforms compared to anything based on CUDA often being a Docker pull away with absolutely first class support on any ML framework. Nvidia gets a lot of shade on HN and elsewhere but if you’ve spent any time in this field you completely understand why they have 80-90% market share of GPGPU. With Willow[0] and the Willow Inference Server[1] I'm often asked by users with no experience in the space why we don't target AMD, Coral TPUs (don't even get me started), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives". I’ve been active in the space for roughly half a decade and when I look back to my early days I’m amazed what a beginner like me was able to do because of CUDA. I still routinely am. What you’re able to actually accomplish with a $1000 Nvidia card and a few lines with transformers and/or a Docker container is incredible. That said I am really looking forward to Apple stepping it up here - I’ve given up on AMD ever getting it together on GPGPU and Intel (with Arc) is even further behind. The space needs some real competition somewhere. [0] - https://github.com/toverainc/willow https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server https://github.com/toverainc/willow-inference-server
- mk_stjames 3y ago
  I recently sat and thought about it, and I made the bold claim to a friend that the development of CUDA over the last 15 years.. just that activity, as a whole, is one of the largest use of human intelligence to ever be focused on any one thing in history. It's less people than worked on getting to the Moon, but if you measure the project in PhD-person-years it is probably off the charts. And if you include all the engineers and scientists that the program affects in the same way you would have included all the subcontractors for the Apollo program, it is way, way bigger. I think the only reason CUDA isn't talked about like the monumentally important human milestone in technological development that it is, is that it is a pretty abstract thing that is difficult for laypeople to visualize.
adsfgiodsnrio 3y ago
A "prosumer" desktop is just plain not up to the job. The Mac Studio is comparable to a high-end gaming PC. It is not really a workstation and definitely isn't a server. Its 16 performance cores and 192 GiB of non-ECC memory cannot compete with nearly 200 cores and multiple terabytes. Its GPU compute and memory bandwidth are many times less than what Nvidia can build at the high end. The Studio would run large workload many times slower than a high-end server, if it could run them at all.

edf13 3y ago

What is the answer?

throwaway485 3y ago
I am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.
- less_less 3y ago
  Maybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.
zirgs 3y ago
CUDA.
sterlind 3y ago
Compute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090. And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff. tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away. seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.
- threeseed 3y ago
  There is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores. Either way quite a bit better than 30x slower. [1] https://www.youtube.com/watch?v=k_rmHRKc0JM https://www.youtube.com/watch?v=k_rmHRKc0JM
  - jorgemf 3y ago
    In that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).
- senko 3y ago
  > seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit. Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference. If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)
  - nightski 3y ago
    You can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.

kkielhofner 3y ago

A 15 year head start in a category they essentially defined plus an entire generation of executives, developers, and users doesn’t hurt either. People complain about the “Nvidia tax” but the hardware is superior (untouchable at datacenter scale) and the “tax” turns into a dividend as soon as your (very expensive) team spends hour after hour (week after week) dealing with issues on other platforms compared to anything based on CUDA often being a Docker pull away with absolutely first class support on any ML framework. Nvidia gets a lot of shade on HN and elsewhere but if you’ve spent any time in this field you completely understand why they have 80-90% market share of GPGPU. With Willow[0] and the Willow Inference Server[1] I'm often asked by users with no experience in the space why we don't target AMD, Coral TPUs (don't even get me started), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives". I’ve been active in the space for roughly half a decade and when I look back to my early days I’m amazed what a beginner like me was able to do because of CUDA. I still routinely am. What you’re able to actually accomplish with a $1000 Nvidia card and a few lines with transformers and/or a Docker container is incredible. That said I am really looking forward to Apple stepping it up here - I’ve given up on AMD ever getting it together on GPGPU and Intel (with Arc) is even further behind. The space needs some real competition somewhere. [0] - https://github.com/toverainc/willow https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server https://github.com/toverainc/willow-inference-server

mk_stjames 3y ago
I recently sat and thought about it, and I made the bold claim to a friend that the development of CUDA over the last 15 years.. just that activity, as a whole, is one of the largest use of human intelligence to ever be focused on any one thing in history. It's less people than worked on getting to the Moon, but if you measure the project in PhD-person-years it is probably off the charts. And if you include all the engineers and scientists that the program affects in the same way you would have included all the subcontractors for the Apollo program, it is way, way bigger. I think the only reason CUDA isn't talked about like the monumentally important human milestone in technological development that it is, is that it is a pretty abstract thing that is difficult for laypeople to visualize.

adsfgiodsnrio 3y ago

A "prosumer" desktop is just plain not up to the job. The Mac Studio is comparable to a high-end gaming PC. It is not really a workstation and definitely isn't a server. Its 16 performance cores and 192 GiB of non-ECC memory cannot compete with nearly 200 cores and multiple terabytes. Its GPU compute and memory bandwidth are many times less than what Nvidia can build at the high end. The Studio would run large workload many times slower than a high-end server, if it could run them at all.