6 ms·
Hm, that's a heft price, 200K per card. Mac Studio M2 ULTRA has 192Gb of RAM, potentially 188Gb available for GPU, for 5K. Wouldn't apple be able to compete w
by tmikaeld 3y ago
Hm, that's a heft price, 200K per card.
Mac Studio M2 ULTRA has 192Gb of RAM, potentially 188Gb available for GPU, for 5K.
Wouldn't apple be able to compete with that if they scaled it up?
- ChuckNorris89 3y agoGo ahead and ask datacenters why they all use overpriced Nvidia chips for AI training instead of shoving cheaper Mac Studios in there. Their answer might blow your mind. Spoiler alert: CUDA ecosystem, Linux suport, and most importantly for data centers, Mellanox high speed interfacing with virtually infinite scalability and great virtualization support so they can rent out slices of their HW to customers in exchange for money.
- edf13 3y agoWhat is the answer?
- throwaway485 3y agoI am speculating the answer is that "Nvidia just works", where Apple may be more niche & hassle to get working with their preferred frameworks/stacks/tools.
- less_less 3y agoMaybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops). By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2 https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU. So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.
- zirgs 3y agoCUDA.
- sterlind 3y agoCompute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090. And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff. tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away. seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.
- threeseed 3y agoThere is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores. Either way quite a bit better than 30x slower. [1] https://www.youtube.com/watch?v=k_rmHRKc0JM https://www.youtube.com/watch?v=k_rmHRKc0JM
- jorgemf 3y agoIn that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).
- senko 3y ago> seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit. Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference. If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)
- nightski 3y agoYou can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.
- kkielhofner 3y agoA 15 year head start in a category they essentially defined plus an entire generation of executives, developers, and users doesn’t hurt either. People complain about the “Nvidia tax” but the hardware is superior (untouchable at datacenter scale) and the “tax” turns into a dividend as soon as your (very expensive) team spends hour after hour (week after week) dealing with issues on other platforms compared to anything based on CUDA often being a Docker pull away with absolutely first class support on any ML framework. Nvidia gets a lot of shade on HN and elsewhere but if you’ve spent any time in this field you completely understand why they have 80-90% market share of GPGPU. With Willow[0] and the Willow Inference Server[1] I'm often asked by users with no experience in the space why we don't target AMD, Coral TPUs (don't even get me started), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives". I’ve been active in the space for roughly half a decade and when I look back to my early days I’m amazed what a beginner like me was able to do because of CUDA. I still routinely am. What you’re able to actually accomplish with a $1000 Nvidia card and a few lines with transformers and/or a Docker container is incredible. That said I am really looking forward to Apple stepping it up here - I’ve given up on AMD ever getting it together on GPGPU and Intel (with Arc) is even further behind. The space needs some real competition somewhere. [0] - https://github.com/toverainc/willow https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server https://github.com/toverainc/willow-inference-server
- mk_stjames 3y agoI recently sat and thought about it, and I made the bold claim to a friend that the development of CUDA over the last 15 years.. just that activity, as a whole, is one of the largest use of human intelligence to ever be focused on any one thing in history. It's less people than worked on getting to the Moon, but if you measure the project in PhD-person-years it is probably off the charts. And if you include all the engineers and scientists that the program affects in the same way you would have included all the subcontractors for the Apollo program, it is way, way bigger. I think the only reason CUDA isn't talked about like the monumentally important human milestone in technological development that it is, is that it is a pretty abstract thing that is difficult for laypeople to visualize.
- adsfgiodsnrio 3y agoA "prosumer" desktop is just plain not up to the job. The Mac Studio is comparable to a high-end gaming PC. It is not really a workstation and definitely isn't a server. Its 16 performance cores and 192 GiB of non-ECC memory cannot compete with nearly 200 cores and multiple terabytes. Its GPU compute and memory bandwidth are many times less than what Nvidia can build at the high end. The Studio would run large workload many times slower than a high-end server, if it could run them at all.
- PragmaticPulp 3y agoThe compute capacity of these nVidia parts is significantly higher than that of an M2 Ultra. It’s not just about memory capacity. The memory bandwidth of the nVidia GPUs is also significantly higher than the M2 parts. The Apple silicone parts are impressive for what they are, but they don’t have a huge efficiency advantage for GPU compute. The full-size GPUs with huge memory buses and a large number of cores are still significantly more powerful. There’s also a matter of getting data into and out of the GPUs and across the network, which takes a lot more than 10Gbe The Apple silicon is great for running development workloads locally, but it’s significantly slower than full size GPUs. I’m kind of surprised at how quickly everyone forgot that Apple’s marketing material greatly exaggerates their GPU performance.
- deleted 3y ago[deleted]
- anaganisk 3y ago*Apple Silicon, I'm no grammar Nazi, it made me chuckle.
- mrguyorama 3y agoCan you imagine how much Apple would charge for branded boobs?
- anaganisk 3y agoBut for sure they would be impressive for what they are:p
- delfinom 3y agoMemory is the cheap part at the end of the day. It's about the rest of the chip and why GPUs are fundamentally and radically different from a CPU.
- vlovich123 3y agoMemory is rapidly becoming the expensive part compared with compute and memory doesn’t get as cheap as quick as CPU. Notice how main memory price/TiB has basically flat lined for the past 10 years: https://ourworldindata.org/grapher/historical-cost-of-computer-memory-and-storage https://ourworldindata.org/grapher/historical-cost-of-comput... Sure, they’ve gotten a bit faster but it’s still fairly expensive to outfit more and more RAM. GPUs though are indeed much more than memory. However, Apple has a unique unified memory model that no one else has matched yet where the memory is the heart of the machine. That means you don’t even need as much bandwidth because you can transparently access the same data by any chip at the same speed. That’s a pretty powerful design. I doubt Apple will really go into the training side of things because that’s not germane to their use cases because that happens in the cloud where they don’t have a presence yet. Inference is so expect more LLM / stablediffusion acceleration. Now if fine tuning models with additional training becomes a thing, then you’ll see acceleration of that modality on Apple’s machines. But training won’t be a focus because Apple doesn’t care about fickle nerd points that aren’t relevant to their business.
- gmm1990 3y agoProbably apple are the only ones doing the newer tsmc process too
- giobox 3y agoWhile Apple likely gets the first production batches, there are several major vendors lined up as customers for TSMC's latest N3 (used on M2 etc) process: > https://www.guru3d.com/news-story/tsmc-has-secured-seven-major-customers-for-its-upcoming-3-nm-node.html https://www.guru3d.com/news-story/tsmc-has-secured-seven-maj... There will very likely be products from Intel, Qualcomm, Nvidia and AMD stuff using it too this year.
- Synaesthesia 3y agoApple could also market their fast efficient chip to datacentres and servers if they were so inclined. They choose not to compete in certain areas, focusing on consumer products. I don't know why but it's their choice.
- wahahah 3y ago192GB (1,536Gb)
- jakjak123 3y agoIn datacenters, sparse rack space is rather expensive. Plus the compute per watt of these nvidia chips are off the scale compared to anyone else.