4 ms·
Yes, and it revolves around memory. If you're running code on the CPU, Apple Silicon systems have significantly higher memory bandwidth than most x86 systems -
by duskwuff 3y ago
Yes, and it revolves around memory.
If you're running code on the CPU, Apple Silicon systems have significantly higher memory bandwidth than most x86 systems -- up to 800 GB/sec on M2 Ultra.
If you're running code on the GPU, an Apple Silicon system can be configured with up to 192 GB of unified memory, whereas most discrete graphics cards top out around 16-24 GB of VRAM. (There are a few larger compute-focused cards like Nvidia A100, but they're incredibly expensive.)
- omneity 3y agoThere’s no _single_ consumer or datacenter GPU that is currently breaking 100GB VRAM. In this regard, the Apple offering is pretty unique, but their compute capability is lagging. Also the 800GB/s figure for memory bandwidth is impressive ... for a CPU. GPUs regularly hit 2TB/s or more. TL;DR on Apple for ML: Bigger models, slower calculations.
- 65a 3y agoNot quite, there are 128GiB AMD instinct parts, though the prices are very...enterprise: https://www.amd.com/en/products/server-accelerators/instinct-mi250 https://www.amd.com/en/products/server-accelerators/instinct...
- omneity 3y agoWow that's cool. I didn't know about it, thanks for sharing! Side note, it looks like the AMD MI250 supports FP16, and the raw TFLOPS is ~15% higher than an Nvidia A100 80GB, with a significant advantage for AMD in terms of memory bandwidth. The price range for the two cards seem to roughly overlap. It seems this card is a pretty good deal barring crazy software edge cases. @AMD, if you ever see this, I would be interested to test your card and publish a complete writeup. I'm currently fine-tuning LLMs on A100/H100 and have a good reference point.