3 ms·
Apple sells a Mac Studio with the M3 Ultra chip and 512GB VRAM (unified memory between CPU and GPU). It costs $9,500. Their secret is that the memory is manufa
by pavlov 2y ago
Apple sells a Mac Studio with the M3 Ultra chip and 512GB VRAM (unified memory between CPU and GPU). It costs $9,500.
Their secret is that the memory is manufactured within the chip package.
- codedokode 2y agoWhy NVIDIA cannot manufacture 512 Gb chips and put 16 of them on the board?
- jsheard 2y agoApples architecture comes with its own trade-offs, it gives them huge capacity and pretty good bandwidth, but not nearly as much as Nvidia's architectures have. The M3 Ultra is 800GB/sec, the RTX 5090 is 1.8TB/sec, and the H200 is 4.8TB/s(!). Huge capacity with middling bandwidth is in vogue because it's a good fit for AI inference, but AI training and most other applications of GPUs need as much bandwidth as they can get.
- deleted 2y ago[deleted]
- codedokode 2y agoWell, if you have 16 M3-equivalent chips you can multiply the bandwidth by 16, right? Also, as I understand, ML is basically matrix multiplication and it has O(N³) operations on O(N²) numbers, so bandwidth might be not as important as number of ALUs.
- ein0p 2y agoIt's actually not within the chip's package. It's soldered to the board. It's just regular, fairly high spec LPDDR5X IIRC, there are just a TON of memory channels.
- pavlov 2y agoIt's not in the package? TIL... My misunderstanding seems to be common across the interwebs. I remember Apple used to show slides depicting the M1 SoC as one unit containing a CPU, GPU, Neural Engine, cache, and DRAM all together. But slides shown at an Apple event definitely qualify for artistic license.
- jsheard 2y agoThe memory is soldered on top of the package, here's an actual real life photo of the M1 package with two LPDDR packages sitting on it: https://en.wikipedia.org/wiki/Apple_M1#/media/File:Mac_Mini_M1_chip.jpg https://en.wikipedia.org/wiki/Apple_M1#/media/File:Mac_Mini_... You could call that on-package, but it's not on-package in the same way that GPU HBM is, with that the main die and memory dies are packaged together on the same substrate. That's a much more difficult and expensive process, apparently packaging is the main bottleneck for H100/H200 production. https://i.imgur.com/hFTPyjk.jpeg https://i.imgur.com/hFTPyjk.jpeg
- ein0p 2y agoNor is it a chiplet, in which case it would qualify as "in-SoC".
- nomel 2y agoFor that H100 image, how do they get the top surface of all the chips on the same plane? Do they solder them "upside down", so the chips are sitting on a reference plane, and the PCB is allowed to "float"? Or is the soldering controlled enough that they can just heat and be done?
- TiredOfLife 2y agoWait till you learn that unified memory was in common use by pc's long before Apple "invented" it.
- xxs 2y agoThe statement is correct; it's not on the substrate similar to AMD's 3D cache or it doesn't use interposer like HBM. You can consider it like a small PCB that has the CPU die and the memory soldered very closely nearby (and like mentioned +memory channels)
- jayd16 2y agoLPDDR5X ~550GB/s vs GDDR7 which is ~1.8 TB/s
- angoragoats 2y agoThis doesn’t really answer the parent’s question. That the memory is on a PCB close to the CPU/GPU certainly helps with signal integrity, but it is not by any means relevant here. The Apple platform has high memory bandwidth compared to x86 PCs because the CPU has a wide memory bus. You can get similar memory bandwidth out of high-end Epyc and Xeon CPUs which use standard DIMMs but with many more memory channels than a regular desktop computer.