6 ms·
Why NVIDIA cannot manufacture 512 Gb chips and put 16 of them on the board?
by codedokode 2y ago
Why NVIDIA cannot manufacture 512 Gb chips and put 16 of them on the board?
- jsheard 2y agoApples architecture comes with its own trade-offs, it gives them huge capacity and pretty good bandwidth, but not nearly as much as Nvidia's architectures have. The M3 Ultra is 800GB/sec, the RTX 5090 is 1.8TB/sec, and the H200 is 4.8TB/s(!). Huge capacity with middling bandwidth is in vogue because it's a good fit for AI inference, but AI training and most other applications of GPUs need as much bandwidth as they can get.
- deleted 2y ago[deleted]
- codedokode 2y agoWell, if you have 16 M3-equivalent chips you can multiply the bandwidth by 16, right? Also, as I understand, ML is basically matrix multiplication and it has O(N³) operations on O(N²) numbers, so bandwidth might be not as important as number of ALUs.