4 ms·
Up to 128 GB RAM available for running AI models. Theoretically awesome, but this might have some interesting market consequences for everyone else.
by Kim_Bruning 2y ago
Up to 128 GB RAM available for running AI models.
Theoretically awesome, but this might have some interesting market consequences for everyone else.
- dragonwriter 2y ago$2,0000 to $3,000 desktop devices with 128GB of shared GPU/CPU RAM seems to be a segment that is seeing lots of announcements from lots of vendors.
- alwayslikethis 2y agoThe bandwidth still doesn't quite compare to a GPU, and 128GB doesn't fit DeepSeek R1, however. If they bump it to 512GB for $5000 or so, that will disrupt the market.
- ricardobeat 2y ago256GB/s is on par with the M4 Pro at a much lower price. You could run R1 671B using unsloth’s quantized version that fits in <80GB. Not sure why that would be a benchmark though, there’s nothing that can run the model at full precision right now except for (very slow) server hardware.
- deleted 2y ago[deleted]
- ant6n 2y agoBut you can use an AMD EPYC cpu and get 460GB/s bandwidth. Could probably get 512GB RAM for a lot cheaper than 5K.
- josephg 2y agoYou’ve gotta deal with amd’s AI software stack though. How is that these days? I assume cuda is still king?
- rglullis 2y agoI've been running ollama on an XTX7900 (AMD GPU with 24GB of RAM) with any model that fits in it, and absolutely no issues there.
- ac29 2y ago> You’ve gotta deal with amd’s AI software stack though Not if you are using the CPU. I am under the impression most inference use cases are memory bandwidth limited, not compute limited, so running on the GPU would gain you little to nothing unless the GPU has faster access to the shared memory.
- Havoc 2y agoInference fine, training less so
- ein0p 2y agoWith what memory bandwidth? Remember, without e.g. speculative decoding you need to read the entire model and KV cache for every token. Let's be extremely generous and say you get 512GB/sec in memory bandwidth, on par with a high end M4 MacBook Pro. This means you can only read the entire DRAM 4 times a second, generating at most 4 tokens per second. Smaller models will of course run proportionally faster, but 128GB isn't by itself a sufficient statistic to say whether this is "theoretically awesome" or not.
- jsheard 2y ago> With what memory bandwidth? 256GB/sec, so roughly M4 Pro throughput.
- ein0p 2y agoWhat's frustrating is there's no real reason why regular DDR5 can't reach 1TB/sec with a sufficient number of channels. The manufacturers are just holding it back to drip feed that memory bandwidth over several generations. Except for Apple, which lets you have 800GB/sec now, and will let you have 1TB/sec+ in M5 Ultra next year. It's $$$$, but still - the true alternatives are much less cost effective.
- sliken 2y agoRight, like the mac studio ultra. Or a dual socket AMD Turin. Or a grace+hopper (assuming you offload to the hopper). Latency is dictated by the laws of physics, more bandwidth is easy, but not cheap.
- ein0p 2y agoIt can be relatively cheap too under the constraints imposed by typical AI workloads, at least when it comes to getting to a 1TB/s or so. All you need is high-spec DDR5 and _a ton_ of memory channels in your SOC. During transformer inference you will easily be able to use those parallel, multichannel reads. I get why you'd need HBM and several TB/s of memory bandwidth for extremely memory intensive training workloads. But for inference 1TB/s gives you a lot to work with (especially if your model is a MoE), and it doesn't have to be ultra expensive.