6 ms·
The bottleneck is mainly memory bandwidth. AMD EPYC hw is appealing for local inference because it has a higher memory bandwidth than desktop gear (because 8-12
by throw-qqqqq 2y ago
The bottleneck is mainly memory bandwidth. AMD EPYC hw is appealing for local inference because it has a higher memory bandwidth than desktop gear (because 8-12 memory channels vs 2 on almost everything else), but not as fast as the Apple architectures and nowhere near VRAM speeds. If you want to drastically exceed ~3-5 tokens/s on 70b-q4 models, you usually still need GPUs.
- timschmidt 2y agoOn Zen5 you also get AVX512 which llamafile takes advantage of for drastically improved speeds during prompt processing, at least. And the 12 channel Epycs actually seem to have more memory bandwidth available than the Apple M series. Especially considering it's all available to the CPU as opposed to just some portion of it.
- Gracana 2y agoMaybe EPYC can make better use of the available bandwidth, but for comparison I have a water cooled Xeon W5-3435X running at 4.7GHz all-core with 8 channels of DDR5-6400, and CPU inference is still dog slow. With a 70B Q8 model I get 1 tok/s, which is a lot less than I thought I would get with 410GB/s max RAM bandwidth. If I run on 5x A4000s I get 6.1 tok/s, which makes sense... 448GB/s / 70GB = 6.4 tok/s max.
- iamnotagenius 2y agovery strange as I get on old i5-12400+DDR4 2 tok/sec with 14B/q8 model.
- magicalhippo 2y agoThis was beautifully illustrated in the recent Phoronix 5090 LLM benchmark[1], which I noted here[2]. The tested GPUs had an almost perfect linear relationship between generated token/s and GB/s memory bandwidth, except the 5090 where it dipped slightly. I guess the 5090 either started ever so slightly to become compute limited as well, or hit some overhead limitation. [1]: https://www.phoronix.com/review/nvidia-rtx5090-llama-cpp https://www.phoronix.com/review/nvidia-rtx5090-llama-cpp [2]: https://news.ycombinator.com/item?id=42847284 https://news.ycombinator.com/item?id=42847284
- jmb99 2y agoIt’s more expensive, but Zen4 Threadripper Pro is probably the way to go on that front. 8 memory channels, with DIMMs available up to DDR5-7200 for 8x32GB (256GB), or DDR5-6800 for 8x48GB (384GB). It’ll set you back ~$3k for the RAM and ~$6k for a CPU with 8 CCDs (the 7985WX, at least), and then ~$1k for motherboard and however much you want to spend on NVME. Basically ~$10k for a 384GB DDR5 system with ~435GB/s actual bandwidth. Not quite as fast as the 192GB Apple machines, but twice as much memory and more compute for “only” a few thousand more.
- sourcecodeplz 2y agoAt these prices, I would just get 2xDigits for $6k and have 256gb.
- jadbox 2y agoI have a feeling that Digits will probably get sold out and will pricing will get hiked WAY up.
- riku_iki 2y agois it confirmed that you can get 256gb of vram for that amount? Because my understanding is that digits pricing will start at $3k for some basic config.
- fluoridation 2y agoWhat they meant is buying two whole separate computers.
- riku_iki 2y agoI understand. It is still unclear if you can get 128GB vram for $3k.
- fluoridation 2y agoWell, I mean, the press release is pretty unambiguous. >Each Project DIGITS features 128GB of unified, coherent memory and up to 4TB of NVMe storage. Even if $3k is only the starting price, it doesn't sound like spending more buys you more memory.