3 ms·
The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of me
by sofixa 2y ago
The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of memory, and the current gen processor, M4, doesn't even have an Ultra, while the Max is only available in a laptop form factor (so thermal constraints).
An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s.
https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167
- jmyeet 2y agoThat's a good point. I checked the M2 Mac Studio and it's 400GB/s for the M2 Max and 800GB/s for the M2 Ultra so the M4 Ultra when we get it later this year should really be a beast. Oh and the top end Macbook Pro 16 (the only current Mac with an M4 Max) has 410GB/s memory bandwidth. Obviously the Mac Studio is at a much higher price point. Still, you need to spend $1500+ to get an NVidia GPU with >12GB of RAM. Multiple of those starts adding up quick. Put multiple in the same box and you're talking more expensive case, PSU, mainboard, etc and cooling too. Apple has a really interesting opportunity here with their unified memory architecture and power efficiency.