3 ms·

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the

by jmyeet 2y ago

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac Mini: 24GB of RAM. Stocked in store. 3. Add RAM to either of the above up to 32GB. It's not cheap at $200/8GB but you can buy a Mac Mini with 32GB of shared RAM for $999, substantially cheaper than the author's PC build but less storage (although you can upgrade that too). 4. M4 Pro: $1399 w/ 24GB of RAM. Stocked in store. You can customize this all the way to 64GB of RAM for +$600 so $1999 in total. That is amazing value for this kind of workload. 5. The Mac Studio is really the ultimate option. Way more cores and you can go all the way to 192GB of unified memory (for a $6000 machine). The problem here is that the Mac Studio is old, still on the M2 architecture. An M4 Ultra update is expected sometime this year, possibly late this year. 6. You can get into clustering these (eg [1]). 7. There are various Macbook Pro options, the highest of which is a 16" Mackbook Pro with 128GB of unified memory for $4999. But the main takeaway is the M4 Mac Mini is fantastic value. Some more random thoughts: - Some Mac Minis have Thunderbolt 5 ("TB5"), which is up to either 80Gbps or 120Gbps bidirectional (I've seen it quoted as both); - Mac Minis have the option of 10GbE (+$200); - The Mac Mini has 2 USB3 ports and either 3 TB4 or 3 TB5 ports. [1]: https://blog.exolabs.net/day-2/ https://blog.exolabs.net/day-2/

sofixa 2y ago
The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of memory, and the current gen processor, M4, doesn't even have an Ultra, while the Max is only available in a laptop form factor (so thermal constraints). An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167
- jmyeet 2y ago
  That's a good point. I checked the M2 Mac Studio and it's 400GB/s for the M2 Max and 800GB/s for the M2 Ultra so the M4 Ultra when we get it later this year should really be a beast. Oh and the top end Macbook Pro 16 (the only current Mac with an M4 Max) has 410GB/s memory bandwidth. Obviously the Mac Studio is at a much higher price point. Still, you need to spend $1500+ to get an NVidia GPU with >12GB of RAM. Multiple of those starts adding up quick. Put multiple in the same box and you're talking more expensive case, PSU, mainboard, etc and cooling too. Apple has a really interesting opportunity here with their unified memory architecture and power efficiency.
diggan 2y ago
How is the performance difference between using a dedicated GPU from Nvidia for example compared to whatever Apple does? So lets say we'd run a model on a Mac Mini M4 with 24GB RAM, how many tokens/s are you getting? Then if we run the exact same model but with a RTX 3090ti for example, how many tokens/s are you getting? Do these comparisons exist somewhere online already? I understand it's possible to run the model on Apple hardware today, with the unified memory, but how fast is that really?
- redman25 2y ago
  Not the exact same comparison but I have an M1 mac with 16gb ram and can get about 10 t/s with a 3B model. The same model on my 3060ti gets more than 100 t/s. Needless to say, ram isn't everything.
  - diggan 2y ago
    Could you say what exact model+quant you're using for that specific test + settings + runtime? Just so I could try to compare with other numbers I come across.
iamleppert 2y ago
The hassle of not being able to work with native CUDA isn't worth it for a huge amount of AI. Good luck getting that latest paper or code working quickly just to try it out, if the author didn't explicitly target M4 (unlikely but all the most mainstream of stuff).
- darkwater 2y ago
  In a homelab scenario, having your own AI assistant not ran by someone else, that is not an issue. If you want to tinker/learn AI it's definitely an issue.
sethd 2y ago
For sure and the Mac Mini M4 Pro with 64GB of RAM feels like the sweet spot right now. That said, the base storage option is only 512GB, and if this machine is also a daily driver, you’re going to want to bump that up a bit. Still, it’s an amazing machine for under $3K.
- wolfhumble 2y ago
  It would be better/cheaper to buy an external Thunderbolt 5 enclosure for the NVME drive you need.
  - sethd 2y ago
    I looked into this a couple months ago and external TB5 was still more expensive at 1-2 TB not sure about above, though.
    - wolfhumble 2y ago
      Going from 500GB to 2TB built-in is €600 in the US at the moment. Samsung 990PRO 2TB is $170 and Acasis T5 80Gbps is €300. So it makes sense to buy external for ≥ 2TB, more flexible as well :-) For 1TB it makes more sense to buy built-in as you note above.
      - sethd 2y ago
        It also depends on the quality of that enclosure and whether or not it adds heat or fan noise.
atwrk 2y ago
Worth pointing out that you "only" get <= 270GB/s of memory bandwith with those Macs, unless you choose the max/ultra models. If that is enough for your use case, it may make sense to wait 2 months and get a Ryzen AI Max+ 395 APU, which will have the same memory bandwith, but allows for up to 128GB RAM. For probably ~half the Mac's price. Usual AMD driver disclaimer applies, but then again inference is most often way easier to get running than training.
oofbaroomf 2y ago
Unified memory is great because it's fast, but you can also get a lot of system memory on a "conventional" machine like OP's, and offload MOE layers like what Ktransformers did, so you can run huge models with acceptable speeds. While the Mac mini may have better value for anything that fits in the unified memory, if you want to run Deepseek R1 or other large models, then it's best to max out system RAM and get a GPU to offload.

by jmyeet 2y ago

sofixa 2y ago

The issue with Macs is that below Max/Ultra processors, the memory bandwidth is pretty slow. So you need to spend a lot on a high level processor and lots of memory, and the current gen processor, M4, doesn't even have an Ultra, while the Max is only available in a laptop form factor (so thermal constraints). An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167

jmyeet 2y ago
That's a good point. I checked the M2 Mac Studio and it's 400GB/s for the M2 Max and 800GB/s for the M2 Ultra so the M4 Ultra when we get it later this year should really be a beast. Oh and the top end Macbook Pro 16 (the only current Mac with an M4 Max) has 410GB/s memory bandwidth. Obviously the Mac Studio is at a much higher price point. Still, you need to spend $1500+ to get an NVidia GPU with >12GB of RAM. Multiple of those starts adding up quick. Put multiple in the same box and you're talking more expensive case, PSU, mainboard, etc and cooling too. Apple has a really interesting opportunity here with their unified memory architecture and power efficiency.

diggan 2y ago

How is the performance difference between using a dedicated GPU from Nvidia for example compared to whatever Apple does? So lets say we'd run a model on a Mac Mini M4 with 24GB RAM, how many tokens/s are you getting? Then if we run the exact same model but with a RTX 3090ti for example, how many tokens/s are you getting? Do these comparisons exist somewhere online already? I understand it's possible to run the model on Apple hardware today, with the unified memory, but how fast is that really?

redman25 2y ago
Not the exact same comparison but I have an M1 mac with 16gb ram and can get about 10 t/s with a 3B model. The same model on my 3060ti gets more than 100 t/s. Needless to say, ram isn't everything.
- diggan 2y ago
  Could you say what exact model+quant you're using for that specific test + settings + runtime? Just so I could try to compare with other numbers I come across.

iamleppert 2y ago

The hassle of not being able to work with native CUDA isn't worth it for a huge amount of AI. Good luck getting that latest paper or code working quickly just to try it out, if the author didn't explicitly target M4 (unlikely but all the most mainstream of stuff).

darkwater 2y ago
In a homelab scenario, having your own AI assistant not ran by someone else, that is not an issue. If you want to tinker/learn AI it's definitely an issue.

sethd 2y ago

For sure and the Mac Mini M4 Pro with 64GB of RAM feels like the sweet spot right now. That said, the base storage option is only 512GB, and if this machine is also a daily driver, you’re going to want to bump that up a bit. Still, it’s an amazing machine for under $3K.

wolfhumble 2y ago
It would be better/cheaper to buy an external Thunderbolt 5 enclosure for the NVME drive you need.
- sethd 2y ago
  I looked into this a couple months ago and external TB5 was still more expensive at 1-2 TB not sure about above, though.
  - wolfhumble 2y ago
    Going from 500GB to 2TB built-in is €600 in the US at the moment. Samsung 990PRO 2TB is $170 and Acasis T5 80Gbps is €300. So it makes sense to buy external for ≥ 2TB, more flexible as well :-) For 1TB it makes more sense to buy built-in as you note above.
    - sethd 2y ago
      It also depends on the quality of that enclosure and whether or not it adds heat or fan noise.

atwrk 2y ago

Worth pointing out that you "only" get <= 270GB/s of memory bandwith with those Macs, unless you choose the max/ultra models. If that is enough for your use case, it may make sense to wait 2 months and get a Ryzen AI Max+ 395 APU, which will have the same memory bandwith, but allows for up to 128GB RAM. For probably ~half the Mac's price. Usual AMD driver disclaimer applies, but then again inference is most often way easier to get running than training.

oofbaroomf 2y ago

Unified memory is great because it's fast, but you can also get a lot of system memory on a "conventional" machine like OP's, and offload MOE layers like what Ktransformers did, so you can run huge models with acceptable speeds. While the Mac mini may have better value for anything that fits in the unified memory, if you want to run Deepseek R1 or other large models, then it's best to max out system RAM and get a GPU to offload.