5 ms·

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likel

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

diggan 2y ago
> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.
- cruffle_duffle 2y ago
  I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to establish “the best practice hardware configuration for training / inferencing large language models locally”. Some day there will be established guides with solid. In fact someday there will be be PC’s that specifically target LLMs and will feature all kinds of stats aimed at getting you to bust out your wallet. And I even predict they’ll come up with metrics that all the players will chase well beyond when those metrics make sense (megapixels, clock frequency, etc)… but we aren’t there yet!
  - diggan 2y ago
    Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.
    - kamranjon 2y ago
      A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/llama-bench https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc. It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even basic training.
      - diggan 2y ago
        I guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)
        t1amat 2y ago
        In fairness it’s become even more difficult now than ever before. * hardware spec * inference engine * specific model - differences to tokenizer will make models faster/slower with equivalent parameter count * quantization used - and you need to be aware of hardware specific optimizations for particular quants * kv cache settings * input context size * output token count This is probably not a complete list either.
    - nickthegreek 2y ago
      Best place to get that kinda info is gonna be /r/LocalLlama
  - motorest 2y ago
    > I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. What's hard about it? You get the hardware, you run the software, you take measurements.
    - GTP 2y ago
      Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.
      - diggan 2y ago
        > but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere
        unshavedyak 2y ago
        I don't think they're saying anything counter to that. The people who don't require the volume of data will run these. Ie the super early adopters.
        colonCapitalDee 2y ago
        We've already started, we just haven't finished yet
- fkyoureadthedoc 2y ago
  On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit
  - diggan 2y ago
    > gemma2:27b What quantization are you using? What's the runtime+version you run this with? And the rest of the settings? Edit: Turns out parent is using Q4 for their test. Doing the same test with LM Studio and a 3090ti + Ryzen 5950X (with 44 layers on GPU, 2 on CPU) I get ~15 tokens/second.
    - fkyoureadthedoc 2y ago
      Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"
      - diggan 2y ago
        When you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.
        fkyoureadthedoc 2y ago
        not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b https://pastes.io/ollama-run-gemma2-27b
- un_ess 2y ago
  Per the screenshot, this is a DeepSeek running on a 192GB M2 Studio https://nitter.poast.org/ggerganov/status/1884612770093842728#m https://nitter.poast.org/ggerganov/status/188461277009384272... The same on Nvidia (various models) https://github.com/ggerganov/llama.cpp/issues/11474 https://github.com/ggerganov/llama.cpp/issues/11474 [1] this is a the model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_S https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
  - diggan 2y ago
    So Apple M2 Studio does ~15 tks/second and A100-SXM4-80GB does 9 tks/second? I'm not sure I'm reading the results wrong or missing some vital context, but that sounds unlikely to me.
    - achierius 2y ago
      The studio has a lot more ram available to the GPU (up to 192gb) than the a100 (80gb), and iirc at least comparable memory bandwidth -- those are what matter when you're doing LLM inference, so the studio tends to win out there. Where the a100 and other similar chips dominate is in training &c, which is mostly a question of flops.
      - diggan 2y ago
        > and iirc at least comparable memory bandwidth I don't think they do. From Wikipedia: > the M2 Pro, M2 Max, and M2 Ultra have approximately 200 GB/s, 400 GB/s, and 800 GB/s respectively From techpowerup: > NVIDIA A100 SXM4 80 GB - Memory bandwidth - 2.04 TB/s Seems to be a magnitude of difference, and that's just the bandwidth.
- deleted 2y ago
  [deleted]
- vladgur 2y ago
  as someone who is paying $0.50 per kwh, id also like to include kw per 1000 tokens or something to give me a sense of cost of ownership these local systems
  - troyvit 2y ago
    That would be an awesome thing across the industry -- even for the big commercial models -- for those who care not only about price but also carbon footprint.
ekianjo 2y ago
Mac Mini will be very slow for context ingestion compared to nvidia GPU, and the other issue is that they are not usable for Stable Diffusion... So if you just want to use LLMs, maybe, but if you have other interests in AI models, probably not the right answer.
- drcongo 2y ago
  I use a Mac Studio for Stable Diffusion, what's special about the Mac Mini that means it won't work?
  - vunderba 2y ago
    What models are you using? Stable diffusion 1.5, SDXL, or flux? I've heard that Macs are pretty slow with XL and borderline unusable for flux requiring minutes at a time to generate a single image - whereas an RTX4090 can generate a 1024x1024 image with the higher quality Flux Dev model (not schnell) in 14 seconds. OP is probably correct that if you want to branch out of just strictly LLM's, cuda is the way to go. I've never heard of anyone getting LTX or hunyuan running on a Mac for example.
    - drcongo 2y ago
      I've used 1.5 through 3.5, XL has been kinda fine for me but tiny adjustments can take it from good to terrible. Good point about Flux though, that's awful on the Mac.
JKCalhoun 2y ago
For this use case though, I would prefer something more modular than Apple hardware — where down the road I could upgrade the GPUs, for example.
motorest 2y ago
> For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. Around half that price tag was attributed to the blogger reusing an old workstation he had lying around. Beyond this point, OP slapped two graphics cards into an old rig. A better description would be something like "what buying two graphics cards gets you in terms of AI".
- Capricorn2481 2y ago
  > Beyond this point, OP slapped two graphics cards into an old rig Meaning what? This is largely what you do on a budget since RAM is such a difference maker in token generation. This is what's recommended. OP could buy an a100, but that wouldn't be a budget build.
oofbaroomf 2y ago
The bottleneck for single batch inference is memory bandwidth. The M4 Pro has less memory bandwidth than the P40, so it would be slower. Also, the setup presented in the OP has system RAM, allowing you to run models than what fits in 48GB of VRAM (and with good speeds too if you offload with something like ktransformers).
- anthonyskipper 2y ago
  >>M4 Pro has less memory bandwidth than the P40, so it would be slower Why do you say this? I thought the p40 only had a memory bandwidth of 346 Gbytes/sec. The m4 is 546 GB/s. So the macbook should kick the crap out of the p40.
  - oofbaroomf 2y ago
    The M4 Max has up to 546 GB/s. The M4 Pro, what GP was talking about, has only 273 GB/s. An M4 Max with that much RAM would most likely exceed OP's budget.
UncleOxidant 2y ago
I wish Apple would offer a 128GB option in the Mac Mini - That would require an M4 Max which they don't offer in the mini. I know they have a MBP with M4 Max and 128GB, but I don't need another laptop.
- kridsdale1 2y ago
  I’m waiting until this summer with the M4 Ultra Studio.
  - UncleOxidant 2y ago
    Which will likely be over five grand for 128GB.

diggan 2y ago

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

cruffle_duffle 2y ago
I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to establish “the best practice hardware configuration for training / inferencing large language models locally”. Some day there will be established guides with solid. In fact someday there will be be PC’s that specifically target LLMs and will feature all kinds of stats aimed at getting you to bust out your wallet. And I even predict they’ll come up with metrics that all the players will chase well beyond when those metrics make sense (megapixels, clock frequency, etc)… but we aren’t there yet!
- diggan 2y ago
  Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.
  - kamranjon 2y ago
    A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/llama-bench https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc. It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even basic training.
    - diggan 2y ago
      I guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)
      - t1amat 2y ago
        In fairness it’s become even more difficult now than ever before. * hardware spec * inference engine * specific model - differences to tokenizer will make models faster/slower with equivalent parameter count * quantization used - and you need to be aware of hardware specific optimizations for particular quants * kv cache settings * input context size * output token count This is probably not a complete list either.
  - nickthegreek 2y ago
    Best place to get that kinda info is gonna be /r/LocalLlama
- motorest 2y ago
  > I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. What's hard about it? You get the hardware, you run the software, you take measurements.
  - GTP 2y ago
    Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.
    - diggan 2y ago
      > but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere
      - unshavedyak 2y ago
        I don't think they're saying anything counter to that. The people who don't require the volume of data will run these. Ie the super early adopters.
      - colonCapitalDee 2y ago
        We've already started, we just haven't finished yet
fkyoureadthedoc 2y ago
On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit
- diggan 2y ago
  > gemma2:27b What quantization are you using? What's the runtime+version you run this with? And the rest of the settings? Edit: Turns out parent is using Q4 for their test. Doing the same test with LM Studio and a 3090ti + Ryzen 5950X (with 44 layers on GPU, 2 on CPU) I get ~15 tokens/second.
  - fkyoureadthedoc 2y ago
    Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"
    - diggan 2y ago
      When you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.
      - fkyoureadthedoc 2y ago
        not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b https://pastes.io/ollama-run-gemma2-27b
un_ess 2y ago
Per the screenshot, this is a DeepSeek running on a 192GB M2 Studio https://nitter.poast.org/ggerganov/status/1884612770093842728#m https://nitter.poast.org/ggerganov/status/188461277009384272... The same on Nvidia (various models) https://github.com/ggerganov/llama.cpp/issues/11474 https://github.com/ggerganov/llama.cpp/issues/11474 [1] this is a the model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-UD-IQ1_S https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
- diggan 2y ago
  So Apple M2 Studio does ~15 tks/second and A100-SXM4-80GB does 9 tks/second? I'm not sure I'm reading the results wrong or missing some vital context, but that sounds unlikely to me.
  - achierius 2y ago
    The studio has a lot more ram available to the GPU (up to 192gb) than the a100 (80gb), and iirc at least comparable memory bandwidth -- those are what matter when you're doing LLM inference, so the studio tends to win out there. Where the a100 and other similar chips dominate is in training &c, which is mostly a question of flops.
    - diggan 2y ago
      > and iirc at least comparable memory bandwidth I don't think they do. From Wikipedia: > the M2 Pro, M2 Max, and M2 Ultra have approximately 200 GB/s, 400 GB/s, and 800 GB/s respectively From techpowerup: > NVIDIA A100 SXM4 80 GB - Memory bandwidth - 2.04 TB/s Seems to be a magnitude of difference, and that's just the bandwidth.
deleted 2y ago
[deleted]
vladgur 2y ago
as someone who is paying $0.50 per kwh, id also like to include kw per 1000 tokens or something to give me a sense of cost of ownership these local systems
- troyvit 2y ago
  That would be an awesome thing across the industry -- even for the big commercial models -- for those who care not only about price but also carbon footprint.

ekianjo 2y ago

Mac Mini will be very slow for context ingestion compared to nvidia GPU, and the other issue is that they are not usable for Stable Diffusion... So if you just want to use LLMs, maybe, but if you have other interests in AI models, probably not the right answer.

drcongo 2y ago
I use a Mac Studio for Stable Diffusion, what's special about the Mac Mini that means it won't work?
- vunderba 2y ago
  What models are you using? Stable diffusion 1.5, SDXL, or flux? I've heard that Macs are pretty slow with XL and borderline unusable for flux requiring minutes at a time to generate a single image - whereas an RTX4090 can generate a 1024x1024 image with the higher quality Flux Dev model (not schnell) in 14 seconds. OP is probably correct that if you want to branch out of just strictly LLM's, cuda is the way to go. I've never heard of anyone getting LTX or hunyuan running on a Mac for example.
  - drcongo 2y ago
    I've used 1.5 through 3.5, XL has been kinda fine for me but tiny adjustments can take it from good to terrible. Good point about Flux though, that's awful on the Mac.

JKCalhoun 2y ago

For this use case though, I would prefer something more modular than Apple hardware — where down the road I could upgrade the GPUs, for example.

motorest 2y ago

> For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. Around half that price tag was attributed to the blogger reusing an old workstation he had lying around. Beyond this point, OP slapped two graphics cards into an old rig. A better description would be something like "what buying two graphics cards gets you in terms of AI".

Capricorn2481 2y ago
> Beyond this point, OP slapped two graphics cards into an old rig Meaning what? This is largely what you do on a budget since RAM is such a difference maker in token generation. This is what's recommended. OP could buy an a100, but that wouldn't be a budget build.

oofbaroomf 2y ago

The bottleneck for single batch inference is memory bandwidth. The M4 Pro has less memory bandwidth than the P40, so it would be slower. Also, the setup presented in the OP has system RAM, allowing you to run models than what fits in 48GB of VRAM (and with good speeds too if you offload with something like ktransformers).

anthonyskipper 2y ago
>>M4 Pro has less memory bandwidth than the P40, so it would be slower Why do you say this? I thought the p40 only had a memory bandwidth of 346 Gbytes/sec. The m4 is 546 GB/s. So the macbook should kick the crap out of the p40.
- oofbaroomf 2y ago
  The M4 Max has up to 546 GB/s. The M4 Pro, what GP was talking about, has only 273 GB/s. An M4 Max with that much RAM would most likely exceed OP's budget.

UncleOxidant 2y ago

I wish Apple would offer a 128GB option in the Mac Mini - That would require an M4 Max which they don't offer in the mini. I know they have a MBP with M4 Max and 128GB, but I don't need another laptop.

kridsdale1 2y ago
I’m waiting until this summer with the M4 Ultra Studio.
- UncleOxidant 2y ago
  Which will likely be over five grand for 128GB.