4 ms·

Comparison for previous generations https://www.hardware-corner.net/guides/gpu-benchmark-large-language-models/ https://www.hardware-corner.net/guides/gpu-benc

Comparison for previous generations https://www.hardware-corner.net/guides/gpu-benchmark-large-language-models/ https://www.hardware-corner.net/guides/gpu-benchmark-large-l...

mirekrusin 2y ago
When running on apple silicon you want to use mlx, not llama.cpp as this benchmark does. Performance is much better than what's plotted there and seems to be getting better, right? Power consumption is almost 10x smaller for apple. Vram is more than 10x larger. Price wise for running same size models apple is cheaper. Upper limit (larger models, longer context) is far larger for apple (for nvidia you can easily put 2x cards, more than that it becomes whole complex setup no ordinary person can do). Am I missing something or apple is simply currently better for local llms?
- nicman23 2y ago
  there is a plateau where you simply need more compute and the m4 cores are not enough, so even if they have enough ram for the model the token/s is not useful
  - mirekrusin 2y ago
    For all models fitting 2x 5090 (2x 32GB) that's not a problem, so you can say if you have this problem then RTX is also not an option. On apple silicons you can always use MoE models, which work beautifully. On RTX it's kind of waste to be honest to run MoE, you'd be better off running single, whole active model that fills available memory (with enough space for the context).
- sgt 2y ago
  I'm trying to find out about that as well as I'm considering a local LLM for some heavy prototyping. I don't mind which HW I buy, but it's on a relative budget and energy efficiency is also not a bad thing. Seems the Ultra can do 40 tokens/sec on DeepSeek and nothing even comes close at that price point.
  - hnfong 2y ago
    The DeepSeek R1 distilled onto Llama and Qwen base models are also unfortunately called “DeepSeek” by some. Are you sure you’re looking at the right thing? The OG DeepSeek models are hundreds of GB quantized, nobody is using RTX GPUs to run them anyway…
- hnfong 2y ago
  Unless something changed recently, I’m not aware of big perf differences between MLX and llama.cpp on Apple hardware.
  - loufe 2y ago
    I'm under the same impression. Llama.cpp's readme used to start with "Apple Silicon as a First Class Citizen", and IIRC Georgi works on Mac himself.
- Tostino 2y ago
  You are missing something. This is a single stream of inference. You can load up the Nvidia card with at least 16 inference streams and get at much higher throughout tokens/sec. This just is just a single user chat experience benchmark.

Comparison for previous generations https://www.hardware-corner.net/guides/gpu-benchmark-large-language-models/ https://www.hardware-corner.net/guides/gpu-benchmark-large-l...

mirekrusin 2y ago

When running on apple silicon you want to use mlx, not llama.cpp as this benchmark does. Performance is much better than what's plotted there and seems to be getting better, right? Power consumption is almost 10x smaller for apple. Vram is more than 10x larger. Price wise for running same size models apple is cheaper. Upper limit (larger models, longer context) is far larger for apple (for nvidia you can easily put 2x cards, more than that it becomes whole complex setup no ordinary person can do). Am I missing something or apple is simply currently better for local llms?

nicman23 2y ago
there is a plateau where you simply need more compute and the m4 cores are not enough, so even if they have enough ram for the model the token/s is not useful
- mirekrusin 2y ago
  For all models fitting 2x 5090 (2x 32GB) that's not a problem, so you can say if you have this problem then RTX is also not an option. On apple silicons you can always use MoE models, which work beautifully. On RTX it's kind of waste to be honest to run MoE, you'd be better off running single, whole active model that fills available memory (with enough space for the context).
sgt 2y ago
I'm trying to find out about that as well as I'm considering a local LLM for some heavy prototyping. I don't mind which HW I buy, but it's on a relative budget and energy efficiency is also not a bad thing. Seems the Ultra can do 40 tokens/sec on DeepSeek and nothing even comes close at that price point.
- hnfong 2y ago
  The DeepSeek R1 distilled onto Llama and Qwen base models are also unfortunately called “DeepSeek” by some. Are you sure you’re looking at the right thing? The OG DeepSeek models are hundreds of GB quantized, nobody is using RTX GPUs to run them anyway…
hnfong 2y ago
Unless something changed recently, I’m not aware of big perf differences between MLX and llama.cpp on Apple hardware.
- loufe 2y ago
  I'm under the same impression. Llama.cpp's readme used to start with "Apple Silicon as a First Class Citizen", and IIRC Georgi works on Mac himself.
Tostino 2y ago
You are missing something. This is a single stream of inference. You can load up the Nvidia card with at least 16 inference streams and get at much higher throughout tokens/sec. This just is just a single user chat experience benchmark.