5 ms·

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigab

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new chip m3 for AI.

wkat4242 2y ago
True, I also noticed that bigger models run slower at the same memory bandwidth (makes sense).
memhole 2y ago
Yeah, I don’t think RAM is the bottleneck. Which is unfortunate. It feels like a missed opportunity for them. I think Apple partly became popular because it enabled creatives and developers.
- throw-qqqqq 2y ago
  > I don’t think RAM is the bottleneck Not the size/amount, but the memory bandwidth usually is.
espadrine 2y ago
> if a llm will run with usable performance at that scale? Yes. The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores. For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1] (very roughly 13 GB per forward pass). With a memory bandwidth of 800 GB/s[3], the Mac Studio will be able to output 800/13 = 62 tokens per second. [0]: https://ollama.com/library/deepseek-r1 https://ollama.com/library/deepseek-r1 [1]: https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 [2]: https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac-studio-the-most-powerful-mac-ever/ https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac...
- fullstackchris 2y ago
  You seem like you know what you are talking about... mind if I ask what your thoughts on quantization are? Its unclear to me if quantization affects quality... I feel like I've heard yes and no arguments
  - sosuke 2y ago
    I don't know what I'm talking about but when I first asked your question this https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... helped start me on a path to understanding. I think. But if you don't already know the question your asking is not at all something I could distill down into a sentence or to that would make sense to a lay-person. Even then I know I couldn't distill it at all sorry. Edit: I found this link I referenced above on quantized models by bartowski on huggingface https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF#which-file-should-i-choose https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF#whic...
  - espadrine 2y ago
    There is no question that quantization degrades quality. The GGUF R1 uses Q4_K_M, which, on Llama-3-8B, increases the perplexity by 0.18[0]. Many plots show increasing degradation as you quantize more[1]. That said, it is possible to train a model in a quantization-aware way[2][3], which improves the quality a bit, although not higher than the raw model. Also, a loss in quality may not be perceptible in a specific use-case. Famously LMArena.ai tested Llama 3.1 405B with bf16 and fp8, and the latter was only 2 Elo points below, well within measurement error. [0]: https://github.com/ggml-org/llama.cpp/blob/master/examples/quantize/quantize.cpp#L46 https://github.com/ggml-org/llama.cpp/blob/master/examples/q... [1]: https://github.com/ggml-org/llama.cpp/discussions/5063#discussioncomment-10906969 https://github.com/ggml-org/llama.cpp/discussions/5063#discu... [2]: https://pytorch.org/blog/quantization-aware-training/ https://pytorch.org/blog/quantization-aware-training/ [3]: https://mistral.ai/news/ministraux https://mistral.ai/news/ministraux
  - Ambix 2y ago
    I did my own experiments and it looks like (surprisingly) Q4KM models often outperforms Q6 and Q8 quantised models. For bigger models (in range of 8B - 70B) the Q4KM is very good, there are no any degradation compared to full FP16 models.
jazzyjackson 2y ago
I returned an M2 Max studio with 96GB RAM, unquantized llama 70B 3.1 was dog slow, not an interactive pace. I'm interested in offline LLM but couldn't see how it was going to produce $3,000 ROI.
- FloatArtifact 2y ago
  It would be really cool if there was awebsite "we there yet" for reasonable offline AI. It could track different hardware configurations and reasonably standardized benchmark performance per model. I know there's benchmarks buried in github Llama repository.
  - robbomacrae 2y ago
    There seems to be a LOT of interest in such a site in the comments here. There seem to be multiple IP issues with sharing your code repo with an online service so I feel a lot of folks are waiting for the hardware to make this possible. We need a SWE-bench for open source LLM's and for each model to have 3Dmark like benchmarks on various hardware setups. I did find this which seems very helpful but is missing the latest models and hardware options. https://kamilstanuch.github.io/LLM-token-generation-simulator/ https://kamilstanuch.github.io/LLM-token-generation-simulato...
    - FloatArtifact 2y ago
      Looks like he bases the benchmarks off of https://github.com/ggml-org/llama.cpp/discussions/4167 https://github.com/ggml-org/llama.cpp/discussions/4167 I get why he calls it a simulator, as it can simulate token output. It's an important aspect for evaluating use case if you need to get a sense of how much token output is relevant beyond the simple tokens per second text.
slama 2y ago
The M3 Ultra is the only configuration that supports 512GB and it has memory bandwidth of 819GB/s.
Ecko123 2y ago
[dead]

wkat4242 2y ago

True, I also noticed that bigger models run slower at the same memory bandwidth (makes sense).

memhole 2y ago

Yeah, I don’t think RAM is the bottleneck. Which is unfortunate. It feels like a missed opportunity for them. I think Apple partly became popular because it enabled creatives and developers.

throw-qqqqq 2y ago
> I don’t think RAM is the bottleneck Not the size/amount, but the memory bandwidth usually is.

espadrine 2y ago

> if a llm will run with usable performance at that scale? Yes. The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores. For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1] (very roughly 13 GB per forward pass). With a memory bandwidth of 800 GB/s[3], the Mac Studio will be able to output 800/13 = 62 tokens per second. [0]: https://ollama.com/library/deepseek-r1 https://ollama.com/library/deepseek-r1 [1]: https://arxiv.org/pdf/2412.19437 https://arxiv.org/pdf/2412.19437 [2]: https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac-studio-the-most-powerful-mac-ever/ https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac...

fullstackchris 2y ago
You seem like you know what you are talking about... mind if I ask what your thoughts on quantization are? Its unclear to me if quantization affects quality... I feel like I've heard yes and no arguments
- sosuke 2y ago
  I don't know what I'm talking about but when I first asked your question this https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... helped start me on a path to understanding. I think. But if you don't already know the question your asking is not at all something I could distill down into a sentence or to that would make sense to a lay-person. Even then I know I couldn't distill it at all sorry. Edit: I found this link I referenced above on quantized models by bartowski on huggingface https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF#which-file-should-i-choose https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF#whic...
- espadrine 2y ago
  There is no question that quantization degrades quality. The GGUF R1 uses Q4_K_M, which, on Llama-3-8B, increases the perplexity by 0.18[0]. Many plots show increasing degradation as you quantize more[1]. That said, it is possible to train a model in a quantization-aware way[2][3], which improves the quality a bit, although not higher than the raw model. Also, a loss in quality may not be perceptible in a specific use-case. Famously LMArena.ai tested Llama 3.1 405B with bf16 and fp8, and the latter was only 2 Elo points below, well within measurement error. [0]: https://github.com/ggml-org/llama.cpp/blob/master/examples/quantize/quantize.cpp#L46 https://github.com/ggml-org/llama.cpp/blob/master/examples/q... [1]: https://github.com/ggml-org/llama.cpp/discussions/5063#discussioncomment-10906969 https://github.com/ggml-org/llama.cpp/discussions/5063#discu... [2]: https://pytorch.org/blog/quantization-aware-training/ https://pytorch.org/blog/quantization-aware-training/ [3]: https://mistral.ai/news/ministraux https://mistral.ai/news/ministraux
- Ambix 2y ago
  I did my own experiments and it looks like (surprisingly) Q4KM models often outperforms Q6 and Q8 quantised models. For bigger models (in range of 8B - 70B) the Q4KM is very good, there are no any degradation compared to full FP16 models.

jazzyjackson 2y ago

I returned an M2 Max studio with 96GB RAM, unquantized llama 70B 3.1 was dog slow, not an interactive pace. I'm interested in offline LLM but couldn't see how it was going to produce $3,000 ROI.

FloatArtifact 2y ago
It would be really cool if there was awebsite "we there yet" for reasonable offline AI. It could track different hardware configurations and reasonably standardized benchmark performance per model. I know there's benchmarks buried in github Llama repository.
- robbomacrae 2y ago
  There seems to be a LOT of interest in such a site in the comments here. There seem to be multiple IP issues with sharing your code repo with an online service so I feel a lot of folks are waiting for the hardware to make this possible. We need a SWE-bench for open source LLM's and for each model to have 3Dmark like benchmarks on various hardware setups. I did find this which seems very helpful but is missing the latest models and hardware options. https://kamilstanuch.github.io/LLM-token-generation-simulator/ https://kamilstanuch.github.io/LLM-token-generation-simulato...
  - FloatArtifact 2y ago
    Looks like he bases the benchmarks off of https://github.com/ggml-org/llama.cpp/discussions/4167 https://github.com/ggml-org/llama.cpp/discussions/4167 I get why he calls it a simulator, as it can simulate token output. It's an important aspect for evaluating use case if you need to get a sense of how much token output is relevant beyond the simple tokens per second text.

slama 2y ago

The M3 Ultra is the only configuration that supports 512GB and it has memory bandwidth of 819GB/s.

Ecko123 2y ago

[dead]