4 ms·

For the uninitiated, Strix Halo is the same as the AMD Ryzen AI Max+ 395 which will be in the Framework Desktop and is starting to show up in some mini PCs as w

For the uninitiated, Strix Halo is the same as the AMD Ryzen AI Max+ 395 which will be in the Framework Desktop and is starting to show up in some mini PCs as well. The memory bandwidth on that thing is 200GB/s. That's great compared to most other consumer-level x86 platforms, but quite far off of an Nvidia GPU (a 5090 has 1792GB/s, dunno about the pro level cards) or even Apple's best (M3 Ultra has 800GB/s). It certainly seems like a great value. But for memory bandwidth intensive applications like LLMs, it is just barely entering the realm of "good enough".

yieldcrv 1y ago
Apple is just being stupid, handicapping their own hardware so they can sell the fixed one next year or the year after This is time tested Apple strategy is now undermining their AI strategy and potential competitiveness tl;dr they could have done 1600GB/s
- saagarjha 1y ago
  They could have shipped a B200 too. Obviously there are reasons they don't do that.
- Nevermark 1y ago
  So their products are so much better, in customer demand terms that they don’t need to rush tech out the door? Whatever story you want to create, if customers are happy year after year then Apple is serving them well. Maybe not with same feature dimension balance you want, or other artificial/wishful balances you might make up for them. (When Apple drops the ball it is usually painful, painfully obvious and most often a result of a deliberate and transparent priority tradeoff. No secret switcherooos or sneaky downgrading. See: Mac Pro for years…)
  - yieldcrv 1y ago
    Apple is absolutely fumbling on their AI strategy despite their vertical hardware integration, there is no strategy. Its a known problem inside Apple, not a 4-D chess thing to wow everyone with a refined version in 2030
Rohansi 1y ago
You're comparing theoretical maximum memory bandwidth. It's not enough to only look at memory bandwidth because you're a lot more likely to be compute limited when you have a lot of memory bandwidth available. For example, M1 had so much bandwidth available that it couldn't make use of even when fully loaded.
- zargon 1y ago
  GPUs have both the bandwidth and the compute. During token generation, no compute is needed. But both Apple silicon and Strix Halo fall on their face during prompt ingestion, due to lack of compute.
  - supermatt 1y ago
    Compute (and lots of it) is absolutely needed for generation - 10s of billions of FLOPs per token on the smaller models (7B) alone - with computations of the larger models scaling proportionally. Each token requires a forward pass through all transformer layers, involving large matrix multiplications at every step, followed by a final projection to the vocabulary.
    - zargon 1y ago
      Obviously I don't mean literally zero compute. The amount of compute needed scales with the number of parameters, but I have yet to use a model that has so many parameters that token generation becomes compute bound. (Up to 104B for dense models.) During token generation most of the time is spent idle waiting for weights to transfer from memory. The processor is bored out of its mind waiting for more data. Memory bandwidth is the bottleneck.
      - supermatt 1y ago
        It sounds like you aren’t batching efficiently if you are being bound by memory bandwidth.
        1y ago

yieldcrv 1y ago

Apple is just being stupid, handicapping their own hardware so they can sell the fixed one next year or the year after This is time tested Apple strategy is now undermining their AI strategy and potential competitiveness tl;dr they could have done 1600GB/s

saagarjha 1y ago
They could have shipped a B200 too. Obviously there are reasons they don't do that.
Nevermark 1y ago
So their products are so much better, in customer demand terms that they don’t need to rush tech out the door? Whatever story you want to create, if customers are happy year after year then Apple is serving them well. Maybe not with same feature dimension balance you want, or other artificial/wishful balances you might make up for them. (When Apple drops the ball it is usually painful, painfully obvious and most often a result of a deliberate and transparent priority tradeoff. No secret switcherooos or sneaky downgrading. See: Mac Pro for years…)
- yieldcrv 1y ago
  Apple is absolutely fumbling on their AI strategy despite their vertical hardware integration, there is no strategy. Its a known problem inside Apple, not a 4-D chess thing to wow everyone with a refined version in 2030

Rohansi 1y ago

You're comparing theoretical maximum memory bandwidth. It's not enough to only look at memory bandwidth because you're a lot more likely to be compute limited when you have a lot of memory bandwidth available. For example, M1 had so much bandwidth available that it couldn't make use of even when fully loaded.

zargon 1y ago
GPUs have both the bandwidth and the compute. During token generation, no compute is needed. But both Apple silicon and Strix Halo fall on their face during prompt ingestion, due to lack of compute.
- supermatt 1y ago
  Compute (and lots of it) is absolutely needed for generation - 10s of billions of FLOPs per token on the smaller models (7B) alone - with computations of the larger models scaling proportionally. Each token requires a forward pass through all transformer layers, involving large matrix multiplications at every step, followed by a final projection to the vocabulary.
  - zargon 1y ago
    Obviously I don't mean literally zero compute. The amount of compute needed scales with the number of parameters, but I have yet to use a model that has so many parameters that token generation becomes compute bound. (Up to 104B for dense models.) During token generation most of the time is spent idle waiting for weights to transfer from memory. The processor is bored out of its mind waiting for more data. Memory bandwidth is the bottleneck.
    - supermatt 1y ago
      It sounds like you aren’t batching efficiently if you are being bound by memory bandwidth.
      - 1y ago