5 ms·

Running a One Trillion-Parameter LLM Locally on AMD Ryzen AI Max+ Cluster

75 points by mindcrime 8mo ago

verdverm 8mo ago
The setup was around $10k, but maybe more now with mem/ssd prices. This is a good list, I like my Beelink a lot, my Minisforum likes to turn itself off every couple of weeks, not sure why yet. https://www.techradar.com/pro/there-are-15-amd-ryzen-ai-max-395-mini-pcs-in-the-world-right-now-heres-where-you-can-buy-them https://www.techradar.com/pro/there-are-15-amd-ryzen-ai-max-... --- Performance is pretty bad (<10/tps) and context is quite limited. Still good to see progress Prompt Size (tokens) | TFT (s) - Flash Attention Disabled | TFT (s) - Flash Attention Enabled 4096 | 53.7s | 39.7s 8192 | Out Of Memory (OOM) | 90.5s 16384 | Out Of Memory (OOM) | 239.1s
- rootusrootus 8mo ago
  > Minisforum likes to turn itself off every couple of weeks, not sure why yet AFAICT, the answer is "because Minisforum". I don't know if they have a design principle that they should run their systems near the edge of the thermal envelope or what, but Minisforum is the only brand I've had consistent trouble with stability on. My last one got to where it stopped booting altogether, just looped. Since then I've written off Minisforum as a brand, just not worth the hassle.
- shrubble 8mo ago
  I would try to add a fan or other cooling; my guess is that the CPU is handling thermal properly but something else is not.
  - verdverm 8mo ago
    Reply to one, but appreciate all three replies, thank you! Thermal is a good call, I have some cpu pastes around
- gmerc 8mo ago
  I put one of those notebook cooling pads underneath, that usually does the trick
ibeckermayer 8mo ago
Cool that it's possible but basically unusable performance characteristics. For an 8192 token prompt they report a ~1.5 minute time-to-first-token and then 8.30tk/s from there. For context ChatGPT is typically <<1s ttft and ~50tk/s.
- fc417fc802 8mo ago
  Given that APU only has 4 channels isn't this setup comically starved for bandwidth? By the same token, wouldn't you expect performance to scale approximately linearly as you add additional boxes? And wouldn't you be better off with smaller nodes (ie less RAM and CPU power per box)? If I'm right about that then if you're willing to go in for somewhere in the vicinity of $30k (24x the Max 385 model) you should be able to achieve ChatGPT performance.
  - ibeckermayer 8mo ago
    Good thought... I think you're wrong because the dominant factor is bandwidth over the interconnect. In this case they're using 5Gbps over Ethernet; compare that to 80-120 Gbps for a Thunderbolt 5 connected Mac Studio cluster: https://www.youtube.com/watch?v=bFgTxr5yst0 https://www.youtube.com/watch?v=bFgTxr5yst0
    - fc417fc802 8mo ago
      > I think you're wrong because the dominant factor is bandwidth over the interconnect. Is it? Why do you say that? I understand inference to be almost entirely bottlenecked on memory bandwidth. There are n^2 weights per layer but only n state values in the vector that exists between layers. Transmitting a few thousand (or even tens of thousands) of fp values does not require a notable amount of bandwidth by modern standards. Training is an entirely different beast of course. And depending on the workload latency can also impact performance. But for running inference with a single query from a single user I don't see how inter-node bandwidth is going to matter.
- JKCalhoun 8mo ago
  I've never understood the obsession with token/s. I'm fine with asking a question and then going on to another task (which might be making coffee). Even with a cloud-based LLM where the response is pretty snappy, I still find that I wander off and return when I am ready to digest the entire response.
elcritch 8mo ago
That’s pretty awesome! Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs?
- evanjrowley 8mo ago
  I appreciate them showcasing Framework Computer hardware, but they would have got 40 GB/s network performance if they had chosen the Minisorum MS-S1 Max: https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen%E2%84%A2-ai-max-395-mini-workstation-minisforum https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen...
  - mycall 8mo ago
    The FC hardware has room for a PCIe x4 with maybe ConnectX-7 @ 100GbE mode.
- jauntywundrkind 8mo ago
  I really wonder if AMD is going to keep getting walloped on the interconnect or if they'll start upping what's available to consumers, at some point.
- TacticalCoder 8mo ago
  > Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs? Does the network speed matter that much when TFA talks about outputting a few tens of tokens per second? Ain't 5 Gbit/s plenty for that? (I understand the need to load the model but that'd be local already right?)
  - elcritch 8mo ago
    Running inference requires sharing intermediate matrix results between nodes. Faster networking speeds that up.
    - wokkel 8mo ago
      I read (but cannot find this anymore) that the information sent from layer to layer is minimal. The actual matrix work happens within a layer. They are not doing matrix multiplication over the netwerk (that would be insane latency wise).
burnt-resistor 8mo ago
Framework has gone fully in the tank of Apple consumerization route of unrepairability and unupgradeability with a nonstandard machine, soldered-on RAM, and no meaningful PCIe slots. There's only the superficial appearance of longevity and future-proofness when it's really yet another silo. There's no way to add an IB, FC, or 100/400 GbE NICs to these machines. 5 GbE is a joke. Non-ECC RAM is a joke.
tills13 8mo ago
I set up ollama today and can barely run a 3b parameter model before the lag makes it unbearable. How much is one of these gonna run me?
- deleted 8mo ago
  [deleted]
- zeta0134 8mo ago
  I've been pretty happy with my Framework Desktop, though I managed to snag it before RAM prices shot through the roof. Currently, a tricked out model is around $2500. https://frame.work/desktop https://frame.work/desktop Mine sees more use as a Steam machine, but it can run decently large models. Ollama was trivial to get working, and qwen3-coder-next spits out paragraphs of text/code in seconds. I don't really do anything with that, but it's fun to mess around with. (LLMs are still pretty bad at assembly language.)
- jcgrillo 8mo ago
  You can buy a 128GB mainboard from framework for $2300, so maybe somewhere a bit over $9k by the time you've got power, storage, cables, racks (they sell those too). I was thinking about getting into one of these Strix Halo setups but decided to go a slightly different route with a lot higher TDP, better throughput, and a bit less VRAM. https://frame.work/products/framework-desktop-mainboard-amd-ryzen-ai-max-300-series?v=FRAFMK0006 https://frame.work/products/framework-desktop-mainboard-amd-...

75 points by mindcrime 8mo ago

The setup was around $10k, but maybe more now with mem/ssd prices. This is a good list, I like my Beelink a lot, my Minisforum likes to turn itself off every couple of weeks, not sure why yet. https://www.techradar.com/pro/there-are-15-amd-ryzen-ai-max-395-mini-pcs-in-the-world-right-now-heres-where-you-can-buy-them https://www.techradar.com/pro/there-are-15-amd-ryzen-ai-max-... --- Performance is pretty bad (<10/tps) and context is quite limited. Still good to see progress Prompt Size (tokens) | TFT (s) - Flash Attention Disabled | TFT (s) - Flash Attention Enabled 4096 | 53.7s | 39.7s 8192 | Out Of Memory (OOM) | 90.5s 16384 | Out Of Memory (OOM) | 239.1s

rootusrootus 8mo ago
> Minisforum likes to turn itself off every couple of weeks, not sure why yet AFAICT, the answer is "because Minisforum". I don't know if they have a design principle that they should run their systems near the edge of the thermal envelope or what, but Minisforum is the only brand I've had consistent trouble with stability on. My last one got to where it stopped booting altogether, just looped. Since then I've written off Minisforum as a brand, just not worth the hassle.
shrubble 8mo ago
I would try to add a fan or other cooling; my guess is that the CPU is handling thermal properly but something else is not.
- verdverm 8mo ago
  Reply to one, but appreciate all three replies, thank you! Thermal is a good call, I have some cpu pastes around
gmerc 8mo ago
I put one of those notebook cooling pads underneath, that usually does the trick

ibeckermayer 8mo ago

Cool that it's possible but basically unusable performance characteristics. For an 8192 token prompt they report a ~1.5 minute time-to-first-token and then 8.30tk/s from there. For context ChatGPT is typically <<1s ttft and ~50tk/s.

fc417fc802 8mo ago
Given that APU only has 4 channels isn't this setup comically starved for bandwidth? By the same token, wouldn't you expect performance to scale approximately linearly as you add additional boxes? And wouldn't you be better off with smaller nodes (ie less RAM and CPU power per box)? If I'm right about that then if you're willing to go in for somewhere in the vicinity of $30k (24x the Max 385 model) you should be able to achieve ChatGPT performance.
- ibeckermayer 8mo ago
  Good thought... I think you're wrong because the dominant factor is bandwidth over the interconnect. In this case they're using 5Gbps over Ethernet; compare that to 80-120 Gbps for a Thunderbolt 5 connected Mac Studio cluster: https://www.youtube.com/watch?v=bFgTxr5yst0 https://www.youtube.com/watch?v=bFgTxr5yst0
  - fc417fc802 8mo ago
    > I think you're wrong because the dominant factor is bandwidth over the interconnect. Is it? Why do you say that? I understand inference to be almost entirely bottlenecked on memory bandwidth. There are n^2 weights per layer but only n state values in the vector that exists between layers. Transmitting a few thousand (or even tens of thousands) of fp values does not require a notable amount of bandwidth by modern standards. Training is an entirely different beast of course. And depending on the workload latency can also impact performance. But for running inference with a single query from a single user I don't see how inter-node bandwidth is going to matter.
JKCalhoun 8mo ago
I've never understood the obsession with token/s. I'm fine with asking a question and then going on to another task (which might be making coffee). Even with a cloud-based LLM where the response is pretty snappy, I still find that I wander off and return when I am ready to digest the entire response.

elcritch 8mo ago

That’s pretty awesome! Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs?

evanjrowley 8mo ago
I appreciate them showcasing Framework Computer hardware, but they would have got 40 GB/s network performance if they had chosen the Minisorum MS-S1 Max: https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen%E2%84%A2-ai-max-395-mini-workstation-minisforum https://www.minisforum.com/pages/new-release-ms-s1-max-ryzen...
- mycall 8mo ago
  The FC hardware has room for a PCIe x4 with maybe ConnectX-7 @ 100GbE mode.
jauntywundrkind 8mo ago
I really wonder if AMD is going to keep getting walloped on the interconnect or if they'll start upping what's available to consumers, at some point.
TacticalCoder 8mo ago
> Though only 5gig Ethernet? Can’t they do usb-c / thunderbolt 40 Gb/s connections like Macs? Does the network speed matter that much when TFA talks about outputting a few tens of tokens per second? Ain't 5 Gbit/s plenty for that? (I understand the need to load the model but that'd be local already right?)
- elcritch 8mo ago
  Running inference requires sharing intermediate matrix results between nodes. Faster networking speeds that up.
  - wokkel 8mo ago
    I read (but cannot find this anymore) that the information sent from layer to layer is minimal. The actual matrix work happens within a layer. They are not doing matrix multiplication over the netwerk (that would be insane latency wise).

burnt-resistor 8mo ago

Framework has gone fully in the tank of Apple consumerization route of unrepairability and unupgradeability with a nonstandard machine, soldered-on RAM, and no meaningful PCIe slots. There's only the superficial appearance of longevity and future-proofness when it's really yet another silo. There's no way to add an IB, FC, or 100/400 GbE NICs to these machines. 5 GbE is a joke. Non-ECC RAM is a joke.

tills13 8mo ago

I set up ollama today and can barely run a 3b parameter model before the lag makes it unbearable. How much is one of these gonna run me?

deleted 8mo ago
[deleted]
zeta0134 8mo ago
I've been pretty happy with my Framework Desktop, though I managed to snag it before RAM prices shot through the roof. Currently, a tricked out model is around $2500. https://frame.work/desktop https://frame.work/desktop Mine sees more use as a Steam machine, but it can run decently large models. Ollama was trivial to get working, and qwen3-coder-next spits out paragraphs of text/code in seconds. I don't really do anything with that, but it's fun to mess around with. (LLMs are still pretty bad at assembly language.)
jcgrillo 8mo ago
You can buy a 128GB mainboard from framework for $2300, so maybe somewhere a bit over $9k by the time you've got power, storage, cables, racks (they sell those too). I was thinking about getting into one of these Strix Halo setups but decided to go a slightly different route with a lot higher TDP, better throughput, and a bit less VRAM. https://frame.work/products/framework-desktop-mainboard-amd-ryzen-ai-max-300-series?v=FRAFMK0006 https://frame.work/products/framework-desktop-mainboard-amd-...