7 ms·
Benchmark Framework Desktop Mainboard and 4-node cluster
https://www.jeffgeerling.com/blog/2025/i-clustered-four-framework-mainboards-test-huge-llms https://www.jeffgeerling.com/blog/2025/i-clustered-four-fram...
- jeffbee 1y agoI had been hoping that these would be a bit faster than the 9950X because of the different memory architecture, but it appears that due to the lower power design point the AI Max+ 395 loses across the board, by large margins. So I guess these really are niche products for ML users only, and people with generic workloads that want more than the 9950X offers are shopping for a Threadripper.
- dijit 1y agoSounds about right. I’m struggling to justify the cost of a Threadripper (let alone pro!) for a AAA game studio though. I wonder who can justify these machines. High frequency trading? data science? shouldn’t that be done on servers?
- jeffbee 1y agoYeah I don't get it either. To get marginally more resources than the 9950X you have to make a significant leap in price to a $1500+ CPU on a $1000 motherboard.
- wpm 1y agoNo, you get it. This is pure market segmentation. If you need that little bit extra, you’re forced to compromise, or to open your wallet big time, and AMD is betting on people who “really” need that slightly extra oomph to pay.
- toast0 1y agoThe premium is about getting more i/o. More memory channel, a lot more lanes. Maybe also a very high power limit. If you need that in a single system, you gotta pay up. Lower tier SP6 processors are actually pretty reasonably priced, boards are still spendy though.
- sliken 1y agoDoubly so when a $650 Epyc motherboard and a $650 Epyc CPU gets you 3x the threadipper's bandwidth and 1.5 the thread ripper pro's bandwidth. Seems like threadripper is too low volume to be price competitive with Epyc and there's a relatively price insensitive workstation market out there.
- kadoban 1y agoThreadripper very rarely seems to make any sense. The only times it seems like you want it are for huge memory support/bandwidth and/or a huge number of pcie slots. But it's not cheap or supported enough compared to epyc to really make sense to me any time I've been specing out a system along those lines.
- StrangeDoctor 1y agoI bought a threadripper pro system out of desperation, trying to get secondhand PCIe 80G A100s to run locally. The huge rebar allocations confused/crashed every Intel/AMD system I had access to. I think the Xeon systems should have worked and that it was actually a motherboard bios issue, but I had seen a photo of it running in a threadripper and prayed I wasn’t digging an even deeper hole.
- kadoban 1y agoYeah, that makes sense if you just have ~proof that some configuration works and want to just be done with it.
- jeffbee 1y agoThis is why a business like Puget Systems, or a line like HP Z Workstations, persist. You know in advance that your rig will work.
- sliken 1y agoI've been tempted, but had a hard time finding a case where I needed more than the 9950x but less than a single socket epyc. Especially since the epyc motherboards are cheaper, CPUs are cheaper, and the Epycs have 3x the memory bandwidth of the thread ripper and 1.5x the memory bandwidth of the Threadripper pro.
- rtkwe 1y agoIt also seems like the tools aren't there to fully utilize them. Unless I misunderstood he was running off CPU only for all the test so there's still the iGPU and NPU performance that's not been utilized in these tests.
- geerlingguy 1y agoNo, only a couple initial tests with Ollama used CPU. I ran most tests on Vulkan / iGPU, and some on ROCm (read further down the thread). I found it difficult to install ROCm on Fedora 42 but after upgrading to Rawhide it was easy, so I re-tested everything with ROCm vs Vulkan. Ollama, for some silly reason, doesn't support Vulkan even though I've used a fork many times to get full GPU acceleration with it on Pi, Ampere, and even this AMD system... (moral of the story just stick with llama.cpp).
- edwinjones 1y agoSadly, the reason they give is subjectively terrible: https://x.com/ollama/status/1952783981000446029 https://x.com/ollama/status/1952783981000446029 No experimental flag option, no "you can use the fork that works fine but we don't have capacity to support this" just a hard "no, we think it's unreliable". I guess they just want you to drop them and use llama.cpp.
- geerlingguy 1y agoYeah, my conspiracy theory is Nvidia is somehow influencing the decision. If you can do Vulkan with Ollama, it opens up people to using Intel/AMD/other iGPUs and you might not be incentivized to buy an Nvidia GPU. ROCm support is not wonderful. It's certainly worse for an end user to deal with than Vulkan, which usually 'just works'.
- edwinjones 1y agoI agree. AMD should just go all in on vulkan I think, The ROCm compatibility list is terrible compared to...every modern device and probably some ancient gpus that can be made to work with vulkan as well. Considering they created mantle, you would think it would be the obvious move too.
- sliken 1y agoAcross the board, by a large margin? Phoronix ran 200 benchmarks on the 9950x vs 395x max and found a difference of less than 5%. Not bad considering the average power use was 91 watts vs 154 watts. If you need the memory bandwidth the strix halo looks good, if you are cache friendly and don't care about using almost double the power than the 9950x is a better deal.
- jeffbee 1y agoThe way Phoronix weights an average score for a machine is ridiculous, because there aren't any users who do all of machine learning, fluid dynamics, video compression, database hosting, and software development, and games on the same machine. I looked at the applications that matter to me and the 9950X wins by 40% in those.
- mhitza 1y agoI've ran a comparison benchmark for the smaller models https://gist.github.com/mhitza/f5a8eeb298feb239de10f9f60f8411a7 https://gist.github.com/mhitza/f5a8eeb298feb239de10f9f60f841... Comparing it against the RTX 4000 SFF Ada (20GB) which is around $1.2k (if you believe the original price on the nvidia website https://marketplace.nvidia.com/en-us/enterprise/laptops-workstations/nvidia-rtx-4000-sff-ada-generation/ https://marketplace.nvidia.com/en-us/enterprise/laptops-work...). Which I have access to on a Hetzner GEX44. I'm going to ballpark it between 2.5-3x faster than the desktop. Except for the tg128 test, where the difference is "minimal" (but I didn't do the math).
- yencabulator 1y agoThe whole point of these integrated memory designs is to go beyond that 20 GB VRAM.
- throwdbaaway 1y agoActually, you can combine them. When compared to Mac Studio, the main advantage of these Strix Halo boxes is that you still add a bunch of egpu over usb4/oculink, for better PP.
- Tsiklon 1y agoI see Wendell of Level1Techs combines the two in his video on this system. Theoretically you can have the best of both worlds if you don’t mind running an Occulink E-GPU enclosure https://youtu.be/L-xgMQ-7lW0 https://youtu.be/L-xgMQ-7lW0
- reissbaker 1y agoThanks for the excellent writeup. I'm pleasantly surprised that ROCm worked as well as it did — for the price these aren't bad for LLM workloads and some moderate gaming. (Apple is probably still the king of affordable at-home inference, but for games... Amazing these days but Linux is so much better.)
- mulmen 1y agoI switched to Fedora Sway as my daily driver nearly two years ago. A Windows title wasn’t working on my brand new PC. I switched to Steam+Proton+Fedora and it worked immediately. Valve now offers a more stable and complete Windows API through Proton than Microsoft does through Windows itself.
- xemdetia 1y agoI was about to be annoyed until you said you got preprod units. I guess I'll have to build on this when my desktop shows up.
- iamtheworstdev 1y agofor those who are already in the field and doing these things - if I wanted to start running my own local LLM.. should I find an Nvidia 5080 GPU for my current desktop or is it worth trying one of these Framework AMD desktops?
- wmf 1y agoIf you think the future is small models (27B) get Nvidia; if you think larger models (70-120B) are worth it then you need AMD or Apple.
- yencabulator 1y agoI wonder how much MoE will disrupt this. qwen3:30b-a3b is pretty good even on pure CPU, but a lot smarter than a 3B parameter model. If the CPU-GPU bottleneck isn't too tight, a large model might be able to sustainably cache the currently active experts in GPU RAM.
- hengheng 1y agoThe recent qwen3 models run fine on CPU + GPU, and so does gpt-oss. LM Studio and Ollama are turnkey solutions where the user has to know nothing about memory management. But finding benchmarks for these hybrid setups is astonishingly difficult. I keep thinking that the bottleneck has to be CPU RAM, and for a large model the difference would be minor. For example with an 100 GByte model such as quantised gpt-oss-120B, I imagine that going from 10G to 24G would scale up my tk/s like 1/90 -> 1/76, so 20% advantage? But I can't find much on the high-level scaling math. People seem to either create calculators that oversimplify, or they seem too deep into the weeds. I'd like a new anandtech please.
- whizzter 1y agoDoesn't matter, people will always find ways to eat RAM despite finding more clever ways to do things.
- yencabulator 1y ago
- Havoc 1y agoJeff - check out the distributed-llama project...you should be able to distribute over entire cluster
- burnte 1y agoHe mentioned that in the video.
- yjftsjthsd-h 1y agohttps://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama ?
- geerlingguy 1y agoI've been testing Exo (seems dead), llama.cpp RPC (has a lot of performance limitations) and distributed-llama (faster but has some Vulkan quirks and only works with a few models). See my AI cluster automation setup here: https://github.com/geerlingguy/beowulf-ai-cluster https://github.com/geerlingguy/beowulf-ai-cluster I was building that through the course of making this video, because it's insane how much manual labor people put into building home AI clusters :D
- jvanderbot 1y agoSo, TL;DR? I saw mixed results but comments suggest very good performance relative to other at-home setups. Can someone summarize?
- geerlingguy 1y agoI put most of the top-line numbers and some graphs on my blog: https://www.jeffgeerling.com/blog/2025/i-clustered-four-framework-mainboards-test-huge-llms https://www.jeffgeerling.com/blog/2025/i-clustered-four-fram...
- jvanderbot 1y agoGreat! As always fantastic writeup
- mixmastamyk 1y agoImpressive yet disappointing due to software. Better use for now is for a compact render farm.
- adolph 1y agoThe Framework Desktop has at least two M.2 connectors for NVME. I wonder if an interconnect with higher performance than Ethernet or Thunderbolt could be established using one of the M.2 to connect to PCIe via Oculink?
- nrp 1y agoThere is also a PCIe x4 slot that you can use for other high throughput network options.
- syntaxing 1y agoKinda bummed, I get why he used Ollama but I feel like using llama cpp directly would provide better and more consistent results
- mkl 1y agoAs the article describes, most of this was done with llama.cpp, not Ollama.
- syntaxing 1y agoAhh good catch, I didn’t notice if you scroll lower, he has the llama cpp results. The ollama-benchmark repo name is a misnomer.
- geerlingguy 1y agoI'm slowly migrating all my testing to https://github.com/geerlingguy/beowulf-ai-cluster https://github.com/geerlingguy/beowulf-ai-cluster
- RossBencina 1y agoI heard that ik_llama.cpp performs better for CPU use: https://github.com/ikawrakow/ik_llama.cpp/ https://github.com/ikawrakow/ik_llama.cpp/
- nektro 1y agono compilation tests?
- geerlingguy 1y agoThose are in my SBC-reviews repo: https://github.com/geerlingguy/sbc-reviews/issues/80 https://github.com/geerlingguy/sbc-reviews/issues/80
- nektro 1y agothanks!
- _joel 1y ago> usually resulting in one word repeating ad infinitum I've had that using gemini (via windsurf). Doesn't seem to happen with other models. No idea if there's any correlation but it's an interesting failure mode.
- mattnewton 1y agoThis is usually a symptom of greedy sampling (always picking the most probable token) on smaller models. It's possible that configuration had different sampling defaults, ie. was not using top p or temperature. I'm not familiar with distributed-llama but from searching the git repo it looks like it at least takes a --temperature flag and probably has one for top p. I'd recommend rerunning the benchmarks with the sampling methods explicitly configured the same in each tool. It's tempting to benchmark with all the nondeterminism turned off, but I think it's less useful since in practice for any model you're self hosting for real work you're going to probably want top-p sampling or something like it and you want to benchmark the implementation of that too. I've never seen gemini do this though, that'd be kinda wild if they shipped something that samples that way. I wonder if windsurf was sending a different config over the api or if this was a different bug.
- mrbungie 1y agoYep, sometimes Gemini for some reason ends up in what I call "ergodic self-flagellation". Here are some examples: https://www.reddit.com/r/GeminiAI/comments/1lxqbxa/i_am_actually_terrified/ https://www.reddit.com/r/GeminiAI/comments/1lxqbxa/i_am_actu...
- justinclift 1y agoI've seen that occasionally with one of the deepseek models when using the default Ollama context size of 4096, rather than whatever the model's preferred context size was. After having that happen, I switched my stuff to check the model's preferred context size, then set the context size to match, before using any given model.
- oblio 1y agoThose numbers are better than I was expecting.
- hinkley 1y agoJeff! Someone needs to make Framework MBs work in a blade arrangement, and you seem to be the likely person to get it done.
- chickensong 1y agoRelated: https://www.printables.com/model/1075458-framework-mainboard-blade-center/collections https://www.printables.com/model/1075458-framework-mainboard...
- lifeinthevoid 1y agoSetup looks very sexy.
- sliken 1y agoApparently the frameworks desktop's 5g bit network isn't fast enough to scale well with LLM inference workloads, even for a modest GPU. Anyone know what kind of network is required to scale well for a single modest GPU?
- geerlingguy 1y agoIn the case of llama.cpp's RPC mode, the network isn't the limiting factor for inference, but for distributing layers to nodes. I was monitoring the network while running various models, and for all models, the first step was to copy over layers (a few gigabytes to 100 or so GB for the huge models), and that would max out the 5 Gbps connection. But then while warming up and processing, there were only 5-10 Mbps of traffic, so you could do it over a string and tin cans, almost. But that's a limitation of the current RPC architecture, it can't really parallelize processing, so as I noted in the post and in my video, it kinda uses resources round-robin style, and you can only get worse performance across the entire cluster than on a single node for any model you can fit on the single node.
- rtkwe 1y agoNo network interconnect is going to scale well until you get into the expensive enterprise realm where infiniband and other direct connect copper/fiber reigns. The issue is less raw bandwidth but latency. Network is inherently 100x+ slower than memory access so when you start sharing a memory intensive workload like an LLM across a normal network it's going to crater your performance unless the work can be somewhat chunked to keep communication between nodes on the network to a minimum.
- sliken 1y agoReally? Seems like scaling is pretty tolerant of latency, but very bandwidth intensive. Thus the move from IB to various flavors of ethernet (for AMD's GPUs, Tenstorrent, various others). Not to mention broadcom pushing various 50 and 100tbit ethernet switching chips for AI. Even 25gbit these days is pretty affordable for home, if it scaled 5x better than 5gbit that might be enough to make larger models MUCH more practical.
- bawana 1y agoMem bandwidth sucks compared to Mac Studio ultra 3. And you cant add gpus easily although as an apu it is impressive and way better than nvidias gold box. Wendell said it better. Im waiting for the Mac Studio ultra 5
- jauntywundrkind 1y agoYes. And this is so crucial! It's still a huge leap forward for x86. Quad-channel (4x) DDR5-8000 is both double the (client/non-server) lane count, and at a blisteringly high clock rate. That's very impressive. Upcoming Zen6 Epyc was just confirmed to go to 12-> 16 channel. That'll be very good to see. The Strix Halo successor Medusa Halo is supposed to be 6x channel. (Most of these rumors/leaks via Moore's Law is Dead fwiw). It's absolutely needed to scale to more cores. But still seems so short of what AI demands. I really can't congratulate Apple enough for being deadly serious about memory bandwidth. What is just gobsmacking to me is that no one else has responded, half a decade latter. Put the ram on chip! DDR, not crazy expensive HBM. The practice of building super chips out of 4x chips, getting scalability that way, feels so obvious too, is so commendable! Different end of the spectrum, but Intels tablet-size Lakefield had Package-on-Package (PoP) ram, and pretty fast for its day (4266MHz). But it didn't scale up the width, like Apple has. Its hard to see x86 seem so stuck, be so unable to make what feels like such a necessary push to
- jauntywundrkind 1y ago> For networking, I expected more out of the Thunderbolt / USB4 ports, but could only get 10 Gbps. I really wish we saw more testing of USB subsystems! With PCIe being so limited, there's such allure to having two USB4 ports! But will they work? IIRC we saw similar very low bandwidth on Apple's ARM chips too. This was during M1 or so; dunno if things got better with that chip or future ones! Presumably so or I feel like we'd be hearing about it, but also, these things can just go so hidden! It was really cool back in Ryzen 1 era seeing their CPU get some USB on the cpu itself, not have to go through the IO/peripheral Hub (southbridge?), with its limited connection to the CPU. There's a great up breakout chart here, showing both the 1800x and the various chipsets available: relishable data. https://www.techpowerup.com/cpu-specs/ryzen-7-1800x.c1879 https://www.techpowerup.com/cpu-specs/ryzen-7-1800x.c1879 I feel like there's been some recent improvements to USB4/thunderbolt in the kernel, to really insure all lanes get used. But I'm struggling to find a reference/link. What kernel was this tested against? If nothing else, it's be great to poke around at debugfs, to make sure it's getting all the lanes configured. https://www.phoronix.com/news/Linux-6.13-USB-Changes https://www.phoronix.com/news/Linux-6.13-USB-Changes
- OrangeMusic 1y agoCan you imagine a Beowulf cluster of these?