8 ms·

1.5 TB of VRAM on Mac Studio – RDMA over Thunderbolt 5

617 points by rbanffy 11mo ago

behnamoh 11mo ago
My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM than more devices (centralized will always be faster than distributed). - +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP.
- tylerflick 11mo ago
  > TOTALLY okay with it consuming +600W energy The 2019 i9 Macbook Pro has entered the chat.
- burnt-resistor 11mo ago
  Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/filer. I'm disappointed Apple has a persistent blindspot preventing them from succeeding in data center-quality gear category when they could've had virtualized servers, networking, and storage, things that would eventually find their way into my home lab after 5-7 years.
  - Terretta 11mo ago
    > I'm disappointed Apple has a persistent blindspot preventing them from succeeding in ... things that would eventually find their way into my home lab after 5-7 years. I can see the dollar signs in their eyes right now. Aftermarkets are a nice reflection of durable value, and there's a massive one for iPhones and a smaller one for quick flameout startup servers, but not much money in 5 - 7 year old servers.
  - donavanm 11mo ago
    Enterprise never ever mattered, and there arent enough digits available to show your “home lab” use case in the revenue numbers. Xserve, the RAID shelves, and the directory services were kinda there as a half hearted attempt for that late 90-00s AV setup. All of that fell on the cutting room floor once personal devices, esp iphone, was realized. By the time I left in ‘10 the total revenue from mac hardware was like 15% of revenue. Im honestly surprised theres anyone who cared enough to package the business services for mac minis. So if everything else is printing cash for a HUGE addressable consumer market at premium price points why would they try and compete with their own ODMs on more-or-less commodity enterprise gear?
    - SoftTalker 11mo ago
      Seems like I remember the main reason Macs survived as a product at all was because you needed one to develop for iOS. That may be an exaggeration but there certainly was a time when Macs were few and far between outside of creative shops. Certainly they were almost unseen in the corporate world, where now they are fairly common at least in laptops.
      - pjmlp 11mo ago
        Macs survived because Apple got a cash injection, survived long enough to come out with colorful iMacs with an hockey puck mouse, still running on Mac OS 8, and the iPod. Requiring one for doing iOS development they were already back into the green.
        11mo ago
- angoragoats 11mo ago
  > Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with that, * And perhaps most importantly adds measurable latency to a networking stack that is already higher latency than the RDMA approach used by the TB5 setup in the OP.
  - fenced_load 11mo ago
    Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cloud_router.html/overview?ap=y&smp=Y&srsltid=AfmBOorjdgkxvg6LYd2W9GeOOQ8NXfmJAzGXk5fAejYsIV24uwzikaFfZ4k https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
    - angoragoats 11mo ago
      Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
      - nicky_nickell 11mo ago
        how significant is the latency hit?
        angoragoats 11mo ago
        The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
- zozbot234 11mo ago
  > Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of?
  - fooblaster 11mo ago
    Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
    - sroussey 11mo ago
      onnx supports CoreML, is that how?
  - liuliu 11mo ago
    They were talking about neural accelerators (a silicon piece on GPU): https://releases.drawthings.ai/p/metal-flashattention-v25-w-neural https://releases.drawthings.ai/p/metal-flashattention-v25-w-...
  - csdreamer7 11mo ago
    > Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. Or, Apple could pay for the engineers to add it.
    - ls612 11mo ago
      Apple already paid software engineers to add Tensorflow support for the ANE hardware.
  - pdpi 11mo ago
    > Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.
- Dylan16807 11mo ago
  > +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... M4 already hit the necessary speed per channel, and M5 is well above it. If they actually release an Ultra that much bandwidth is guaranteed on the full version. Even the smaller version with 25% fewer memory channels will be pretty close. We already know Max won't get anywhere near 1TB/s since Max is half of an Ultra.
- wtallis 11mo ago
  > Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/
  - QuantumNomad_ 11mo ago
    Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
    - sroussey 11mo ago
      I bet there is one piece of silicon per two ports.
    - geerlingguy 11mo ago
      From my brief discussion with Exo/Apple, it sounds like that is just a limitation of this initial rollout, but it's not a hardware limitation. Though, I am always leery to recommend any decisions be made over something that's not already proven to work, so I would say don't bet on all ports being able to be used. They very well may be able to though.
  - kappuchino 11mo ago
    He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one. He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17
    - re-thc 11mo ago
      > He just didn't want to buy another one. Wasn’t it loaned ie didn’t buy any at all? Apple should have loaned enough to flex.
- dev_l1x_be 11mo ago
  Mine is to remove the extreme Macos bloat.
- Marsymars 11mo ago
  > - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.
  - jauntywundrkind 11mo ago
    The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work, and doing so at double or triple the power budget is an egregious waste. So many of the chips about are already delivered at way higher than optimum efficiency, largely for bragging rights. The exponential decay of efficiency you keep pushing for is an anti-quest, is against good. The absolute performance wins are ridiculous to seek. In almost all cases. If your problem will not scale and dumping a ton of power into one GPU or one cpu socket is all you got, fine, your problem is bad and you have to deal with that. But for 90% of people, begging for more power proces you don't actually know jack & my personal recommendation is that all such points of view deserve massive down voting by anyone with half a brain. Go back to 2018 and look at Matthew Dillon on DragobflyBSD underpowering the heck out of their 2990wx ThreadRipper. Efficiency just soars as you tell the chip to take less power. The situation has not improved! Efficiency skyrockets today at least as much as it did then by telling chips not to go all out. Good chips behave & reward. I believe Apple competent enough to thoroughly disabuse this position that this chip would be far better if we could dump 2x 3x more power into it. Just a fools position, beyond a joke, imo. https://apollo.backplane.com/DFlyMisc/threadripper.txt https://apollo.backplane.com/DFlyMisc/threadripper.txt
    - Marsymars 11mo ago
- checker659 11mo ago
  For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/
  - rbanffy 11mo ago
    QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.
    - justincormack 11mo ago
      Also given that Apple are using these in their datacenters I think they will ship much more server like hw.
    - checker659 11mo ago
      Apple already ships an MLX5 driver for ConnectX NICs.
- lostmsu 11mo ago
  > 3090 would be nice They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacpp_benchmark_on_a100/ https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...
  - doctorpangloss 11mo ago
    the GB10 is only the same performance as a 3090. gb10 uses way less power. i'm not sure why anyone would buy a mac studio instead of a gb10 machine for this use case.
    - villgax 11mo ago
      it's just people looking to do experiments locally on the main machine rather than just get a dedicated spark, which can be used properly as a headless box than a Mac of which you are at the mercy of system shenanigans albiet still bearable compared to windows
    - rbanffy 11mo ago
      > i'm not sure why anyone would buy a mac studio instead of a gb10 For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS updates, and it was still going strong.
      - FuriouslyAdrift 11mo ago
        DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.
        rbanffy 11mo ago
- wartywhoa23 11mo ago
  Would you please mind leaving some RAM to remain available for purchase at an affordable price for us mere mortals ? 1Tb for what, like, "Come on AI, make the humankind happy now"? /"s"
  - RunSet 11mo ago
    AI bubble will do wonders for used RAM prices when it pops.
    - ddalex 11mo ago
      There is no popping. We cannot have enough compute for the forseeable future.
      - zhengyi13 11mo ago
        "The market can stay irrational longer than you can stay solvent"
      - SV_BubbleTime 11mo ago
        Look at this guy on his first ram shortage.
        burnte 11mo ago
        I've been in this game so long, seen so many shortages that I'm not even worried. Right now prices are high, manufacturers are switching production, and in 6 months there's going to be a glut of supply. It's all a long game, folks. Play it long.
        brianwawok 11mo ago
        Long game is fine for optional upgrades. “I really wish my game system had 20% better graphics”. Less good when your system crashes and you need something new to work on Monday.
        11mo ago
delaminator 11mo ago
> Working with some of these huge models, I can see how AI has some use, especially if it's under my own local control. But it'll be a long time before I put much trust in what I get out of it—I treat it like I do Wikipedia. Maybe good for a jumping-off point, but don't ever let AI replace your ability to think critically! It is a little sad that they gave someone an uber machine and this was the best he could come up with. Question answering is interesting but not the most interesting thing one can do, especially with a home rig. The realm of the possible Video generation: CogVideoX at full resolution, longer clips Mochi or Hunyuan Video with extended duration Image generation at scale: FLUX batch generation — 50 images simultaneously Fine-tuning: Actually train something — show LoRA on a 400B model, or full fine-tuning on a 70B but I suppose "You have it for the weekend" means chatbot go brrrrr and snark
- benjismith 11mo ago
  > show LoRA on a 400B model, or full fine-tuning on a 70B Yeah, that's what I wanted to see too.
- theshrike79 11mo ago
  Yea, I don't understand why people use LLMs for "facts". You can get them from Wikipedia or a book. Use them for something creative, write a short story on spec, generate images. Or the best option: give it tools and let it actually DO something like "read my message history with my wife, find top 5 gift ideas she might have hinted at and search for options to purchase them" - perfect for a local model, there's no way in hell I'd feed my messages to a public LLM, but the one sitting next to me that I can turn off the second it twitches the wrong way? - sure.
  - mft_ 11mo ago
    > Yea, I don't understand why people use LLMs for "facts". You can get them from Wikipedia or a book. Because web search is so broken these days, if you want a clean answer instead of wading through pages of SEO nonsense. It's really common (even) amongst non-techy friends that "I'll ask ChatGPT" has replaced "I'll Google it".
newsclues 11mo ago
https://m.youtube.com/watch?v=4l4UWZGxvoc https://m.youtube.com/watch?v=4l4UWZGxvoc Seems like the ecosystem is rapidly evolving
- A4ET8a8uTh0_v2 11mo ago
  What it kinda reminds me of is PS3 cluster era. Now if I could do something similar to the minisforum..
mmorse1217 11mo ago
Hey Jeff, wherever you are: this is awesome work! I’ve wanted to try something like this for a while and was very excited for the RDMA over thunderbolt news. But I mostly want to say thanks for everything you do. Your good vibes are deeply appreciated and you are an inspiration.
rahimnathwani 11mo ago
The largest nodes in his cluster each have 512GB RAM. DeepSeek V3.1 is a 671B parameter model whose weights take up 700GB RAM: https://huggingface.co/deepseek-ai/DeepSeek-V3.1 https://huggingface.co/deepseek-ai/DeepSeek-V3.1 I would have expected that going from one node (which can't hold the weights in RAM) to two nodes would have increased inference speed by more than the measured 32% (21.1t/s -> 27.8t/s). With no constraint on RAM (4 nodes) the inference speed is less than 50% faster than with only 512GB. Am I missing something?
- zeusk 11mo ago
  the TB5 link (RDMA) is much slower than direct access to system memory
- elorant 11mo ago
  You only get 80Gbps network bandwidth. There's your bottleneck right there. Infiniband in comparison can give you up to x10 times that.
  - storus 11mo ago
    I think the op meant pipeline parallelism where during inference you only transfer the activation between layers where you cut the model in two, which shouldn't be too large.
- zozbot234 11mo ago
  Weights are read-only data so they can just be memory mapped and reside on SSD (only a small fraction will be needed in VRAM at any given time), the real constraint is activations. MoE architecture should help quite a bit here.
  - hu3 11mo ago
    > only a small fraction will be needed in VRAM at any given time I don't think that's true. At least not without heavy performance loss in which case "just be memory mapped" is doing a lot of work here. By that logic GPUs could run models much larger than their VRAM would otherwise allow, which doesn't seem to be the case unless heavy quantization is involved.
deleted 11mo ago
[deleted]
forestmars 11mo ago
[dead]
lvl155 11mo ago
Seriously, Jeff has the best job. Him and STH Patrick.
- geerlingguy 11mo ago
  I got to spend a day with Patrick this week, and try out his massive CyPerf testing rig with multiple 800 Gbps ConnectX-8 cards!
  - lvl155 11mo ago
    Patrick’s enthusiasm is so contagious and you perfected tech YouTube format. There’s not a dead spot in your video.
andy99 11mo ago
Very cool, I’m probably thinking too much but why are they seemingly hyping this now (I’ve seen a bunch of this recently) with no M5 Max/Ultra machines in sight. Is it because their release is imminent (I have heard Q1 2026) or is it to try and stretch out demand for M4 Max / M3 Ultra. I plan to buy one (not four) but would feel like I’m buying something that’s going to be immediately out of date if I don’t wait for the M5.
- GeekyBear 11mo ago
  I imagine that they want to give developers time to get their RDMA support stabilized, so third party software will be ready to take advantage of RDMA when the M5 Ultra lands. I definitely would not be buying an M3 Ultra right now on my own dime.
  - spacedcowboy 11mo ago
    I am typing this on my own 512GB M3 Ultra. I've just put out some feelers for 2nd-hand sale price... I have an M4 Max I can use to bridge any gap...
- fooblaster 11mo ago
  Does it actually creates a unified memory pool? it looks more like an accelerated backend for a collective communications library like nccl, which is very much not unified memory.
- 9dev 11mo ago
  The yearly release cadence annoys me to no end. There is literally zero reason to have a new CPU generation every year, it just devalues Mac hardware faster. Which I guess is the point of this for Apple, but still.
chis 11mo ago
I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like this are downstream of that?
- vsgherzi 11mo ago
  last I heard for the private compute features they were racking and stacking m2 mac pros
  - jeffbee 11mo ago
    I honestly forgot they still made the Mac Pro. Amazing that they have these ready to ship on their website. But at a 50% premium over similar but faster Mac Studio models, what is the point? You can't usefully put GPUs in them as far as I know. You'd have to have a different PCIe need to make it make sense.
    - deleted 11mo ago
      [deleted]
    - Asmod4n 11mo ago
      all PCIe lanes combined in that machine can do over 1 terabit. Would be quite the networking beast.
      - jeffbee 11mo ago
        The M2 Ultra has 32 off-world PCIe lanes, 8 of which are obligated to the SSDs. That leaves only 24 lanes for the 7 slots. That's 8 times less than you'd get from an EPYC, which is the kind of thing a normal user would put in a rack if they did not need to use macos.
- xienze 11mo ago
Retr0id 11mo ago
I wonder if there's any possibility that an RDMA expansion device could exist in the future - i.e. a box full of RAM on the other end of a thunderbolt cable. Although I guess such a device would cost almost as much as a mac mini in any case...
- roadbuster 11mo ago
  You still need an interface which does at least two things: handles incoming read/write requests using some kind of network protocol, and operates as a memory controller for the RAM. Texas Memory Systems was in the business of making large 'RAM Drives'. They had a product line known as "RamSan" which made many gigabytes/terabytes of DDR available via a block storage interface over infiniband and fibre channel. The control layer was implemented via FPGA. I recall a press release from 2004 which publicized the US govt purchase of a 2.5TB RamSan. They later expanded into SSDs and were acquired by IBM in 2012. https://en.wikipedia.org/wiki/Texas_Memory_Systems https://en.wikipedia.org/wiki/Texas_Memory_Systems https://www.lhcomp.com/vendors/tms/TMS-RamSan300-DataSheet.pdf https://www.lhcomp.com/vendors/tms/TMS-RamSan300-DataSheet.p... https://gizmodo.com/u-s-government-purchases-worlds-largest-ram-disk-8754 https://gizmodo.com/u-s-government-purchases-worlds-largest-... https://www.lhcomp.com/vendors/tms/TMS-RamSan20-DataSheet.pdf https://www.lhcomp.com/vendors/tms/TMS-RamSan20-DataSheet.pd... https://www.ibm.com/support/pages/ibm-plans-acquire-texas-memory-systems https://www.ibm.com/support/pages/ibm-plans-acquire-texas-me...
- RantyDave 11mo ago
  Couldn't you "just" use a honking fast SSD and set it as a swap drive?
  - Retr0id 11mo ago
    You might get close in peak bandwidth, but not in random access and latency.
- amluto 11mo ago
  RDMA is not really intended for this. RDMA is really just a bunch of functionality of a PCIe device, and even PCIe isn’t really quite right to use like RAM because its cache semantics aren’t intended for this use case. But the industry knows this, and there’s a technology that is electrically compatible with PCIe that is intended for use as RAM among other things: CXL. I wonder if a anyone will ever build CXL over USB-C.
xmddmx 11mo ago
I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? Also, TB5 has to be wired so that every CPU is connected to every other over TB, limiting you to 4 macs. By comparison, with Ethernet, you could use a hub & spoke configuration with a Ethernet switch, theoretically letting you use more than 4 CPUs.
- MBCook 11mo ago
  That’s llama, which didn’t scale nearly as well in the tests. Assumedly because it’s not optimized yet. RDMA is always going to have lower overhead than Ethernet isn’t it?
  - Neywiny 11mo ago
    Possibly RDMA over thunderbolt. But for RoCE (RDMA over converged Ethernet) obviously not because it's sitting on top of Ethernet. Now that could still have a higher throughput when you factor in CPU time to run custom protocols that smart NICs could just DMA instead, but the overhead is still definitively higher
    - _zoltan_ 11mo ago
      what do you think "ethernet's overhead" is?
      - Neywiny 11mo ago
        Header and FCS, interpacket gap, and preamble. What do you think "Ethernet overhead" is?
        _zoltan_ 11mo ago
polynomial 11mo ago
BUILD AI has a post about this and in particular sharding k-v cache across GPUs, and how network is the new memory hierarchy: https://buildai.substack.com/p/kv-cache-sharding-and-distributed https://buildai.substack.com/p/kv-cache-sharding-and-distrib...
gloyoyo 11mo ago
Wow. $40k for a friendly chat(bot)... Hey, at least this post allows us to feel as though we spent the money ourselves. Bravo!
simonw 11mo ago
I'd be interested in seeing numbers that split out the speed of reading input (aka prefill) and the speed of generating output (aka decode). Those numbers are usually different and I remember from this Exo article that they could be quite radically different on Mac hardware: https://blog.exolabs.net/nvidia-dgx-spark/ https://blog.exolabs.net/nvidia-dgx-spark/
- geerlingguy 11mo ago
  See https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 for more data — I didn't save all the prompt processing times (Exo just outputs a time in ms, no other data for that), but will try to have another pass. Maybe also convince the Exo team to add a proper benchmarking capability ala `llama-bench` :)
  - jonrouach 11mo ago
    or better, like you mentioned, try to convince Exo to develop in the open, so everyone gets any capability as PRs.
    - geerlingguy 11mo ago
      They are now, this morning they pushed all the code to the Exo repo, and archived the earlier Exo branch. We'll see how open they are now that whatever embargoed work they did with Apple is public..
deleted 11mo ago
[deleted]
periodjet 11mo ago
> That's definitely fast enough for vibe coding, if that's your thing, but it's not mine. Why even…?
mikestaas 11mo ago
> You have to click buttons in the UI. I like doing development work on a Mac, but this has to be my biggest bugbear with the system.
jauntywundrkind 11mo ago
I really hope AMD or Intel can get on the clue train and respond. Intel in particular has half a decade of having extremely amazing Thunderbolt ports on their mobile chips, built in (alas not present on desktop chips, for shame). There's been not bad but not great thunderbolt host-to-host networking, that TCP can go over, but the system to system connectivity had been a total afterthought, not at all tuned for obvious smart readily available options like RDMA here. But nothing stops anyone from having better host-to-host protocols. There are also so many smart good excellent next steps competitors could go for. CXL is showing up on server systems as a much lighter weight much lower latency transport that is PCIe PHY compatible but lighter weight. Adding this to consumer chips and giving even a third of a shit could blow what we see here out of the water. It could probably be done over USB4 & radically blast this bespoke RDMA capability. Connectivity had been a bespoke special capability for too long. Intel did amazing with Xeon having integrated OmniPath 100Gb a long time ago, that was amazing, for barely any extra bucks. But the market didn't reward them kicking total ass and everyone gave up on connecting chips together. Today we are hostage to fantastically expensive shitty inefficient NIC that cost a crap ton of money to do a worse job, paying enormous penalty for not having the capability on chip, making at best asmedia io hubs do the USB4 dance a hip away from the CPU. I really hope Intel can appreciate how good they were, see the threat of Apple kicking as here doing what Intel uniquely has been offering for half a decade with incredible Thunderbolt offerings on-chip (limited alas only to mobile chips). I hope AMD feels the heat and gets some god dMned religion and sees the pressure and thread: man they delivered so strong on PCIe lane counts but man they have been so so so slacking on io capabilities for so long, especially on consumer platforms, and Apple is using both their awesome awesome awesome on-chip memory here and their fan-tastic exceptional ability to care just even the tiniest bit about using the consumer interconnect (that already exists in hardware). I really really really hope someone else other than Apple can ante up and care. There are so many wins to be had, so close. These companies feel so distracted from the plot. Fucking shame. Good on Apple for being the only mofos to a Tually seize the obvious that was just sitting here, they took no effort nor innovation. What a shame no other players are trying at all.
- PunchyHamster 11mo ago
  Intel is allergic for making consumer stuff good. Remember how in consumer range like half of the chips had fucking virtualisation disabled, long after competition had it on everything ?
CaliforniaKarl 11mo ago
The "all nodes connecting to all other nodes" setup reminds me of NUMALink, the interconnect that SGI used on many (most? all?) of their supercomputers. In an ideal configuration, each 4-socket node has two NUMALink connections to every other node. As Jeff says, it's a ton of cables, and you don't have to think of framing or congestion in the same way as with RDMA over Ethernet.
- T3OU-736 11mo ago
  SGI's HW also had ccNUMA (cache-coherent Non-Uniform Memory Access), which, given the latencies possible in systems _physically_ spanning entire rooms, was quite a feat. The IRIX OS even had functionality to migrate kobs and theor working memory closer to each other to lower the latency of access. We see echoes of this when companies like high-frequency traders pay attention to motherboard layouts and co-locate and pin the PTS (proprietary trading systems) processes to specific cores based on which DIMMs are on which side of the memory controller.
- _zoltan_ 11mo ago
  just as an NVL72 rack today has 7271 links (18 probably) in the rack connecting all those GPUs together.
poemxo 11mo ago
Really cool article, I liked these details that weren't exactly related to the thesis: - the mysterious disappearance of Exo - Jeff wants something like SMB Direct but for the Mac. Wait what? SMB Direct is a thing, wha?? I always thought networked storage was untrustworthy. - A single M3 Ultra is fast for inference - A framework desktop ai max 395 is only $2100 Now I have some more rabbit holes to jump down.
fortran77 11mo ago
On Intel Motherboards, it's easy to find ones that can take 2TB of RAM, for example: https://www.supermicro.com/en/products/motherboard/x14sbw-tf https://www.supermicro.com/en/products/motherboard/x14sbw-tf This seems suboptimal.
- TheTxT 11mo ago
  The gpu can’t access that directly however. On Apple Silicon it can all be used as vram.
- ErneX 11mo ago
  2TB of system memory, not unified memory like Apple Silicon Macs.
dsrtslnd23 11mo ago
Any thoughts on the GB300 workstation with 768GB RAM (from NVIDA, Asus, Dell, ...)? Although many announcements were made it seems not to be available yet. It does have faster interconnects but will probably be much more expensive.
cboyardee 11mo ago
[dead]
polsevev 11mo ago
As much as i hate Apples attitude towards hackers and modifying systems. I have to commend them for building awesome features like this
Rakshath_1 11mo ago
[flagged]
- tomhow 11mo ago
  Please stop.
Tepix 11mo ago
Linux already has RDMA support but it cannot yet use Thunderbolt. It's probably quite a bit of work to add everything that's required. Is anyone working on it? It would be great to have this for those cheap Strix Halo boxes with 128GB quad channel DDR5-8000 for using two or three of them with their 2 USB4 ports (which are Thunderbolt capable) to fit larger models.
- dylan604 11mo ago
  > Linux already has RDMA support but it cannot yet use Thunderbolt. Is TB still encumbered by licensing requirements causing this lack of use?
- yonatan8070 11mo ago
  I assume that's Thunderbolt 5? From my experience, eGPUs over USB 4/TB3 work just fine (from a technical point of view, in practice 40Gbps isn't enough BW and performance is shit)
  - Tepix 11mo ago
    Yes, that works but what's needed here is remotely accessing RAM on another PC via RDMA via Thunderbolt. Not accessing a standalone eGPU.
pjmlp 11mo ago
In an ideal world, Apple would have released a Mac Pro with card slots for doing this kind of stuff. Instead we get gimmicks over Thunderbolt.
- g947o 11mo ago
  I can imagine Apple shipping Mac Pros with add-ons that allows running local inference with minimal setups. "Look, just spend $50k on this machine and you get a usable LLM server that can be shared for a team." But they don't seem particularly interested in that market.
clan 11mo ago
As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster. But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rather than TCP. 10G Thunderbolt adapters are fairly common. But you can find 40G and 80G Thunderbolt ethernet adapters from Atto. Probably not cheap - but would be fun to test! But ieven if the bandwidth is there we might get killed with latency. Imagine this hardware with a PCIe slot. The Infiniband hardware is there - then we "just" need the driver.
- KeplerBoy 11mo ago
  At that point you could just breakout the thunderbolt to PCIe and use a regular NIC. Actually, I'm pretty sure that's all that to the Atto Thunderlink, a case around a broadcom nic. Then you _just_ need the driver. Fascinating, Apple ships MLX5 drivers, that's crazy imo. I understand that's something they might need internally, but shipping that on ipadOs is wild. https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/
  - clan 11mo ago
    That is what I am suggesting with the Atto adapter. Infiniband is way faster and lower latency than a NIC. These days NIC==Ethernet.
    - KeplerBoy 11mo ago
      What makes you think Infiniband is faster than Ethernet? Aren't they pretty much equal these days with RDMA and kernel bypass?
- jamesfmilne 11mo ago
  macOS ships with drivers for Mellanox ConnectX cards, but I have no idea if they will show up in `ibv_devices` or `ibv_devinfo`.
XCSme 11mo ago
Anyone else getting ERR_TOO_MANY_REDIRECTS trying to access the post?
- pstoll 11mo ago
  Yes only on that page, not the rest of his blog. Guessing he ansible’d it to redirect ;)
- moebrowne 11mo ago
  https://web.archive.org/web/20251219055833/https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5 https://web.archive.org/web/20251219055833/https://www.jeffg...
- XCSme 11mo ago
  It's working now.
bluedino 11mo ago
There have a been a couple videos/posts about this from other influencers today Does anyone remember a guy here posting about linking Mac Studios with Thunderbolt for HPC/clustering? I wasn't able to find it with a quick search. Edit: I think it was this? https://www.youtube.com/watch?v=d8yS-2OyJhw https://www.youtube.com/watch?v=d8yS-2OyJhw
daft_pink 11mo ago
The next Mac studio is going to be a top seller. I don’t think people want to drop $10k on a few M3s, but I think they will do it for the M6. Just hoping the DRAM shortage doesn’t ruin this plan.
- oofbey 11mo ago
  Apple always charges a huge premium for RAM. Maybe it’s enough to buffer their pricing scheme from the supply shock. I have run the numbers though.
  - terhechte 11mo ago
    Tim Cook is famous for locking in their prices years in advance.
dogcowmoof 11mo ago
Wonder if RDMA support can translate to things like SMB direct or other RDMA adjacent things
dogcowmoof 11mo ago
Wonder if support for RDMA will translate into support for things such as SMB Direct or if it's really only useful for RAM pooling
krbaccord94f 11mo ago
rdma_ctl enable in 1tn parameter. TL1 mount, where 1.5 TB allocate mac-mini server.
e28eta 11mo ago
> For example: did you know there's no way to run a system upgrade (like to 26.2) via SSH I did not know this. I thought the `softwareupdate` command was built for this use case, and thought it worked over ssh. It sure looks like it should work, but I don’t have a mac I can try it on right now.
- pudquick 11mo ago
  He's wrong, it's possible. It's just that root privileges alone is insufficient due to how the signing on LocalPolicy works on M series Macs https://support.apple.com/guide/security/contents-a-localpolicy-file-mac-apple-silicon-secc745a0845/1/web/1#secb7cb6686c https://support.apple.com/guide/security/contents-a-localpol... The manpage for the command provides information on credential usage on Apple Silicon devices.
supermatt 11mo ago
What is the max token throughput when batching. Lots of agentic workflows (not just vibe coding) are running many inferences in parallel. It seems like every time someone does an AI hardware “review” we end up with figures for just a single instance, which simply isn’t how the target demographic for a 40k cluster are going to be using it. Jeff, I love reading your reviews, but can’t help but feel this was a wasted opportunity for some serious benchmarking of LLM performance.
saddat 11mo ago
A good part of humanities knowledge under your desk running with a few old light bulbs worth of power
extraduder_ire 11mo ago
Is RDMA only going to be on the studio, or is it coming to anything with a thunderbolt 5 port on it?

617 points by rbanffy 11mo ago

My expectations from M5 Max/Ultra devices: - Something like DGX QSFP link (200Gb/s, 400Gb/s) instead of TB5. Otherwise, the economies of this RDMA setup, while impressive, don't make sense. - Neural accelerators to get prompt prefill time down. I don't expect RTX 6000 Pro speeds, but something like 3090/4090 would be nice. - 1TB of unified memory in the maxed out version of Mac Studio. I'd rather invest in more RAM than more devices (centralized will always be faster than distributed). - +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP.

tylerflick 11mo ago
> TOTALLY okay with it consuming +600W energy The 2019 i9 Macbook Pro has entered the chat.
burnt-resistor 11mo ago
Apple has always sucked at properly embracing properly robust tech for high-end gear for markets outside of individual prosumer or creatives. When Xserves existed, they used commodity IDE drives without HA or replaceable PSUs that couldn't compete with contemporary enterprise servers (HP-Compaq/Dell/IBM/Fujitsu). Xserve RAID interconnection half-heartedly used fiber channel but couldn't touch a NetApp or EMC SAN/filer. I'm disappointed Apple has a persistent blindspot preventing them from succeeding in data center-quality gear category when they could've had virtualized servers, networking, and storage, things that would eventually find their way into my home lab after 5-7 years.
- Terretta 11mo ago
  > I'm disappointed Apple has a persistent blindspot preventing them from succeeding in ... things that would eventually find their way into my home lab after 5-7 years. I can see the dollar signs in their eyes right now. Aftermarkets are a nice reflection of durable value, and there's a massive one for iPhones and a smaller one for quick flameout startup servers, but not much money in 5 - 7 year old servers.
- donavanm 11mo ago
  Enterprise never ever mattered, and there arent enough digits available to show your “home lab” use case in the revenue numbers. Xserve, the RAID shelves, and the directory services were kinda there as a half hearted attempt for that late 90-00s AV setup. All of that fell on the cutting room floor once personal devices, esp iphone, was realized. By the time I left in ‘10 the total revenue from mac hardware was like 15% of revenue. Im honestly surprised theres anyone who cared enough to package the business services for mac minis. So if everything else is printing cash for a HUGE addressable consumer market at premium price points why would they try and compete with their own ODMs on more-or-less commodity enterprise gear?
  - SoftTalker 11mo ago
    Seems like I remember the main reason Macs survived as a product at all was because you needed one to develop for iOS. That may be an exaggeration but there certainly was a time when Macs were few and far between outside of creative shops. Certainly they were almost unseen in the corporate world, where now they are fairly common at least in laptops.
    - pjmlp 11mo ago
      Macs survived because Apple got a cash injection, survived long enough to come out with colorful iMacs with an hockey puck mouse, still running on Mac OS 8, and the iPod. Requiring one for doing iOS development they were already back into the green.
      - 11mo ago
angoragoats 11mo ago
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP. This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which: * Adds thousands of dollars of cost, * Adds 150W or more of power usage and the accompanying loud fan noise that comes with that, * And perhaps most importantly adds measurable latency to a networking stack that is already higher latency than the RDMA approach used by the TB5 setup in the OP.
- fenced_load 11mo ago
  Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cloud_router.html/overview?ap=y&smp=Y&srsltid=AfmBOorjdgkxvg6LYd2W9GeOOQ8NXfmJAzGXk5fAejYsIV24uwzikaFfZ4k https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...
  - angoragoats 11mo ago
    Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
    - nicky_nickell 11mo ago
      how significant is the latency hit?
      - angoragoats 11mo ago
        The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
zozbot234 11mo ago
> Neural accelerators to get prompt prefill time down. Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. > this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of?
- fooblaster 11mo ago
  Might be helpful if they actually provided a programming model for ANE that isn't onnx. ANE not having a native development model just means software support will not be great.
  - sroussey 11mo ago
    onnx supports CoreML, is that how?
- liuliu 11mo ago
  They were talking about neural accelerators (a silicon piece on GPU): https://releases.drawthings.ai/p/metal-flashattention-v25-w-neural https://releases.drawthings.ai/p/metal-flashattention-v25-w-...
- csdreamer7 11mo ago
  > Apple Neural Engine is a thing already, with support for multiply-accumulate on INT8 and FP16. AI inference frameworks need to add support for it. Or, Apple could pay for the engineers to add it.
  - ls612 11mo ago
    Apple already paid software engineers to add Tensorflow support for the ANE hardware.
- pdpi 11mo ago
  > Do you really need a fully connected mesh? Doesn't Thunderbolt just show up as a network connection that RDMA is ran on top of? If you daisy chain four nodes, then traffic between nodes #1 and #4 eat up all of nodes #2 and #3's bandwidth, and you eat a big latency penalty. So, absent a switch, the fully connected mesh is the only way to have fast access to all the memory.
Dylan16807 11mo ago
> +1TB/s bandwidth. For the past 3 generations, the speed has been 800GB/s... M4 already hit the necessary speed per channel, and M5 is well above it. If they actually release an Ultra that much bandwidth is guaranteed on the full version. Even the smaller version with 25% fewer memory channels will be pretty close. We already know Max won't get anywhere near 1TB/s since Max is half of an Ultra.
wtallis 11mo ago
> Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac I do wonder where this limitation comes from, since on the M3 Ultra Mac Studios the front USB-C ports are also Thunderbolt 5, for a total of six Thunderbolt ports: https://www.apple.com/mac-studio/specs/ https://www.apple.com/mac-studio/specs/
- QuantumNomad_ 11mo ago
  Jeff mentioned in the video that only three of the ports can be used for RDMA. But it’s unclear where that limitation is coming from.
  - sroussey 11mo ago
    I bet there is one piece of silicon per two ports.
  - geerlingguy 11mo ago
    From my brief discussion with Exo/Apple, it sounds like that is just a limitation of this initial rollout, but it's not a hardware limitation. Though, I am always leery to recommend any decisions be made over something that's not already proven to work, so I would say don't bet on all ports being able to be used. They very well may be able to though.
- kappuchino 11mo ago
  He corrected that in the comment section of the youtube video. Six is actually the maximum amount. He just didn't want to buy another one. He also published the Benchmarks in Detail and with two/four Macs in Comparison: https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17
  - re-thc 11mo ago
    > He just didn't want to buy another one. Wasn’t it loaned ie didn’t buy any at all? Apple should have loaned enough to flex.
dev_l1x_be 11mo ago
Mine is to remove the extreme Macos bloat.
Marsymars 11mo ago
> - The ability to overclock the system? I know it probably will never happen, but my expectation of Mac Studio is not the same as a laptop, and I'm TOTALLY okay with it consuming +600W energy. Currently it's capped at ~250W. I don't think the Mac Studio has a thermal design capable of dissipating 650W of heat for anything other than bursty workloads. Need to look at the Mac Pro design for that.
- jauntywundrkind 11mo ago
  The thermal design is irrelevant, and people saying they want insane power density are, in my personal view, deluded ridiculous individuals who understand very very little. Overclocking long ago was an amazing saintly act, milking a lot of extra performance that was just there waiting, without major downsides to take. But these days, chips are usually already well tuned. You can feed double or tripple the power into the chip with adequate cooling, but the gain is so unremarkable. +10% +15% +20% is almost never going to be a make or break difference for your work, and doing so at double or triple the power budget is an egregious waste. So many of the chips about are already delivered at way higher than optimum efficiency, largely for bragging rights. The exponential decay of efficiency you keep pushing for is an anti-quest, is against good. The absolute performance wins are ridiculous to seek. In almost all cases. If your problem will not scale and dumping a ton of power into one GPU or one cpu socket is all you got, fine, your problem is bad and you have to deal with that. But for 90% of people, begging for more power proces you don't actually know jack & my personal recommendation is that all such points of view deserve massive down voting by anyone with half a brain. Go back to 2018 and look at Matthew Dillon on DragobflyBSD underpowering the heck out of their 2990wx ThreadRipper. Efficiency just soars as you tell the chip to take less power. The situation has not improved! Efficiency skyrockets today at least as much as it did then by telling chips not to go all out. Good chips behave & reward. I believe Apple competent enough to thoroughly disabuse this position that this chip would be far better if we could dump 2x 3x more power into it. Just a fools position, beyond a joke, imo. https://apollo.backplane.com/DFlyMisc/threadripper.txt https://apollo.backplane.com/DFlyMisc/threadripper.txt
  - Marsymars 11mo ago
checker659 11mo ago
For a company that has repeatedly ignored macOS, your wishlist seems anything but a pipe dream. QSFP on a mac. Yeah right. If anything, they’ll double down on TB or some nonstandard interconnect. What is a computer? (Although, I do hope with the new work on supporting RDMA, the MLX5 driver shipped with macOS will finally support RDMA for ConnectX NICs) https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/
- rbanffy 11mo ago
  QSFP makes sense on a MacPro platform - and might be where Apple chooses to differentiate (one could dream of an M5 Mega, with four chiplets). The Mac Studio is a general purpose compact workstation that doesn’t need ludicrously fast networking beyond what 10Gbe and TB5 offer. It’s already overkill for the vast majority of users. Top configuration Studios are already a niche product.
  - justincormack 11mo ago
    Also given that Apple are using these in their datacenters I think they will ship much more server like hw.
  - checker659 11mo ago
    Apple already ships an MLX5 driver for ConnectX NICs.
lostmsu 11mo ago
> 3090 would be nice They would need 3x speedup over the current generation to approach 3090. A100 that has +- the 3090 compute but 80GB VRAM (so fits LLaMA 70B) does prefill at 550tok/s on a single GPU: https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacpp_benchmark_on_a100/ https://www.reddit.com/r/LocalLLaMA/comments/1ivc6vv/llamacp...
- doctorpangloss 11mo ago
  the GB10 is only the same performance as a 3090. gb10 uses way less power. i'm not sure why anyone would buy a mac studio instead of a gb10 machine for this use case.
  - villgax 11mo ago
    it's just people looking to do experiments locally on the main machine rather than just get a dedicated spark, which can be used properly as a headless box than a Mac of which you are at the mercy of system shenanigans albiet still bearable compared to windows
  - rbanffy 11mo ago
    > i'm not sure why anyone would buy a mac studio instead of a gb10 For an AI-only use case, the GB10s make sense, but they are only OK as desktop workstations, and I’m not sure for how long DGX OS will be updated, as dedicated AI machines have somewhat short lives. Apple computers, OTOH, have much longer lives, and desktops live the longest. I retired my Mac Mini a year after the machine was no longer getting OS updates, and it was still going strong.
    - FuriouslyAdrift 11mo ago
      DGX OS (based on Ubuntu) is used for all of nVidias GPU compute systems so it's probably going to be around for a while.
      - rbanffy 11mo ago
wartywhoa23 11mo ago
Would you please mind leaving some RAM to remain available for purchase at an affordable price for us mere mortals ? 1Tb for what, like, "Come on AI, make the humankind happy now"? /"s"
- RunSet 11mo ago
  AI bubble will do wonders for used RAM prices when it pops.
  - ddalex 11mo ago
    There is no popping. We cannot have enough compute for the forseeable future.
    - zhengyi13 11mo ago
      "The market can stay irrational longer than you can stay solvent"
    - SV_BubbleTime 11mo ago
      Look at this guy on his first ram shortage.
      - burnte 11mo ago
        I've been in this game so long, seen so many shortages that I'm not even worried. Right now prices are high, manufacturers are switching production, and in 6 months there's going to be a glut of supply. It's all a long game, folks. Play it long.
        brianwawok 11mo ago
        Long game is fine for optional upgrades. “I really wish my game system had 20% better graphics”. Less good when your system crashes and you need something new to work on Monday.
        11mo ago

delaminator 11mo ago

> Working with some of these huge models, I can see how AI has some use, especially if it's under my own local control. But it'll be a long time before I put much trust in what I get out of it—I treat it like I do Wikipedia. Maybe good for a jumping-off point, but don't ever let AI replace your ability to think critically! It is a little sad that they gave someone an uber machine and this was the best he could come up with. Question answering is interesting but not the most interesting thing one can do, especially with a home rig. The realm of the possible Video generation: CogVideoX at full resolution, longer clips Mochi or Hunyuan Video with extended duration Image generation at scale: FLUX batch generation — 50 images simultaneously Fine-tuning: Actually train something — show LoRA on a 400B model, or full fine-tuning on a 70B but I suppose "You have it for the weekend" means chatbot go brrrrr and snark

benjismith 11mo ago
> show LoRA on a 400B model, or full fine-tuning on a 70B Yeah, that's what I wanted to see too.
theshrike79 11mo ago
Yea, I don't understand why people use LLMs for "facts". You can get them from Wikipedia or a book. Use them for something creative, write a short story on spec, generate images. Or the best option: give it tools and let it actually DO something like "read my message history with my wife, find top 5 gift ideas she might have hinted at and search for options to purchase them" - perfect for a local model, there's no way in hell I'd feed my messages to a public LLM, but the one sitting next to me that I can turn off the second it twitches the wrong way? - sure.
- mft_ 11mo ago
  > Yea, I don't understand why people use LLMs for "facts". You can get them from Wikipedia or a book. Because web search is so broken these days, if you want a clean answer instead of wading through pages of SEO nonsense. It's really common (even) amongst non-techy friends that "I'll ask ChatGPT" has replaced "I'll Google it".

newsclues 11mo ago

https://m.youtube.com/watch?v=4l4UWZGxvoc https://m.youtube.com/watch?v=4l4UWZGxvoc Seems like the ecosystem is rapidly evolving

A4ET8a8uTh0_v2 11mo ago
What it kinda reminds me of is PS3 cluster era. Now if I could do something similar to the minisforum..

mmorse1217 11mo ago

Hey Jeff, wherever you are: this is awesome work! I’ve wanted to try something like this for a while and was very excited for the RDMA over thunderbolt news. But I mostly want to say thanks for everything you do. Your good vibes are deeply appreciated and you are an inspiration.

rahimnathwani 11mo ago

The largest nodes in his cluster each have 512GB RAM. DeepSeek V3.1 is a 671B parameter model whose weights take up 700GB RAM: https://huggingface.co/deepseek-ai/DeepSeek-V3.1 https://huggingface.co/deepseek-ai/DeepSeek-V3.1 I would have expected that going from one node (which can't hold the weights in RAM) to two nodes would have increased inference speed by more than the measured 32% (21.1t/s -> 27.8t/s). With no constraint on RAM (4 nodes) the inference speed is less than 50% faster than with only 512GB. Am I missing something?

zeusk 11mo ago
the TB5 link (RDMA) is much slower than direct access to system memory
elorant 11mo ago
You only get 80Gbps network bandwidth. There's your bottleneck right there. Infiniband in comparison can give you up to x10 times that.
- storus 11mo ago
  I think the op meant pipeline parallelism where during inference you only transfer the activation between layers where you cut the model in two, which shouldn't be too large.
zozbot234 11mo ago
Weights are read-only data so they can just be memory mapped and reside on SSD (only a small fraction will be needed in VRAM at any given time), the real constraint is activations. MoE architecture should help quite a bit here.
- hu3 11mo ago
  > only a small fraction will be needed in VRAM at any given time I don't think that's true. At least not without heavy performance loss in which case "just be memory mapped" is doing a lot of work here. By that logic GPUs could run models much larger than their VRAM would otherwise allow, which doesn't seem to be the case unless heavy quantization is involved.

deleted 11mo ago

[deleted]

forestmars 11mo ago

[dead]

lvl155 11mo ago

Seriously, Jeff has the best job. Him and STH Patrick.

geerlingguy 11mo ago
I got to spend a day with Patrick this week, and try out his massive CyPerf testing rig with multiple 800 Gbps ConnectX-8 cards!
- lvl155 11mo ago
  Patrick’s enthusiasm is so contagious and you perfected tech YouTube format. There’s not a dead spot in your video.

andy99 11mo ago

Very cool, I’m probably thinking too much but why are they seemingly hyping this now (I’ve seen a bunch of this recently) with no M5 Max/Ultra machines in sight. Is it because their release is imminent (I have heard Q1 2026) or is it to try and stretch out demand for M4 Max / M3 Ultra. I plan to buy one (not four) but would feel like I’m buying something that’s going to be immediately out of date if I don’t wait for the M5.

GeekyBear 11mo ago
I imagine that they want to give developers time to get their RDMA support stabilized, so third party software will be ready to take advantage of RDMA when the M5 Ultra lands. I definitely would not be buying an M3 Ultra right now on my own dime.
- spacedcowboy 11mo ago
  I am typing this on my own 512GB M3 Ultra. I've just put out some feelers for 2nd-hand sale price... I have an M4 Max I can use to bridge any gap...
fooblaster 11mo ago
Does it actually creates a unified memory pool? it looks more like an accelerated backend for a collective communications library like nccl, which is very much not unified memory.
9dev 11mo ago
The yearly release cadence annoys me to no end. There is literally zero reason to have a new CPU generation every year, it just devalues Mac hardware faster. Which I guess is the point of this for Apple, but still.

chis 11mo ago

I wonder what motivates apple to release features like RDMA which are purely useful for server clusters, while ignoring basic qol stuff like remote management or rack mount hardware. It’s difficult to see it as a cohesive strategy. Makes one wonder what apple uses for their own servers. I guess maybe they have some internal M-series server product they just haven’t bothered to release to the public, and features like this are downstream of that?

vsgherzi 11mo ago
last I heard for the private compute features they were racking and stacking m2 mac pros
- jeffbee 11mo ago
  I honestly forgot they still made the Mac Pro. Amazing that they have these ready to ship on their website. But at a 50% premium over similar but faster Mac Studio models, what is the point? You can't usefully put GPUs in them as far as I know. You'd have to have a different PCIe need to make it make sense.
  - deleted 11mo ago
    [deleted]
  - Asmod4n 11mo ago
    all PCIe lanes combined in that machine can do over 1 terabit. Would be quite the networking beast.
    - jeffbee 11mo ago
      The M2 Ultra has 32 off-world PCIe lanes, 8 of which are obligated to the SSDs. That leaves only 24 lanes for the 7 slots. That's 8 times less than you'd get from an EPYC, which is the kind of thing a normal user would put in a rack if they did not need to use macos.
xienze 11mo ago

Retr0id 11mo ago

I wonder if there's any possibility that an RDMA expansion device could exist in the future - i.e. a box full of RAM on the other end of a thunderbolt cable. Although I guess such a device would cost almost as much as a mac mini in any case...

roadbuster 11mo ago
You still need an interface which does at least two things: handles incoming read/write requests using some kind of network protocol, and operates as a memory controller for the RAM. Texas Memory Systems was in the business of making large 'RAM Drives'. They had a product line known as "RamSan" which made many gigabytes/terabytes of DDR available via a block storage interface over infiniband and fibre channel. The control layer was implemented via FPGA. I recall a press release from 2004 which publicized the US govt purchase of a 2.5TB RamSan. They later expanded into SSDs and were acquired by IBM in 2012. https://en.wikipedia.org/wiki/Texas_Memory_Systems https://en.wikipedia.org/wiki/Texas_Memory_Systems https://www.lhcomp.com/vendors/tms/TMS-RamSan300-DataSheet.pdf https://www.lhcomp.com/vendors/tms/TMS-RamSan300-DataSheet.p... https://gizmodo.com/u-s-government-purchases-worlds-largest-ram-disk-8754 https://gizmodo.com/u-s-government-purchases-worlds-largest-... https://www.lhcomp.com/vendors/tms/TMS-RamSan20-DataSheet.pdf https://www.lhcomp.com/vendors/tms/TMS-RamSan20-DataSheet.pd... https://www.ibm.com/support/pages/ibm-plans-acquire-texas-memory-systems https://www.ibm.com/support/pages/ibm-plans-acquire-texas-me...
RantyDave 11mo ago
Couldn't you "just" use a honking fast SSD and set it as a swap drive?
- Retr0id 11mo ago
  You might get close in peak bandwidth, but not in random access and latency.
amluto 11mo ago
RDMA is not really intended for this. RDMA is really just a bunch of functionality of a PCIe device, and even PCIe isn’t really quite right to use like RAM because its cache semantics aren’t intended for this use case. But the industry knows this, and there’s a technology that is electrically compatible with PCIe that is intended for use as RAM among other things: CXL. I wonder if a anyone will ever build CXL over USB-C.

xmddmx 11mo ago

I was impressed by the lack of dominance of Thunderbolt: "Next I tested llama.cpp running AI models over 2.5 gigabit Ethernet versus Thunderbolt 5" Results from that graph showed only a ~10% benefit from TB5 vs. Ethernet. Note: The M3 studios support 10Gbps ethernet, but that wasn't tested. Instead it was tested using 2.5Gbps ethernet. If 2.5G ethernet was only 10% slower than TB, how would 10G Ethernet have fared? Also, TB5 has to be wired so that every CPU is connected to every other over TB, limiting you to 4 macs. By comparison, with Ethernet, you could use a hub & spoke configuration with a Ethernet switch, theoretically letting you use more than 4 CPUs.

MBCook 11mo ago
That’s llama, which didn’t scale nearly as well in the tests. Assumedly because it’s not optimized yet. RDMA is always going to have lower overhead than Ethernet isn’t it?
- Neywiny 11mo ago
  Possibly RDMA over thunderbolt. But for RoCE (RDMA over converged Ethernet) obviously not because it's sitting on top of Ethernet. Now that could still have a higher throughput when you factor in CPU time to run custom protocols that smart NICs could just DMA instead, but the overhead is still definitively higher
  - _zoltan_ 11mo ago
    what do you think "ethernet's overhead" is?
    - Neywiny 11mo ago
      Header and FCS, interpacket gap, and preamble. What do you think "Ethernet overhead" is?
      - _zoltan_ 11mo ago

polynomial 11mo ago

BUILD AI has a post about this and in particular sharding k-v cache across GPUs, and how network is the new memory hierarchy: https://buildai.substack.com/p/kv-cache-sharding-and-distributed https://buildai.substack.com/p/kv-cache-sharding-and-distrib...

gloyoyo 11mo ago

Wow. $40k for a friendly chat(bot)... Hey, at least this post allows us to feel as though we spent the money ourselves. Bravo!

simonw 11mo ago

I'd be interested in seeing numbers that split out the speed of reading input (aka prefill) and the speed of generating output (aka decode). Those numbers are usually different and I remember from this Exo article that they could be quite radically different on Mac hardware: https://blog.exolabs.net/nvidia-dgx-spark/ https://blog.exolabs.net/nvidia-dgx-spark/

geerlingguy 11mo ago
See https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 https://github.com/geerlingguy/beowulf-ai-cluster/issues/17 for more data — I didn't save all the prompt processing times (Exo just outputs a time in ms, no other data for that), but will try to have another pass. Maybe also convince the Exo team to add a proper benchmarking capability ala `llama-bench` :)
- jonrouach 11mo ago
  or better, like you mentioned, try to convince Exo to develop in the open, so everyone gets any capability as PRs.
  - geerlingguy 11mo ago
    They are now, this morning they pushed all the code to the Exo repo, and archived the earlier Exo branch. We'll see how open they are now that whatever embargoed work they did with Apple is public..

deleted 11mo ago

[deleted]

periodjet 11mo ago

> That's definitely fast enough for vibe coding, if that's your thing, but it's not mine. Why even…?

mikestaas 11mo ago

> You have to click buttons in the UI. I like doing development work on a Mac, but this has to be my biggest bugbear with the system.

jauntywundrkind 11mo ago

I really hope AMD or Intel can get on the clue train and respond. Intel in particular has half a decade of having extremely amazing Thunderbolt ports on their mobile chips, built in (alas not present on desktop chips, for shame). There's been not bad but not great thunderbolt host-to-host networking, that TCP can go over, but the system to system connectivity had been a total afterthought, not at all tuned for obvious smart readily available options like RDMA here. But nothing stops anyone from having better host-to-host protocols. There are also so many smart good excellent next steps competitors could go for. CXL is showing up on server systems as a much lighter weight much lower latency transport that is PCIe PHY compatible but lighter weight. Adding this to consumer chips and giving even a third of a shit could blow what we see here out of the water. It could probably be done over USB4 & radically blast this bespoke RDMA capability. Connectivity had been a bespoke special capability for too long. Intel did amazing with Xeon having integrated OmniPath 100Gb a long time ago, that was amazing, for barely any extra bucks. But the market didn't reward them kicking total ass and everyone gave up on connecting chips together. Today we are hostage to fantastically expensive shitty inefficient NIC that cost a crap ton of money to do a worse job, paying enormous penalty for not having the capability on chip, making at best asmedia io hubs do the USB4 dance a hip away from the CPU. I really hope Intel can appreciate how good they were, see the threat of Apple kicking as here doing what Intel uniquely has been offering for half a decade with incredible Thunderbolt offerings on-chip (limited alas only to mobile chips). I hope AMD feels the heat and gets some god dMned religion and sees the pressure and thread: man they delivered so strong on PCIe lane counts but man they have been so so so slacking on io capabilities for so long, especially on consumer platforms, and Apple is using both their awesome awesome awesome on-chip memory here and their fan-tastic exceptional ability to care just even the tiniest bit about using the consumer interconnect (that already exists in hardware). I really really really hope someone else other than Apple can ante up and care. There are so many wins to be had, so close. These companies feel so distracted from the plot. Fucking shame. Good on Apple for being the only mofos to a Tually seize the obvious that was just sitting here, they took no effort nor innovation. What a shame no other players are trying at all.

PunchyHamster 11mo ago
Intel is allergic for making consumer stuff good. Remember how in consumer range like half of the chips had fucking virtualisation disabled, long after competition had it on everything ?

CaliforniaKarl 11mo ago

The "all nodes connecting to all other nodes" setup reminds me of NUMALink, the interconnect that SGI used on many (most? all?) of their supercomputers. In an ideal configuration, each 4-socket node has two NUMALink connections to every other node. As Jeff says, it's a ton of cables, and you don't have to think of framing or congestion in the same way as with RDMA over Ethernet.

T3OU-736 11mo ago
SGI's HW also had ccNUMA (cache-coherent Non-Uniform Memory Access), which, given the latencies possible in systems _physically_ spanning entire rooms, was quite a feat. The IRIX OS even had functionality to migrate kobs and theor working memory closer to each other to lower the latency of access. We see echoes of this when companies like high-frequency traders pay attention to motherboard layouts and co-locate and pin the PTS (proprietary trading systems) processes to specific cores based on which DIMMs are on which side of the memory controller.
_zoltan_ 11mo ago
just as an NVL72 rack today has 7271 links (18 probably) in the rack connecting all those GPUs together.

poemxo 11mo ago

Really cool article, I liked these details that weren't exactly related to the thesis: - the mysterious disappearance of Exo - Jeff wants something like SMB Direct but for the Mac. Wait what? SMB Direct is a thing, wha?? I always thought networked storage was untrustworthy. - A single M3 Ultra is fast for inference - A framework desktop ai max 395 is only $2100 Now I have some more rabbit holes to jump down.

fortran77 11mo ago

On Intel Motherboards, it's easy to find ones that can take 2TB of RAM, for example: https://www.supermicro.com/en/products/motherboard/x14sbw-tf https://www.supermicro.com/en/products/motherboard/x14sbw-tf This seems suboptimal.

TheTxT 11mo ago
The gpu can’t access that directly however. On Apple Silicon it can all be used as vram.
ErneX 11mo ago
2TB of system memory, not unified memory like Apple Silicon Macs.

dsrtslnd23 11mo ago

Any thoughts on the GB300 workstation with 768GB RAM (from NVIDA, Asus, Dell, ...)? Although many announcements were made it seems not to be available yet. It does have faster interconnects but will probably be much more expensive.

cboyardee 11mo ago

[dead]

polsevev 11mo ago

As much as i hate Apples attitude towards hackers and modifying systems. I have to commend them for building awesome features like this

Rakshath_1 11mo ago

[flagged]

tomhow 11mo ago
Please stop.

Tepix 11mo ago

Linux already has RDMA support but it cannot yet use Thunderbolt. It's probably quite a bit of work to add everything that's required. Is anyone working on it? It would be great to have this for those cheap Strix Halo boxes with 128GB quad channel DDR5-8000 for using two or three of them with their 2 USB4 ports (which are Thunderbolt capable) to fit larger models.

dylan604 11mo ago
> Linux already has RDMA support but it cannot yet use Thunderbolt. Is TB still encumbered by licensing requirements causing this lack of use?
yonatan8070 11mo ago
I assume that's Thunderbolt 5? From my experience, eGPUs over USB 4/TB3 work just fine (from a technical point of view, in practice 40Gbps isn't enough BW and performance is shit)
- Tepix 11mo ago
  Yes, that works but what's needed here is remotely accessing RAM on another PC via RDMA via Thunderbolt. Not accessing a standalone eGPU.

pjmlp 11mo ago

In an ideal world, Apple would have released a Mac Pro with card slots for doing this kind of stuff. Instead we get gimmicks over Thunderbolt.

g947o 11mo ago
I can imagine Apple shipping Mac Pros with add-ons that allows running local inference with minimal setups. "Look, just spend $50k on this machine and you get a usable LLM server that can be shared for a team." But they don't seem particularly interested in that market.

clan 11mo ago

As Jeff states there are really no Thunderbolt switches which currently limits the size of the cluster. But would it be possible to utilize RoCE with these boxes rather than RDMA over Thunderbolt? And what would the expected performance be? As I understand RDMA should be 7-10 times faster than via TCP. But if I understand it correctly RoCE is RDMA over Converged Ethernet. So using ethernet frames and lower layer rather than TCP. 10G Thunderbolt adapters are fairly common. But you can find 40G and 80G Thunderbolt ethernet adapters from Atto. Probably not cheap - but would be fun to test! But ieven if the bandwidth is there we might get killed with latency. Imagine this hardware with a PCIe slot. The Infiniband hardware is there - then we "just" need the driver.

KeplerBoy 11mo ago
At that point you could just breakout the thunderbolt to PCIe and use a regular NIC. Actually, I'm pretty sure that's all that to the Atto Thunderlink, a case around a broadcom nic. Then you _just_ need the driver. Fascinating, Apple ships MLX5 drivers, that's crazy imo. I understand that's something they might need internally, but shipping that on ipadOs is wild. https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/ https://kittenlabs.de/blog/2024/05/17/25gbit/s-on-macos-ios/
- clan 11mo ago
  That is what I am suggesting with the Atto adapter. Infiniband is way faster and lower latency than a NIC. These days NIC==Ethernet.
  - KeplerBoy 11mo ago
    What makes you think Infiniband is faster than Ethernet? Aren't they pretty much equal these days with RDMA and kernel bypass?
jamesfmilne 11mo ago
macOS ships with drivers for Mellanox ConnectX cards, but I have no idea if they will show up in `ibv_devices` or `ibv_devinfo`.

XCSme 11mo ago

Anyone else getting ERR_TOO_MANY_REDIRECTS trying to access the post?

pstoll 11mo ago
Yes only on that page, not the rest of his blog. Guessing he ansible’d it to redirect ;)
moebrowne 11mo ago
https://web.archive.org/web/20251219055833/https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5 https://web.archive.org/web/20251219055833/https://www.jeffg...
XCSme 11mo ago
It's working now.

bluedino 11mo ago

There have a been a couple videos/posts about this from other influencers today Does anyone remember a guy here posting about linking Mac Studios with Thunderbolt for HPC/clustering? I wasn't able to find it with a quick search. Edit: I think it was this? https://www.youtube.com/watch?v=d8yS-2OyJhw https://www.youtube.com/watch?v=d8yS-2OyJhw

daft_pink 11mo ago

The next Mac studio is going to be a top seller. I don’t think people want to drop $10k on a few M3s, but I think they will do it for the M6. Just hoping the DRAM shortage doesn’t ruin this plan.

oofbey 11mo ago
Apple always charges a huge premium for RAM. Maybe it’s enough to buffer their pricing scheme from the supply shock. I have run the numbers though.
- terhechte 11mo ago
  Tim Cook is famous for locking in their prices years in advance.

dogcowmoof 11mo ago

Wonder if RDMA support can translate to things like SMB direct or other RDMA adjacent things

dogcowmoof 11mo ago

Wonder if support for RDMA will translate into support for things such as SMB Direct or if it's really only useful for RAM pooling

krbaccord94f 11mo ago

rdma_ctl enable in 1tn parameter. TL1 mount, where 1.5 TB allocate mac-mini server.

e28eta 11mo ago

> For example: did you know there's no way to run a system upgrade (like to 26.2) via SSH I did not know this. I thought the `softwareupdate` command was built for this use case, and thought it worked over ssh. It sure looks like it should work, but I don’t have a mac I can try it on right now.

pudquick 11mo ago
He's wrong, it's possible. It's just that root privileges alone is insufficient due to how the signing on LocalPolicy works on M series Macs https://support.apple.com/guide/security/contents-a-localpolicy-file-mac-apple-silicon-secc745a0845/1/web/1#secb7cb6686c https://support.apple.com/guide/security/contents-a-localpol... The manpage for the command provides information on credential usage on Apple Silicon devices.

supermatt 11mo ago

What is the max token throughput when batching. Lots of agentic workflows (not just vibe coding) are running many inferences in parallel. It seems like every time someone does an AI hardware “review” we end up with figures for just a single instance, which simply isn’t how the target demographic for a 40k cluster are going to be using it. Jeff, I love reading your reviews, but can’t help but feel this was a wasted opportunity for some serious benchmarking of LLM performance.

saddat 11mo ago

A good part of humanities knowledge under your desk running with a few old light bulbs worth of power

extraduder_ire 11mo ago

Is RDMA only going to be on the studio, or is it coming to anything with a thunderbolt 5 port on it?