8 ms·
Getting 25 Gbps Thunderbolt Ethernet on My Mac Studio
- pzmarzly 3mo agoOne of the YouTube comments pointed out the issue is most likely the lack of support for SMB Direct (RDMA) in macOS. Would be nice to repeat the test on a Windows/Linux laptop.
- areoform 3mo agohttps://developer.apple.com/documentation/technotes/tn3205-low-latency-communication-with-rdma-over-thunderbolt https://developer.apple.com/documentation/technotes/tn3205-l... MacOS has supported RDMA over thunderbolt since March 19th.
- wrs 3mo agoThat is just RDMA itself and seems to be aimed directly at distributed inference (no surprise there!). Now they need to update the SMB driver to make use of it.
- geerlingguy 3mo agoYeah, raw RDMA is one thing, making sure other protocols can use it is another... hopefully Apple gets there, but right now SMB with RDMA is probably a tiny, tiny niche. I do wonder what Apple's custom-built servers are running for networking. I wish they still made Mac Pro-level chassis specifically for better IO. Thunderbolt 5 is nice, but far inferior to a PCIe x16 slot (which can hit 400 Gbps networking) nowadays.
- xp84 3mo agoI'd suspect that (from Apple's perspective) any form of file sharing (other than iCloud) is now considered 'extremely niche.' If they were focused on the needs of the kind of person who moves enough terabytes of files around to need this kind of speed, we'd see them still making either computers with slots, or first-party 25Gbps networking. I'm not really trying to criticize them for it -- Tim Cook's bean counters clearly did the math and they're certainly free to optimize their Mac R&D spend for things that'll sell college students MacBook Neos and Airs, rather than worrying about the MKBHDs, Linus Sebastians and Jeff Geerlings of the world (this is just a list of people I've seen using extremely fast networking).
- tomjakubowski 3mo agoMight their professionals market segment (designers, video editors, 3d artists, and so on) still be using thunderbolt and LAN storage? This was the case ~7 years ago when I worked adjacent to these sorts of users. Video files are huge and expensive in time and money to transit with cloud storage providers.
- kridsdale1 3mo agoI do 8k video editing on a Mac Studio. Files are often multiple TB (raw camera capture which I have to encode to ProRes). Moving this kind of bandwidth is absolutely what I have to do and Apples’ bad networking is a blocker I’ve discovered.
- geerlingguy 3mo agoApple used to even sell Xserve RAID and SAN software specifically to cater to the higher end users who needed fast networked storage (research, broadcast, media). They kinda hit a standstill on the networking side as they moved to the trash can Mac, and I think that had killed momentum towards any high speed IO besides the Thunderbolt ports moving forward. I was hopeful they'd keep the Mac Pro going into the M-series era but after one version... it's gone.
- xp84 3mo agoI'm sure they are, but there just aren't that many of them. When you include only real professional "Hollywood" people plus the elite tier of YouTubers who can easily afford to drop $10,000 on a computer (The 'Mac Pro' target market, in other words) we're talking about a tiny niche to Apple. Both within the "Mac" segment, and even more so when compared to the company's overall portfolio that includes iPhone. So, it seems that Tim Cook decided at some point that while it's nice to make $5,000 in profit from a Mac Pro sale every day or two (okay, I kid, I'm sure they sell a bit more than 365 a year), even if they hang those "pros" out to dry completely and they defect to Windows, Apple would be better off spending their time focusing on ways to make the MacBook Air and Neo more attractive or profitable because they move orders of magnitude more units of those. Note: I don't like this, but my feelings have nothing to do with it, I think it's just a classic Tim Cook Numbers-first business decision
- throw0101d 3mo agoFor the record this is officially called "SMB Direct": * https://learn.microsoft.com/en-us/windows-server/storage/file-server/smb-direct https://learn.microsoft.com/en-us/windows-server/storage/fil... * https://docs.kernel.org/filesystems/smb/smbdirect.html https://docs.kernel.org/filesystems/smb/smbdirect.html
- dlcarrier 3mo agoIs RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.
- olavgg 3mo agoYes, nfs with rdma will increase your performance significantly.
- gorkish 3mo agoIf your storage devices are native PCIe and your network is actually fast enough and your nics and switches have the proper hardware and driver support and your switches are properly configured and your nfs client and server configurations are tweaked just so, and you hold your face in just the right position The perf is amazing
- dlcarrier 3mo agoI do have stable diffusion running on a non-Nvidia GPU, so I'm familiar with the manner of rituals required before contemplating installing system updates or even suspending the system. Now, I'm off to determine if an eMMC M.2 module is a native PCIe device…
- olavgg 3mo agoROCE do require proper nic/switch tuning. But if you go all in on Mellanox NIC's and switches, its basically on knob on the switch: roce If you buy CX3 and Mellanox SX6036(easy fan adjustment and low power usage) switches on Ebay for cheap. Which are great up to 56Gbps. You can run infiniband which is way easier to setup, just remember you still need to enable IP over IB to start the handshake for the RDMA connection.
- lmz 3mo agoThey also said something about stability, which I doubt would be improved by adding RDMA into the mix.
- 3mo ago
- gorkish 3mo agoThis is a ConnextX4 on a thunderbolt bridge. This is well trodden territory. Unless some hero of open source has come to bat recently, MacOS doesn't support RDMA with this card or over Ethernet at all for that matter. It's only supported over Thunderbolt. His iperf3 results show that it's able to go full beans on a single 25g port. A cross platform SMB file copy is kind of a meaningless benchmark in this context. IMO a pretty cool product would be a network switch providing native thunderbolt interfaces to multiple upstream hosts. That wouldn't magically fix this RDMA issue, though. This is just a separate thought. Seems like the kind of thing Mikrotik should make.
- CobaltFire 3mo agoCommented on another top comment but yeah, I've been running a similar setup for about $200 all in. Using the PCIe LX version takes care of much of the heat issue, and an eGPU dock (Razer Core in my case) takes care of the fan. Craigslist and eBay were the sources. Speeds aren't any better, but at least it was cheap and fiber.
- cyberax 3mo agoI have a 25GbE setup using Thunderbolt, kinda similar to the author's. Except that my fan is held together by a piece of string, not a 3D-printed enclosure. I found that SMB on macOS is just bad. If I try to heavily load it (e.g. by doing mass conversions in my Audible library), it can cause stalls and disconnections. I managed to find a set of parameters (in /etc/nsmbd.conf) that makes it work more-or-less reliably by trial-and-error: dir_cache_off=yes, file_ids_off=yes, notify_off=yes, mc_on=no. It's still terrible, though. It's so bad that Linux in a Parallels VM works _better_.
- monster_truck 3mo agoIt's always been like this, too! SMB or otherwise. Any company/person that has been using macs long enough has been burned by silently corrupted volumes or etc and has a thorough system of multiple on and offsite backups. It's not just software companies or home enthusiasts either. There are a lot of ie medical practices or accounting/benefits agencies out there who have sworn off Apple for life after having to spend extreme amounts of money on data recovery to hopefully get their years of records back, usually in a do or die situation like an IRS audit which causes them to discover that their basement was flooding and all of the paper records are destroyed.
- teaearlgraycold 3mo agoIt’s crazy that the Ethernet chips consume more power (presumably - because of the heat) than the processor in the computer. How is driving 25Gb/s such a hot task compared to driving multiple 40Gb/s Thunderbolt ports?
- wmf 3mo agoA Mac Studio is over 100 W while a 25G NIC is under 25 W (that's the power limit for a normal PCIe slot).
- ssl-3 3mo agoA normal x16 PCIe slot has always been able to supply up to 75w since the beginning.
- wmf 3mo agoA normal non-GPU PCIe x4/x8 slot that you'd use for a NIC provides 25 W.
- altairprime 3mo agoFor a 25G NIC, I’d be perfectly unsurprised if it demanded an x16 slot. I’d be more surprised if it didn’t have a power connector like a video card, though.
- StillBored 3mo agoI might be hallucinating, but IIRC this is now dependent on PCIe revision. More recent revisions have removed the slot power limits line, and so it works out that in theory a x4 slot (or any other size) can have a 75W load. IIRC this is stated in the middle of a giant section on up/down plugging. (but its also displayed in the pcie PowerLimit capabilities you can see with lspci)
- throwaway219450 3mo agoThey’re not that bad, but even a couple watts of power with no way to dissipate is going to get hot enough to burn you. Forced air cooling over a heatsink is really effective. 15W under load is much lower than a high end desktop CPU that can pull 150-200W. See also NVMe drives? They draw enough idle power to drain a laptop battery overnight, and will easily hit 60+C in a bad enclosure. That’s only a 10Gbps module. The USB4 enclosures are all heatsink.
- nutjob2 3mo agoSomewhat off-topic: has anyone implemented USB C based networking? So for instance you have two or more machines that have USB 4 ports at 20 to 80 Gbps and you plug in the appropriate USB C cables to connect them. What do you need to make the connections appear as a (point to point) part of a regular TCP/IP network? Seems much cheaper than Ethernet if you already have the ports.
- robin_reala 3mo agoUSB-4 is basically Thunderbolt right? Thunderbolt networking is pretty automatic on Macs at least: https://support.apple.com/en-gb/guide/mac-help/mchld53dd2f5/mac https://support.apple.com/en-gb/guide/mac-help/mchld53dd2f5/...
- ianburrell 3mo agoUSB4 includes standardization of Thunderbolt 3. It allows tunneling USB data, DisplayPort, and PCIe. It also adds 40Gbps and 80Gbps speeds. Thunderbolt 4 is profile with USB4 tunneling, 40Gbps speed, and 100W power. Thunderbolt 5 is 80Gbps and 240W power.
- imoverclocked 3mo agoYup, totally doable. Of course, it’s a point to point link so if you want to build a whole cluster that way then you’ll need a lot of usb-c ports or you’ll need to make more of a mesh topology. I believe MacOS supports it via the usual network control panel. I’ve used it for transferring data from an older laptop to a newer one. The cable is more expensive than fiber, last time I checked.
- QuantumNomad_ 3mo agoI have also used a USB 4 cable to network two MacBook Pros to one another and can confirm that it was very simple to connect and it was fast as heck. The USB 4 cable I used was rated for up to 40 Gbps.
- warpflyght 3mo ago
- SlavikCA 3mo agoWhat are typical use cases for needing 25Gbps? Author talks about 4k video, but AI tells me that for "4K production mastering (ProRes 422 HQ, 4:2:2, 10-bit)" the rate is under 2Gbps. I'm not saying it's not needed, - just curious.
- QuantumNomad_ 3mo agoWhat if it’s uncompressed footage at 60fps? And dealing with many shots. And on top of that maybe working with multiple projects in parallel.
- geerlingguy 3mo agoNormal 4K editing workflows are fine with even 2.5 Gbps wired networking (wired to make latency like when scrubbing not-horrid)... 10 Gbps more necessary if editing ProRes RAW and doing multicam (4-8 streams simultaneously). I'm in a never-ending pursuit of cutting time when transferring large files though — AI models is one thing, but I have 100-200 GB projects I transfer across for backup or import (using an external TB5 drive capable of 3-5 GB/sec), and if I can do that in 1 or 2 minutes per project instead of 3 or 4, that's a noticeable speedup. Definitely overkill for 98% of my workflow though. 10 Gbps is fine.
- andy99 3mo agoRunning local LLMs split across machines. Pretty sure the author has some other posts about doing that. Edit: https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5/ https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-stu...
- logicallee 3mo agocorrect me if I'm wrong but wouldn't a Thunderbolt 5 cable $79 from Apple[1] work better for that? It gets 80 Gbps and doesn't need any extra equipment. [1] https://www.apple.com/shop/product/mdw94am/a/thunderbolt-5-usb%E2%80%91c-pro-cable-1-m https://www.apple.com/shop/product/mdw94am/a/thunderbolt-5-u...
- 3mo ago
- alsetmusic 3mo agoDang, here I was feeling slick with 10Gb between my MBP and Mac mini and NAS. That’s the thing, it could always be bigger / faster / more. It’s enough for my workflow, but it’s fun to see people push further.
- GeekyBear 3mo agoAs mentioned in the blog post's footnote, the bottleneck may just be on the NAS side: > this real-world speed limitation could also be related to my use of my lower-power Arm NAS (running on Ampere Altra, with 32 fairly-slow CPU cores). I was testing writing to an array of fast enterprise NVMe SSDs, so they shouldn't be the bottleneck here. He was only getting 1 Gigabyte of throughput out of the built in 10 Gigabit Ethernet connection and moving to the 25 Gigabit connection only made a marginal impact on throughput, after all.
- Neywiny 3mo agoI'm using the sonnet at work. Work paid for it. Yes it's more expensive. I have gotten over 25 gbps bidirectional, but not by much. Maybe 27. A big downside is that it only supports 15W upstream power which I didn't think to check, so since that laptop only has 2 usb-c ports for power input, it's a bit limiting. The upside, though, was that I plugged it in, got my throughput, and never had to worry ever again. Remember that engineering time is mucho $$. So troubleshooting overheating, divising a solution, manufacturing it, etc, that all eats into the savings. The $400 dock would be cool though. It seems smaller than the sonnet and fixes my charging problem. Weirdly I didn't think I'd care about not including transceivers but they did actually come in handy, so that is a downside. Might try to get one of these if I need another one in the future.
- voxadam 3mo ago>I'm using the sonnet at work. I initially read this as SONET[1] and was seriously questioning your technical choices. [1] https://en.wikipedia.org/wiki/Synchronous_optical_networking https://en.wikipedia.org/wiki/Synchronous_optical_networking
- danw1979 3mo agohey. hey. not everyone’s on board with this packet switching fad, buster. what a man does in his own homelab, even if it does involve ATM, is none of our business.
- TacticalCoder 3mo ago> I can edit 4K video straight off my NAS over the network, Nothing against Jeff Geerling and I love his YouTube channel and I'm glad he's one of us HNer but this crop of circular "I make YouTube vids about big tx speeds and big NAS because I make vids hence I need big tx speeds and big NAS so I can speak about big tx speeds and big NAS because I make YouTube vids about big tx speeds and big NAS" videos is a bit tiring IMO. I'm not into cryptocurrencies but, I don't know, host a full Ethereum node, host a Plex/Jellyfin server serving thousands of movies, host LLMs, etc. I mean: talk about something else than editing vids because you make vids about vids.
- wmf 3mo agoTo be fair he also talks about NTP and RPis.
- geerlingguy 3mo agoJellyfin: https://www.jeffgeerling.com/tags/jellyfin/ https://www.jeffgeerling.com/tags/jellyfin/ LLMs: https://www.jeffgeerling.com/tags/llm/ https://www.jeffgeerling.com/tags/llm/ Etc... (I don't do any crypto)
- randusername 3mo agoCan't you just put a real PCIe NIC in an eGPU enclosure with a fan? I bet you can solve this problem for like $150 and two craigslist transactions.
- monster_truck 3mo agoAbsolutely not. It's another layer of throughput/compatability bingo on top of the already often quite annoying SFP bingo. If you are unfamiliar, the reality of the situation is quite bad. Slight hardware revisions of adapters/switches, which if you are lucky can be differentiated by manufacture date or serial number sequence, will not recognize known good modules, and if they do you then have to actually test if they can establish link, then let them burn for 24+ hours to see if they can maintain it without dropping (often, because of forward error correction, you will not start to see flapping until enough correctable errors have accumulated and the buffer overflows). Then there are firmware updates where this process must then be repeated. You would think with how expensive all of this shit is you would simply be able to buy the officially listed as compatible components but my experience with them has been that a shotgun approach is best for your sanity and deadlines, buying sample kits of each module and trying them all, and then once you find the ones that play nice, buying as many of them as is reasonable and testing all of them. If I had a dollar every time two modules from the same brand, that came in the same box, from the same production run, did not work with one another, I would be able to test your theory for free. Network Engineering is Fun
- simoncion 3mo agoPerhaps the situation with 25gbit modules is way worse than 10gbit, but I have twelve 10gbit 10GTek fiber modules and haven't had a problem with them. Though, perhaps this would be a lottery that I'd inevitably lose if I were populating a great many TOR switches and associated server NICs? It also sounds like you're not saying that OP cannot "...just put a real PCIe NIC in an eGPU enclosure with a fan", but rather that regardless of where that NIC is situated, it's your experience that SFP quality is poor. Do I correctly understand what you're saying? Edit: FWIW, I've had no problems with the ~two foot PCI-E riser cable that connects my graphics card to my motherboard. [0] I expect this OCuLink thing [1] will work fine, too. [0] Taking up several PCI-E slots just for a damn graphics card is really dumb. IMO, mid-grade and better cases should have off-mobo mount points for oversized devices and offer riser cables to connect said devices to the installed motherboard. [1] <https://en.wikipedia.org/wiki/PCI_Express#OCULINK https://en.wikipedia.org/wiki/PCI_Express#OCULINK>
- brcmthrowaway 3mo agoHow does Jeff Geerling afford so many Mac Studios?
- airspresso 3mo agoMy default assumption (with no knowledge about this particular case) is that all gear in YouTube videos is sponsored. Too easy to get jealous of all the fancy setups. But I don't want to become a YouTuber so that settles that.
- geerlingguy 3mo agoThe Mac Studio in most of my videos is my own workstation, purchased a while back. The 4 Mac Studio cluster I tested with Exo and Thunderbolt RDMA was a loaner cluster from Apple. There's no universe in which I could justify buying that. I always mention any equipment that's provided / loaned / etc. and the nature of my relationships with vendors—I try to also keep this document updated for transparency's sake: https://github.com/geerlingguy/youtube/blob/master/Sponsors.md https://github.com/geerlingguy/youtube/blob/master/Sponsors....
- brcmthrowaway 3mo agoThe tax deduction through LLC is a good one
- washbasin 3mo agoIn addition to Jeff's comments in this thread, I am guessing all of his purchased gear is capital expense for an LLC he runs for his youtube channel. That makes it deductible from personal tax. (again, all guesses, but a common pattern in the US)
- rconti 3mo agoFor us <25Gbps plebes out there: For god's sake do not buy a USB-C RealTek RTL8156-equipped multi-gig ethernet dongle to use with your Mac. I went through 3, and they're all trash: Plugable 2.5G USB-C to ethernet adapter (bought November 2025), SABRENT USB C to 2.5Gbps ethernet adapter (bought October 2024), UGREEN USB C 2.5G ethernet adapter (also October 2024). In my experience they immediately fall on their face in a speedtest and drop to a couple hundred Mbps, experience packet loss, etc. This is testing across a few modern Apple Silicon Macs. I now have a WavLink WL-NWU341G 5Gbps dongle with the RTL-8157 that seems to work well, but I've only run a couple of tests-- I am away from my home setup and don't have wired ethernet run to all the rooms where I'm living. If you do some Googling, there's some good information out there about it, but when I bought the bad ones, I couldn't find any information corroborating my experience.
- myelin 3mo agoI've had a lot of luck with the Plugable 2.5G USB-C adapters, although on some ports they're incredibly unreliable -- I'm guessing maybe something about the power supply? On a CalDigit TS3 dock, I've had one working off the rear Thunderbolt socket for over a year, but it fails in minutes if plugged into the USB-C one. I've never had issues with a CalDigit Elements (TB4) hub. I've had success plugging them directly into various Chromebooks. On one Windows laptop, they seemed to work on one USB-C port but not the other. So... reliable once working, but getting there can be a pain.
- rconti 3mo agoInteresting! I'm using a TS3 dock and my entire use case is to bypass the 1GbE on the dock and get 2.5G to my work laptop. But once I started having issues I took both my laptops directly to my main switch to rule out house wiring issues, so I didn't spend much time dealing with the different ports on the TS3. My logic was that I didn't want to spend $400+ on a TS4 for my WORK computer, just to get a faster ethernet for the lulz. I was happy to spend $25 for bragging rights. Of course, now that I've spent that same $25-30 4 times, it's beginning to feel like less of a bargain.
- kalleboo 3mo ago
- dcrazy 3mo agoJeff, I’m curious whether you really would have needed the $1,000 version of the Sonnet TB5 PCIe chassis. Would this $400 one have sufficed? https://www.bhphotovideo.com/c/product/1932655-REG/sonnet_echo_exp_se1_t5_echo_se_i_t5.html?ap=y&smp=y&msclkid=d4c03d724a8315126e86571f6eb8db29 https://www.bhphotovideo.com/c/product/1932655-REG/sonnet_ec...
- wmf 3mo ago$400 is for the empty enclosure and $1,000 is for the enclosure with a NIC in it. Obviously it's still a ripoff because a 25G NIC does not cost $600.
- geerlingguy 3mo agoYeah the best option for cost is to buy one of the many TB4 to PCIe adapters on the market now (I've tried two), that are less than $100. They require external PSU power though, and they don't look nearly as nice on a desk.
- qurren 3mo agoWhat NAS drives can support anywhere near 25 Gbps?
- dotwaffle 3mo agoA single PCIe-3 nvme drive will saturate it on a sequential read load.
- geerlingguy 3mo agoAlso a RAIDZ array (or just mdadm or BTRFS) with a cache in front can easily hit a couple GB/sec even on more consumer level hardware.
- aegisora_ai 3mo ago[flagged]
- ccssowa 3mo agoI’m in the process of gradually upgrading an in-house video production studio with <10 seats from 10G to 25/100G. There were attempts to do this years ago, and I found some unused Atto 40G Thunderlink adapters and Sonnet’s 25G adapters. The 40G Attos are EOL, and I couldn’t get the Sonnets to work. The SFP jungle/support/information is really maddening. Atto’s new 100G Thunderlinks work well. I’ve seen speeds of just over 5 gigabytes per second. Atto definitely does a lot of kernel extension wizardry to get macOS to achieve these speeds. A bit OT: I hope Blackmagic really shakes up the 100G connectivity market, and that Apple realizes their platform is extremely well-suited for IP Video 2110 workflows — if they can ever get networking under control.
- mgerdts 3mo agoI wonder if similar results would come from just adding a second 10 Gb NIC. If his workload runs SMB multichannel at sufficient queue depth, the load should spread across the multiple paths. If the queue depth is always 1, SMB multichannel is not helping on the 25 Gb NIC either.
- rowanG077 3mo agoYou can purchase much cheaper TB to pcie converts in china. You add some generic 12v power and you're done.
- PhageGenerator 3mo agoHe mentions this in the footnote: > I have SMB multichannel enabled, but this real-world speed limitation could also be related to my use of my lower-power Arm NAS (running on Ampere Altra, with 32 fairly-slow CPU cores). Based on my own testing with a 25 Gb NIC setup on macOS I strongly suspect the issue is not on the NAS side, and the hardware I was working with is by no means underpowered. The Mac I was working with is one of those rack mount M2 Mac Pros with a 25 Gb PCI-e card. On the NAS side it was a several hundred terabyte all NVMe flash multi-node cluster, each node with Xeon CPUs and 2x 25 Gb bonded NICs. Doing some rough throughput testing from macOS over multi-channel SMB yields throughput rates that are barely faster than the built-in 10 Gb copper NICs on the Mac. Using iperf I can confirm that I can get near enough to 25 Gb/s raw network throughput on the Mac. And other non-macOS systems see significantly faster throughput to the same NAS. I am not sure exactly what the issue is that causes 25 Gb NICs to underperform so significantly for file sharing tasks, but I am reasonably certain that the issue is in macOS.
- nodesocket 3mo agoI predict the next generation of Studio’s will have a 25 Gbps upgrade option built-in.