4 ms·

https://developer.apple.com/documentation/technotes/tn3205-low-latency-communication-with-rdma-over-thunderbolt https://developer.apple.com/documentation/techno

https://developer.apple.com/documentation/technotes/tn3205-low-latency-communication-with-rdma-over-thunderbolt https://developer.apple.com/documentation/technotes/tn3205-l... MacOS has supported RDMA over thunderbolt since March 19th.

wrs 2mo ago
That is just RDMA itself and seems to be aimed directly at distributed inference (no surprise there!). Now they need to update the SMB driver to make use of it.
- geerlingguy 2mo ago
  Yeah, raw RDMA is one thing, making sure other protocols can use it is another... hopefully Apple gets there, but right now SMB with RDMA is probably a tiny, tiny niche. I do wonder what Apple's custom-built servers are running for networking. I wish they still made Mac Pro-level chassis specifically for better IO. Thunderbolt 5 is nice, but far inferior to a PCIe x16 slot (which can hit 400 Gbps networking) nowadays.
  - xp84 2mo ago
    I'd suspect that (from Apple's perspective) any form of file sharing (other than iCloud) is now considered 'extremely niche.' If they were focused on the needs of the kind of person who moves enough terabytes of files around to need this kind of speed, we'd see them still making either computers with slots, or first-party 25Gbps networking. I'm not really trying to criticize them for it -- Tim Cook's bean counters clearly did the math and they're certainly free to optimize their Mac R&D spend for things that'll sell college students MacBook Neos and Airs, rather than worrying about the MKBHDs, Linus Sebastians and Jeff Geerlings of the world (this is just a list of people I've seen using extremely fast networking).
    - tomjakubowski 2mo ago
      Might their professionals market segment (designers, video editors, 3d artists, and so on) still be using thunderbolt and LAN storage? This was the case ~7 years ago when I worked adjacent to these sorts of users. Video files are huge and expensive in time and money to transit with cloud storage providers.
      - kridsdale1 2mo ago
        I do 8k video editing on a Mac Studio. Files are often multiple TB (raw camera capture which I have to encode to ProRes). Moving this kind of bandwidth is absolutely what I have to do and Apples’ bad networking is a blocker I’ve discovered.
      - geerlingguy 2mo ago
        Apple used to even sell Xserve RAID and SAN software specifically to cater to the higher end users who needed fast networked storage (research, broadcast, media). They kinda hit a standstill on the networking side as they moved to the trash can Mac, and I think that had killed momentum towards any high speed IO besides the Thunderbolt ports moving forward. I was hopeful they'd keep the Mac Pro going into the M-series era but after one version... it's gone.
      - xp84 2mo ago
        I'm sure they are, but there just aren't that many of them. When you include only real professional "Hollywood" people plus the elite tier of YouTubers who can easily afford to drop $10,000 on a computer (The 'Mac Pro' target market, in other words) we're talking about a tiny niche to Apple. Both within the "Mac" segment, and even more so when compared to the company's overall portfolio that includes iPhone. So, it seems that Tim Cook decided at some point that while it's nice to make $5,000 in profit from a Mac Pro sale every day or two (okay, I kid, I'm sure they sell a bit more than 365 a year), even if they hang those "pros" out to dry completely and they defect to Windows, Apple would be better off spending their time focusing on ways to make the MacBook Air and Neo more attractive or profitable because they move orders of magnitude more units of those. Note: I don't like this, but my feelings have nothing to do with it, I think it's just a classic Tim Cook Numbers-first business decision
- throw0101d 2mo ago
  For the record this is officially called "SMB Direct": * https://learn.microsoft.com/en-us/windows-server/storage/file-server/smb-direct https://learn.microsoft.com/en-us/windows-server/storage/fil... * https://docs.kernel.org/filesystems/smb/smbdirect.html https://docs.kernel.org/filesystems/smb/smbdirect.html
dlcarrier 2mo ago
Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.
- olavgg 2mo ago
  Yes, nfs with rdma will increase your performance significantly.
  - gorkish 2mo ago
    If your storage devices are native PCIe and your network is actually fast enough and your nics and switches have the proper hardware and driver support and your switches are properly configured and your nfs client and server configurations are tweaked just so, and you hold your face in just the right position The perf is amazing
    - dlcarrier 2mo ago
      I do have stable diffusion running on a non-Nvidia GPU, so I'm familiar with the manner of rituals required before contemplating installing system updates or even suspending the system. Now, I'm off to determine if an eMMC M.2 module is a native PCIe device…
    - olavgg 2mo ago
      ROCE do require proper nic/switch tuning. But if you go all in on Mellanox NIC's and switches, its basically on knob on the switch: roce If you buy CX3 and Mellanox SX6036(easy fan adjustment and low power usage) switches on Ebay for cheap. Which are great up to 56Gbps. You can run infiniband which is way easier to setup, just remember you still need to enable IP over IB to start the handshake for the RDMA connection.
  - lmz 2mo ago
    They also said something about stability, which I doubt would be improved by adding RDMA into the mix.
- 2mo ago

wrs 2mo ago

That is just RDMA itself and seems to be aimed directly at distributed inference (no surprise there!). Now they need to update the SMB driver to make use of it.

geerlingguy 2mo ago
Yeah, raw RDMA is one thing, making sure other protocols can use it is another... hopefully Apple gets there, but right now SMB with RDMA is probably a tiny, tiny niche. I do wonder what Apple's custom-built servers are running for networking. I wish they still made Mac Pro-level chassis specifically for better IO. Thunderbolt 5 is nice, but far inferior to a PCIe x16 slot (which can hit 400 Gbps networking) nowadays.
- xp84 2mo ago
  I'd suspect that (from Apple's perspective) any form of file sharing (other than iCloud) is now considered 'extremely niche.' If they were focused on the needs of the kind of person who moves enough terabytes of files around to need this kind of speed, we'd see them still making either computers with slots, or first-party 25Gbps networking. I'm not really trying to criticize them for it -- Tim Cook's bean counters clearly did the math and they're certainly free to optimize their Mac R&D spend for things that'll sell college students MacBook Neos and Airs, rather than worrying about the MKBHDs, Linus Sebastians and Jeff Geerlings of the world (this is just a list of people I've seen using extremely fast networking).
  - tomjakubowski 2mo ago
    Might their professionals market segment (designers, video editors, 3d artists, and so on) still be using thunderbolt and LAN storage? This was the case ~7 years ago when I worked adjacent to these sorts of users. Video files are huge and expensive in time and money to transit with cloud storage providers.
    - kridsdale1 2mo ago
      I do 8k video editing on a Mac Studio. Files are often multiple TB (raw camera capture which I have to encode to ProRes). Moving this kind of bandwidth is absolutely what I have to do and Apples’ bad networking is a blocker I’ve discovered.
    - geerlingguy 2mo ago
      Apple used to even sell Xserve RAID and SAN software specifically to cater to the higher end users who needed fast networked storage (research, broadcast, media). They kinda hit a standstill on the networking side as they moved to the trash can Mac, and I think that had killed momentum towards any high speed IO besides the Thunderbolt ports moving forward. I was hopeful they'd keep the Mac Pro going into the M-series era but after one version... it's gone.
    - xp84 2mo ago
      I'm sure they are, but there just aren't that many of them. When you include only real professional "Hollywood" people plus the elite tier of YouTubers who can easily afford to drop $10,000 on a computer (The 'Mac Pro' target market, in other words) we're talking about a tiny niche to Apple. Both within the "Mac" segment, and even more so when compared to the company's overall portfolio that includes iPhone. So, it seems that Tim Cook decided at some point that while it's nice to make $5,000 in profit from a Mac Pro sale every day or two (okay, I kid, I'm sure they sell a bit more than 365 a year), even if they hang those "pros" out to dry completely and they defect to Windows, Apple would be better off spending their time focusing on ways to make the MacBook Air and Neo more attractive or profitable because they move orders of magnitude more units of those. Note: I don't like this, but my feelings have nothing to do with it, I think it's just a classic Tim Cook Numbers-first business decision
throw0101d 2mo ago
For the record this is officially called "SMB Direct": * https://learn.microsoft.com/en-us/windows-server/storage/file-server/smb-direct https://learn.microsoft.com/en-us/windows-server/storage/fil... * https://docs.kernel.org/filesystems/smb/smbdirect.html https://docs.kernel.org/filesystems/smb/smbdirect.html

dlcarrier 2mo ago

Is RDMA something worth looking into for connecting two random Linux computers? I've been extremely underwhelmed by the stability and performance of my NFS shares.

olavgg 2mo ago
Yes, nfs with rdma will increase your performance significantly.
- gorkish 2mo ago
  If your storage devices are native PCIe and your network is actually fast enough and your nics and switches have the proper hardware and driver support and your switches are properly configured and your nfs client and server configurations are tweaked just so, and you hold your face in just the right position The perf is amazing
  - dlcarrier 2mo ago
    I do have stable diffusion running on a non-Nvidia GPU, so I'm familiar with the manner of rituals required before contemplating installing system updates or even suspending the system. Now, I'm off to determine if an eMMC M.2 module is a native PCIe device…
  - olavgg 2mo ago
    ROCE do require proper nic/switch tuning. But if you go all in on Mellanox NIC's and switches, its basically on knob on the switch: roce If you buy CX3 and Mellanox SX6036(easy fan adjustment and low power usage) switches on Ebay for cheap. Which are great up to 56Gbps. You can run infiniband which is way easier to setup, just remember you still need to enable IP over IB to start the handshake for the RDMA connection.
- lmz 2mo ago
  They also said something about stability, which I doubt would be improved by adding RDMA into the mix.
2mo ago