3 ms·

Note that you can do point-to-point network links directly with thunderbolt (and usb4). https://support.apple.com/guide/mac-help/ip-thunderbolt-connect-mac-com

by zokier 9mo ago

Note that you can do point-to-point network links directly with thunderbolt (and usb4). https://support.apple.com/guide/mac-help/ip-thunderbolt-connect-mac-computers-mchld53dd2f5/mac https://support.apple.com/guide/mac-help/ip-thunderbolt-conn... etc

kohlschuetter 9mo ago
Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup. You'd also mostly be limited to short cables (1-2m) and a ring topology.
- admash 9mo ago
  Any idea why that would be the case?
  - butvacuum 9mo ago
    The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot). This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link between nodes.
    - sponaugle 9mo ago
      I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.
      - nodesocket 9mo ago
        I figured this was Jeff’s CTO Laboratory. I enjoy your channel. Are you kicking yourself for pulling the trigger on MS-01s now that MS-02 Ultras released?
        sponaugle 9mo ago
        Yea! Those MS-O2s look great, so I may need to upgrade! I did get a couple of DGX Sparks to play with.
      - butvacuum 9mo ago
        AWESOME!! Do you mind linking references? I put the project aside because of these issues.
        sponaugle 9mo ago
        Yea I'll dig up the link I used - there was a great reference about getting the thunderbolt working after reboots/etc.

by zokier 9mo ago

kohlschuetter 9mo ago

Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup. You'd also mostly be limited to short cables (1-2m) and a ring topology.

admash 9mo ago
Any idea why that would be the case?
- butvacuum 9mo ago
  The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot). This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link between nodes.
  - sponaugle 9mo ago
    I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.
    - nodesocket 9mo ago
      I figured this was Jeff’s CTO Laboratory. I enjoy your channel. Are you kicking yourself for pulling the trigger on MS-01s now that MS-02 Ultras released?
      - sponaugle 9mo ago
        Yea! Those MS-O2s look great, so I may need to upgrade! I did get a couple of DGX Sparks to play with.
    - butvacuum 9mo ago
      AWESOME!! Do you mind linking references? I put the project aside because of these issues.
      - sponaugle 9mo ago
        Yea I'll dig up the link I used - there was a great reference about getting the thunderbolt working after reboots/etc.