6 ms·

The benefits are twofold: physical colocation and bandwidth. Thunderbolt 5 offers 80Gbps of bidirectional bandwidth. PCIe 5.0 16x offers 1024Gbps of bidirectio

by jmyeet 7mo ago

The benefits are twofold: physical colocation and bandwidth. Thunderbolt 5 offers 80Gbps of bidirectional bandwidth. PCIe 5.0 16x offers 1024Gbps of bidirectional bandwidth. This matters. TB5 cables can only get so long whereas fiber can go much farther more easily. This means that in a data center type environment, you could virtualize your GPUs and attach them as necessary, putting them in a separate bank (probably on the same rack).

mikepurvis 7mo ago
"same rack" should still be fine for 1m passive TB5 cable though, right?
consp 7mo ago
> 1024Gbps Good luck getting a 1Tbit tranceiver. Anydirectional. Also it's 512Gbitish per direction.
- jmyeet 7mo ago
  Bidirectional is a lot like biweekly. Biweekly depending on context means twice a week or once every two weeks and bidirectional can both mean per direction and total of both directions. But yes I meant 512Gbps each way, to be clear.
  - fc417fc802 7mo ago
    I'm only a single datapoint but I've never encountered that usage. My understanding of a bidirectional link is that it meets the same spec in both directions simultaneously. It's important precisely because many links aren't bidirectional, sharing a single physical link between two logical links.
    - dcrazy 7mo ago
      The more precise terms are full-duplex and half-duplex.
- za_creature 7mo ago
  The video is about a 2x1 link, which the author hopes to eventually scale up to 3x4 using 40 gig transceivers. I'd say thunderbolt is probably safe in the near future.
- throwaway270925 7mo ago
  Easy, fs.com has 1.6Tbps OSFP for about 570€ - though only up to 1m lenght apparently.
  - jmyeet 7mo ago
    I was looking into the highest bandwidth optical transceivers. 400Gbps were easy enough to find so thanks for posting this. I honestly didn't know there were 1.6Tbps transceivers like this. One note: I believe the SMF max fiber length is 2km not 1m [1]. The data sheet [2] also says: > - 2000m max on single mode fiber [1]: https://www.vitextech.com/products/1-6t-osfp-2fr4 https://www.vitextech.com/products/1-6t-osfp-2fr4 [2]: https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-sheet-20251215182215.pdf https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-...
- jauntywundrkind 7mo ago
  That's 64Gb per lane across x16 lanes. That sounds not daunting? There's already 800Gb transceivers readily available, 1.6 is probably getting preview deploys to some hyperscalers & other early adopters as we speak.
dcrazy 7mo ago
Active optical (yes!) Thunderbolt cables can be much longer. After all, optical fiber was the original medium for Thunderbolt, back when it was still called Light Peak. I couldn’t find any optical TB5 cables, but here’s a 4.5m TB4 one: https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active-optical-cable-delivers-supreme-flexibility-to-your-workspace https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active... And if TB3 is enough, Corning makes them in lengths up to 50m: https://www.corning.com/microsites/coc/oem/documents/ocbc/OEM-062-A4-AEN.pdf https://www.corning.com/microsites/coc/oem/documents/ocbc/OE... As for bandwidth, the medium transition seems to actually limit the author’s capabilities by losing some of the more advanced link-training features that are necessary for the highest-bandwidth PCIe 3 connections, never mind PCIe 5.
- zamadatix 7mo ago
  Hundreds of meters is considered short range in the world of *SFP. If you just plan on putting the GPUs in the same rack then I'm not sure it really matters, but you can really put anything anywhere in your DC and have things zoned with *SFP. I don't think there is any reason TB couldn't do the same, beyond it would be even more niche to want non-modular/patchable cables+transceivers at those lengths (especially since fiber is often bundled dozens/hundreds of strands over a single trunk cable between racks).
- dmitrygr 7mo ago
  For the curious, that 50m cable is $500 MSRP. https://1sourcevideo.com/shop/corning-50-meter-thunderbolt-3-optical-cable.html https://1sourcevideo.com/shop/corning-50-meter-thunderbolt-3...
- jauntywundrkind 6mo ago
  Thunderbolt is kind of cursed. To insure maximum compatibility it mandates a legacy usb2 connection via separate connections. TB3/USB4/TB4 are packetized, but afaik there's no defined way to packetized usb2, it's expected there be a whole separate set of wires for it. And because of timings, my admittedly so-aonunderstanding that you can only get about 7m before you absolutely have to have a hub/repeater (unless you can speed up the speed of light considerably). This limit to how long a single length can be can't really be cheated without violating usb specs. It's awesome if folks have packetized USB2. A pity it's not in the flipping spec though!! That Corning made it 50m is wild. You need a virtual hub at the start that can pretend to be hubs 1-5 (so it's close enough to time well). Then a hub on the other side of the cable at (skinny) tree depth 6. Allowing for 4 devices under it (the number of ports on a usb2 hub in the spec. But you could work around by faking being not a skinny tree but a fat tree, maybe?).

by jmyeet 7mo ago

mikepurvis 7mo ago

"same rack" should still be fine for 1m passive TB5 cable though, right?

consp 7mo ago

> 1024Gbps Good luck getting a 1Tbit tranceiver. Anydirectional. Also it's 512Gbitish per direction.

jmyeet 7mo ago
Bidirectional is a lot like biweekly. Biweekly depending on context means twice a week or once every two weeks and bidirectional can both mean per direction and total of both directions. But yes I meant 512Gbps each way, to be clear.
- fc417fc802 7mo ago
  I'm only a single datapoint but I've never encountered that usage. My understanding of a bidirectional link is that it meets the same spec in both directions simultaneously. It's important precisely because many links aren't bidirectional, sharing a single physical link between two logical links.
  - dcrazy 7mo ago
    The more precise terms are full-duplex and half-duplex.
za_creature 7mo ago
The video is about a 2x1 link, which the author hopes to eventually scale up to 3x4 using 40 gig transceivers. I'd say thunderbolt is probably safe in the near future.
throwaway270925 7mo ago
Easy, fs.com has 1.6Tbps OSFP for about 570€ - though only up to 1m lenght apparently.
- jmyeet 7mo ago
  I was looking into the highest bandwidth optical transceivers. 400Gbps were easy enough to find so thanks for posting this. I honestly didn't know there were 1.6Tbps transceivers like this. One note: I believe the SMF max fiber length is 2km not 1m [1]. The data sheet [2] also says: > - 2000m max on single mode fiber [1]: https://www.vitextech.com/products/1-6t-osfp-2fr4 https://www.vitextech.com/products/1-6t-osfp-2fr4 [2]: https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-sheet-20251215182215.pdf https://resource.fs.com/mall/resource/cn_osfp-2fr4-16t-data-...
jauntywundrkind 7mo ago
That's 64Gb per lane across x16 lanes. That sounds not daunting? There's already 800Gb transceivers readily available, 1.6 is probably getting preview deploys to some hyperscalers & other early adopters as we speak.

dcrazy 7mo ago

Active optical (yes!) Thunderbolt cables can be much longer. After all, optical fiber was the original medium for Thunderbolt, back when it was still called Light Peak. I couldn’t find any optical TB5 cables, but here’s a 4.5m TB4 one: https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active-optical-cable-delivers-supreme-flexibility-to-your-workspace https://www.owc.com/blog/the-new-superlong-40gb-s-owc-active... And if TB3 is enough, Corning makes them in lengths up to 50m: https://www.corning.com/microsites/coc/oem/documents/ocbc/OEM-062-A4-AEN.pdf https://www.corning.com/microsites/coc/oem/documents/ocbc/OE... As for bandwidth, the medium transition seems to actually limit the author’s capabilities by losing some of the more advanced link-training features that are necessary for the highest-bandwidth PCIe 3 connections, never mind PCIe 5.

zamadatix 7mo ago
Hundreds of meters is considered short range in the world of *SFP. If you just plan on putting the GPUs in the same rack then I'm not sure it really matters, but you can really put anything anywhere in your DC and have things zoned with *SFP. I don't think there is any reason TB couldn't do the same, beyond it would be even more niche to want non-modular/patchable cables+transceivers at those lengths (especially since fiber is often bundled dozens/hundreds of strands over a single trunk cable between racks).
dmitrygr 7mo ago
For the curious, that 50m cable is $500 MSRP. https://1sourcevideo.com/shop/corning-50-meter-thunderbolt-3-optical-cable.html https://1sourcevideo.com/shop/corning-50-meter-thunderbolt-3...
jauntywundrkind 6mo ago
Thunderbolt is kind of cursed. To insure maximum compatibility it mandates a legacy usb2 connection via separate connections. TB3/USB4/TB4 are packetized, but afaik there's no defined way to packetized usb2, it's expected there be a whole separate set of wires for it. And because of timings, my admittedly so-aonunderstanding that you can only get about 7m before you absolutely have to have a hub/repeater (unless you can speed up the speed of light considerably). This limit to how long a single length can be can't really be cheated without violating usb specs. It's awesome if folks have packetized USB2. A pity it's not in the flipping spec though!! That Corning made it 50m is wild. You need a virtual hub at the start that can pretend to be hubs 1-5 (so it's close enough to time well). Then a hub on the other side of the cable at (skinny) tree depth 6. Allowing for 4 devices under it (the number of ports on a usb2 hub in the spec. But you could work around by faking being not a skinny tree but a fat tree, maybe?).