18 ms·

Jim Keller suggests Nvidia should have used Ethernet to link Blackwell GPUs

25 points by westurner 3y ago

mdaniel 3y ago
I don't have a horse in this race, but I saw today that there was a healthy thread <https://news.ycombinator.com/item?id=40011762 https://news.ycombinator.com/item?id=40011762> about "just pull fiber" when folks were discussing pulling cat6e and a lot of that talk was about the relative cost versus massive bandwidth gains. My mental model of GPU interconnect is that one wishes to have the most bandwidth possible
- westurner 3y ago
  https://www.windowscentral.com/hardware/computers-desktops/nvidia-kills-off-nvlink-on-rtx-4090 https://www.windowscentral.com/hardware/computers-desktops/n... : > "The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence. > For gamers, creators, and workstation users who want a multi-GPU setup with the latest RTX 4000 series, Nvidia is switching from NVLink as a bridge to the PCIe Gen 5 standard. PCIe 5.0 x32 supports 128GB/s. "PCIe 7.0 Draft 0.5 Spec: 512 GB/s over PCIe x16 On Track For 2025" https://news.ycombinator.com/item?id=39933959 https://news.ycombinator.com/item?id=39933959 > The HBM3E [RAM] Wikipedia article says 1.2TB/s (2024) https://news.ycombinator.com/item?id=39938652 https://news.ycombinator.com/item?id=39938652 "Fiber-optic data transfer speeds hit a rapid 301 Tbps" (2024-03) https://news.ycombinator.com/item?id=39864107 https://news.ycombinator.com/item?id=39864107 HPE Cray Slingshot is a copper Ethernet HPC interconnect: https://www.nextplatform.com/2022/01/31/crays-slingshot-interconnect-is-at-the-heart-of-hpes-hpc-and-ai-ambitions/?amp https://www.nextplatform.com/2022/01/31/crays-slingshot-inte... ... "HPE Slingshot Makes The GPUs Do Control Plane Compute" https://www.nextplatform.com/2022/08/15/hpe-slingshot-makes-the-gpus-do-control-plane-compute/?amp https://www.nextplatform.com/2022/08/15/hpe-slingshot-makes-... : > Here’s the gist. With normal GPU aware MPI software, inside the node the GPU vendors – the only two that matter today are Nvidia and AMD – have their own mechanisms for doing peer-to-peer communications between the GPUs – in this case NVLink and Infinity Fabric. For MPI data exchanges between GPUs located in different nodes, which is often required for running large simulations and models as well as for large AI training runs, MPI data movement is done through Remote Direct Memory Access methods, pioneered with InfiniBand adapters decades ago, which allow for the transfer of data between a GPU and a network interface card without the interaction of the host CPU’s network stack. [...] > In a [recent paper], HPE showed off the new MPI offload method, making the distinction between the typical MPI communication that is GPU aware versus the stream triggered method that is GPU stream aware. "Exploring GPU Stream-Aware Message Passing using Triggered Operations" (2022) https://arxiv.org/abs/2208.04817 https://arxiv.org/abs/2208.04817 https://scholar.google.com/scholar?cites=13547610567386067445&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=1354761056738606744...
  - westurner 3y ago
    "New RISC-V microprocessor can run CPU, GPU, and NPU workloads simultaneously" (2024) https://news.ycombinator.com/item?id=39938538 https://news.ycombinator.com/item?id=39938538
  - justinclift 3y ago
    > ... with the latest RTX 4000 series, Nvidia is switching from NVLink as a bridge to the PCIe Gen 5 standard. The theory sounds nice, but isn't the RTX 4090 only PCIe gen 4, not 5?
    - gmerc 3y ago
      That’s the beauty of it.
      - justinclift 3y ago
        Not really sure what you're meaning?
        cyanydeez 3y ago
        Pushing plebs to buy faster gpus requires limiting consumer options
- blagie 3y ago
  * Wired ethernet is now usually slower than my 802.11. 802.11ax is 0.5-10Gbps in theory, and in practice, faster than gigabit ethernet. * Fiber seems... overpriced. I'm not quite sure why. USB TOSlink is $3 on Aliexpress, as is the cable, so there's nothing fundamentally expensive there, except for being enterprisey. A lot more interconnect would make sense to have as optical. * Wifi is increasing in speed. Wifi 8 is planned to hit 100Gbps. That's where the investment seems to be going.
  - rowanG077 3y ago
    How is the latency and consistency on these newer WiFi standards? From my experience if you hit the sweet spot on WiFi it's very good. But then someone else starts a youtube video. Or you move your laptop. Or maybe even do nothing noticeable at all. And suddenly you have a shit experience. For this reason I have always preferred Ethernet. But I admit I have not compared the more recent WiFi standards.
    - blagie 3y ago
      That's actually what's surprised me. In my home, I found wifi to work better in all respects than ethernet. This is a relatively recent phenomenon. Ethernet has issues too, at times. I've certainly seen nominal 10Gbps networks degrade to <1Gbps, and to fix that, you need to debug wires and connectors, and see where the signal integrity issue too. As speeds go up, those issues get more common. 10Mbps, you could almost literally run a pair of wires from Radio Shack and it would work. 1Gbps was the breakpoint where if I didn't use proper everything (which is often mislabelled on Amazon / Aliexpress / eBay / etc.), things would degrade. On the other hand, my wifi has a nice MIMO router. I expect that split will continue. In theory, a MIMO router with an array of antennas could be almost point-to-point, directing signals where they need to go. At the same time, as speeds go up, any parasitic cable / connector inductance or capacitance will have increasing impact.
  - vrighter 3y ago
    toslink is a piece of junk plastic fiber and a red led. it's technically fiber optic, but can't really be compared to real fiber optic system
rowanG077 3y ago
Well I'd wager Nvidia is very happy they use their own proprietary interconnect. They do everything to strap their customers in a straight jacket and milk them for everything they are worth.
- FireBeyond 3y ago
  Perhaps so, but also, "Nvidia competitor suggests Nvidia should have done something that makes it easier to compete against them..." isn't exactly the revelation.
  - davidf18 3y ago
    [dead]
shrubble 3y ago
I think that Infiniband was previously an open standard, but over time only Mellanox was in the market ( as other companies stopped making product) and they were then bought by Nvidia. I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency.
- justinclift 3y ago
  > I don't agree [...] absolute highest performance is found in Infiniband, not only with bandwidth but latency. What's the alternative(s) you're thinking of? There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially. Omni-path (a descendant of Infiniband) doesn't seem to have worked out either. --- Infiniband also has advantages over Ethernet in other things, like bundling links together. With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream. With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.
  - shrubble 3y ago
    I re-worded what I wrote for clarity, it's an obscure sort of sentence construction. "I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. " I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.
    - justinclift 3y ago
      No worries, my misunderstanding. :)
  - oasisaimlessly 3y ago
Proven 3y ago
[dead]
gmerc 3y ago
Blackwell is a platform game. Interoperability would defeat the purpose.
touisteur 3y ago
I say let Jim Keller show us all how it's done, as he's often done. Let him convince AMD (with their own interconnect and not specially on lead in Ethernet these days, maybe with Xilinx or Pensando but I haven't seen any serious integration in the road map?), Intel (Xe-Link on GPUs? Ethernet on Gaudi, both unobtainium? And they stopped working on their actual lead in switching/Ethernet with Tofino, WTAF BTW) and whoever else is shipping accelerator hardware to go full ethernet and maybe there's something. But why would NVIDIA do anything to share this moat? Or even risk to have their whole ecosystem go glitchy and buggy with other hardware vendors? They invested heavily, very heavily, in network tech and did the whole integration (nccl, gpudirect, and even simple memcopy works fine with NVLINK without futzing with actual Ethernet/IP stacks) at a moment where everyone was fucking around either building GPUs back from scratch or buying and dumping good tech? If there was an actually competitive alternative, I could see Jensen have a look but I don't see it right now.

25 points by westurner 3y ago

I don't have a horse in this race, but I saw today that there was a healthy thread <https://news.ycombinator.com/item?id=40011762 https://news.ycombinator.com/item?id=40011762> about "just pull fiber" when folks were discussing pulling cat6e and a lot of that talk was about the relative cost versus massive bandwidth gains. My mental model of GPU interconnect is that one wishes to have the most bandwidth possible

westurner 3y ago
https://www.windowscentral.com/hardware/computers-desktops/nvidia-kills-off-nvlink-on-rtx-4090 https://www.windowscentral.com/hardware/computers-desktops/n... : > "The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence. > For gamers, creators, and workstation users who want a multi-GPU setup with the latest RTX 4000 series, Nvidia is switching from NVLink as a bridge to the PCIe Gen 5 standard. PCIe 5.0 x32 supports 128GB/s. "PCIe 7.0 Draft 0.5 Spec: 512 GB/s over PCIe x16 On Track For 2025" https://news.ycombinator.com/item?id=39933959 https://news.ycombinator.com/item?id=39933959 > The HBM3E [RAM] Wikipedia article says 1.2TB/s (2024) https://news.ycombinator.com/item?id=39938652 https://news.ycombinator.com/item?id=39938652 "Fiber-optic data transfer speeds hit a rapid 301 Tbps" (2024-03) https://news.ycombinator.com/item?id=39864107 https://news.ycombinator.com/item?id=39864107 HPE Cray Slingshot is a copper Ethernet HPC interconnect: https://www.nextplatform.com/2022/01/31/crays-slingshot-interconnect-is-at-the-heart-of-hpes-hpc-and-ai-ambitions/?amp https://www.nextplatform.com/2022/01/31/crays-slingshot-inte... ... "HPE Slingshot Makes The GPUs Do Control Plane Compute" https://www.nextplatform.com/2022/08/15/hpe-slingshot-makes-the-gpus-do-control-plane-compute/?amp https://www.nextplatform.com/2022/08/15/hpe-slingshot-makes-... : > Here’s the gist. With normal GPU aware MPI software, inside the node the GPU vendors – the only two that matter today are Nvidia and AMD – have their own mechanisms for doing peer-to-peer communications between the GPUs – in this case NVLink and Infinity Fabric. For MPI data exchanges between GPUs located in different nodes, which is often required for running large simulations and models as well as for large AI training runs, MPI data movement is done through Remote Direct Memory Access methods, pioneered with InfiniBand adapters decades ago, which allow for the transfer of data between a GPU and a network interface card without the interaction of the host CPU’s network stack. [...] > In a [recent paper], HPE showed off the new MPI offload method, making the distinction between the typical MPI communication that is GPU aware versus the stream triggered method that is GPU stream aware. "Exploring GPU Stream-Aware Message Passing using Triggered Operations" (2022) https://arxiv.org/abs/2208.04817 https://arxiv.org/abs/2208.04817 https://scholar.google.com/scholar?cites=13547610567386067445&as_sdt=5,43&sciodt=0,43&hl=en https://scholar.google.com/scholar?cites=1354761056738606744...
- westurner 3y ago
  "New RISC-V microprocessor can run CPU, GPU, and NPU workloads simultaneously" (2024) https://news.ycombinator.com/item?id=39938538 https://news.ycombinator.com/item?id=39938538
- justinclift 3y ago
  > ... with the latest RTX 4000 series, Nvidia is switching from NVLink as a bridge to the PCIe Gen 5 standard. The theory sounds nice, but isn't the RTX 4090 only PCIe gen 4, not 5?
  - gmerc 3y ago
    That’s the beauty of it.
    - justinclift 3y ago
      Not really sure what you're meaning?
      - cyanydeez 3y ago
        Pushing plebs to buy faster gpus requires limiting consumer options
blagie 3y ago
* Wired ethernet is now usually slower than my 802.11. 802.11ax is 0.5-10Gbps in theory, and in practice, faster than gigabit ethernet. * Fiber seems... overpriced. I'm not quite sure why. USB TOSlink is $3 on Aliexpress, as is the cable, so there's nothing fundamentally expensive there, except for being enterprisey. A lot more interconnect would make sense to have as optical. * Wifi is increasing in speed. Wifi 8 is planned to hit 100Gbps. That's where the investment seems to be going.
- rowanG077 3y ago
  How is the latency and consistency on these newer WiFi standards? From my experience if you hit the sweet spot on WiFi it's very good. But then someone else starts a youtube video. Or you move your laptop. Or maybe even do nothing noticeable at all. And suddenly you have a shit experience. For this reason I have always preferred Ethernet. But I admit I have not compared the more recent WiFi standards.
  - blagie 3y ago
    That's actually what's surprised me. In my home, I found wifi to work better in all respects than ethernet. This is a relatively recent phenomenon. Ethernet has issues too, at times. I've certainly seen nominal 10Gbps networks degrade to <1Gbps, and to fix that, you need to debug wires and connectors, and see where the signal integrity issue too. As speeds go up, those issues get more common. 10Mbps, you could almost literally run a pair of wires from Radio Shack and it would work. 1Gbps was the breakpoint where if I didn't use proper everything (which is often mislabelled on Amazon / Aliexpress / eBay / etc.), things would degrade. On the other hand, my wifi has a nice MIMO router. I expect that split will continue. In theory, a MIMO router with an array of antennas could be almost point-to-point, directing signals where they need to go. At the same time, as speeds go up, any parasitic cable / connector inductance or capacitance will have increasing impact.
- vrighter 3y ago
  toslink is a piece of junk plastic fiber and a red led. it's technically fiber optic, but can't really be compared to real fiber optic system

rowanG077 3y ago

Well I'd wager Nvidia is very happy they use their own proprietary interconnect. They do everything to strap their customers in a straight jacket and milk them for everything they are worth.

FireBeyond 3y ago
Perhaps so, but also, "Nvidia competitor suggests Nvidia should have done something that makes it easier to compete against them..." isn't exactly the revelation.
- davidf18 3y ago
  [dead]

shrubble 3y ago

I think that Infiniband was previously an open standard, but over time only Mellanox was in the market ( as other companies stopped making product) and they were then bought by Nvidia. I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency.

justinclift 3y ago
> I don't agree [...] absolute highest performance is found in Infiniband, not only with bandwidth but latency. What's the alternative(s) you're thinking of? There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially. Omni-path (a descendant of Infiniband) doesn't seem to have worked out either. --- Infiniband also has advantages over Ethernet in other things, like bundling links together. With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream. With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.
- shrubble 3y ago
  I re-worded what I wrote for clarity, it's an obscure sort of sentence construction. "I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. " I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.
  - justinclift 3y ago
    No worries, my misunderstanding. :)
- oasisaimlessly 3y ago

Proven 3y ago

[dead]

gmerc 3y ago

Blackwell is a platform game. Interoperability would defeat the purpose.

touisteur 3y ago

I say let Jim Keller show us all how it's done, as he's often done. Let him convince AMD (with their own interconnect and not specially on lead in Ethernet these days, maybe with Xilinx or Pensando but I haven't seen any serious integration in the road map?), Intel (Xe-Link on GPUs? Ethernet on Gaudi, both unobtainium? And they stopped working on their actual lead in switching/Ethernet with Tofino, WTAF BTW) and whoever else is shipping accelerator hardware to go full ethernet and maybe there's something. But why would NVIDIA do anything to share this moat? Or even risk to have their whole ecosystem go glitchy and buggy with other hardware vendors? They invested heavily, very heavily, in network tech and did the whole integration (nccl, gpudirect, and even simple memcopy works fine with NVLINK without futzing with actual Ethernet/IP stacks) at a moment where everyone was fucking around either building GPUs back from scratch or buying and dumping good tech? If there was an actually competitive alternative, I could see Jensen have a look but I don't see it right now.