4 ms·

Note: I have never worked with an HPC system, take what I say with a grain of salt. I suspect the answer is in the networking, what this article spends most of

Note: I have never worked with an HPC system, take what I say with a grain of salt. I suspect the answer is in the networking, what this article spends most of its time talking about. HPC networking is so ridiculously much better than commodity networking. 1us latencies vs 100us+, 200Gbps bandwidth vs 10Gbps. I imagine certain kinds of especially simulations involve a ton of communication between nodes simulating nearby cells and everything is completely bottlenecked on that. It might even be that you can't even match a supercomputer's speed by just buying more AWS instances because the scaling factors on the communication mean more instances doesn't make your sim faster. Again this is mostly inference on my part from things I've read, no first-hand knowledge.

jiggawatts 6y ago
The hilarity is that most business environments could benefit from the same networking "tricks" used by HPC, but don't bother to set any of it up. I've even seen a few places that have even purchased the prerequisite hardware but then never turned the features on. Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency.
- wmf 6y ago
  When I was doing a private cloud last year these features were way more work than checkboxes, but maybe that's what we got for using OpenStack. I finished the planning for jumbo frames to the VM (this was complicated by double VXLAN), SR-IOV was blocked because we only had ConnectX-4 but SR-IOV over LAG requires ConnectX-5, and the storage team never got around to Ceph over RoCE.
  - jiggawatts 6y ago
    Most enterprises run VMware, Hyper-V, or XenServer. Almost always with Cisco Nexus 9K series switching. With that kit it's surprisingly easy, but not with the default Broadcom NICs, they generally don't have the required features. Even if they do, the advanced features often don't work properly. I've seen jaw-dropping performance when it was set up correctly, something like 99% of the theoretical wire rate of 20 Gbps. With dual NICs and Windows 2016 we got nearly 40 Gbps because it can use both NICs at once! Amazing file copy times. I tested it with the usual 4GB ISO file copy test and it was so fast there wasn't even a progress bar! Meanwhile, on most networks I see about 1.5-3.0 Gbps effective across dual 10 Gbps NICs. If you think about it, that's crazy bad. It means that in effect they're getting less than 10% of the available physical capacity for data traffic. The other 90% is being thrown away by "passive" secondary cables or by the lack of configuration on the switching layer.
    - exikyut 6y ago
      I discovered the other day that you can google `Xgb/Xgbps` and it'll give you a result in seconds. 4GB is 0.8 seconds :) And regarding that 1.5-3.0Gbps across 2x10Gbps... a) wat?!?! and b) I wonder if the number of horribly misconfigured networks exactly like that are out there giving 10Gig a bad name and hurting adoption.
  - timeu 6y ago
    Yeah, I can feel your pain. We deployed a fully virtualized HPC cluster on top of OpenStack and there was a fair amount of testing and playing with knobs (SRIOV, Jumbo frames, CPU pinning, etc) to get close to bare metal performance on a 100 GBit/s SDN fabric.
- duxup 6y ago
  Managing flow control, misc switching changes, mtu is as much an art of trial and error as it is a science. Unless you are in a very controlled environment it is VERY easy to actually slow down a network while trying to make it more efficient.
- eecc 6y ago
  Yup, and quite probably no one wants to turn them on for fear of bugs and sudden disruptions. Besides, IT is a cost right? /s The rock-bottom cheap support company outsourcing the infra maintenance is more careful about minimizing actual work and matching the exact wording in the support contract and not a cent more
- gnufx 6y ago
  Indeed. Infiniband was even meant for "datacentre" use, and I have better experience of it in compute clusters than Ethernet apart from the performance.
  - duxup 6y ago
    Infiniband is pretty neat for a given use case but that cable length kinda killed it for many data center use cases. Still, if folks can deal with those short distances there's lots of inexpensive equipment out there.
    - gnufx 6y ago
      Point taken, but for some time Mellanox has touted at least cross-campus-type IB. I never persuaded anyone to evaluate it for use between two machine rooms, but there were obvious non-HPC cases for it within them. I'd be interested to know how the long distance stuff actually works, not being a beginner with sales talk.
      - duxup 6y ago
        That's a good question. I haven't used it but extending infinaband ... I'm guessing there's some sort of intermediary devices doing some sort of piplining or emulation, that's usually how that works out.
- GhettoMaestro 6y ago
  > Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency. This comment's dismissive tone really leads me to believe you have never really dealt with the details of said technologies.
kijiki 6y ago
You can get 400Gbps bandwidth in commodity networking: https://www.fs.com/products/96982.html https://www.fs.com/products/96982.html Not going to get that sort of latency without extreme custom chips though.
- jiggawatts 6y ago
  Wow, that's not even that expensive! I generally don't get to play with hardware much these days, but my "dream" on-premises kit would be a hyperconverged cluster with AMD EPYC 2 CPUs, NVMe storage, and 400 Gbps networking. It would be interesting to see what kind of performance that would enable compared to the typical cloud platforms everyone is so excited about...
  - briffle 6y ago
    Don't forget to add about ~$1000 per port for a 400GB transceiver.
- gnufx 6y ago
  Gbps isn't a unit of latency (or message rate).
- exikyut 6y ago
  ($15K)

jiggawatts 6y ago

The hilarity is that most business environments could benefit from the same networking "tricks" used by HPC, but don't bother to set any of it up. I've even seen a few places that have even purchased the prerequisite hardware but then never turned the features on. Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency.

wmf 6y ago
When I was doing a private cloud last year these features were way more work than checkboxes, but maybe that's what we got for using OpenStack. I finished the planning for jumbo frames to the VM (this was complicated by double VXLAN), SR-IOV was blocked because we only had ConnectX-4 but SR-IOV over LAG requires ConnectX-5, and the storage team never got around to Ceph over RoCE.
- jiggawatts 6y ago
  Most enterprises run VMware, Hyper-V, or XenServer. Almost always with Cisco Nexus 9K series switching. With that kit it's surprisingly easy, but not with the default Broadcom NICs, they generally don't have the required features. Even if they do, the advanced features often don't work properly. I've seen jaw-dropping performance when it was set up correctly, something like 99% of the theoretical wire rate of 20 Gbps. With dual NICs and Windows 2016 we got nearly 40 Gbps because it can use both NICs at once! Amazing file copy times. I tested it with the usual 4GB ISO file copy test and it was so fast there wasn't even a progress bar! Meanwhile, on most networks I see about 1.5-3.0 Gbps effective across dual 10 Gbps NICs. If you think about it, that's crazy bad. It means that in effect they're getting less than 10% of the available physical capacity for data traffic. The other 90% is being thrown away by "passive" secondary cables or by the lack of configuration on the switching layer.
  - exikyut 6y ago
    I discovered the other day that you can google `Xgb/Xgbps` and it'll give you a result in seconds. 4GB is 0.8 seconds :) And regarding that 1.5-3.0Gbps across 2x10Gbps... a) wat?!?! and b) I wonder if the number of horribly misconfigured networks exactly like that are out there giving 10Gig a bad name and hurting adoption.
- timeu 6y ago
  Yeah, I can feel your pain. We deployed a fully virtualized HPC cluster on top of OpenStack and there was a fair amount of testing and playing with knobs (SRIOV, Jumbo frames, CPU pinning, etc) to get close to bare metal performance on a 100 GBit/s SDN fabric.
duxup 6y ago
Managing flow control, misc switching changes, mtu is as much an art of trial and error as it is a science. Unless you are in a very controlled environment it is VERY easy to actually slow down a network while trying to make it more efficient.
eecc 6y ago
Yup, and quite probably no one wants to turn them on for fear of bugs and sudden disruptions. Besides, IT is a cost right? /s The rock-bottom cheap support company outsourcing the infra maintenance is more careful about minimizing actual work and matching the exact wording in the support contract and not a cent more
gnufx 6y ago
Indeed. Infiniband was even meant for "datacentre" use, and I have better experience of it in compute clusters than Ethernet apart from the performance.
- duxup 6y ago
  Infiniband is pretty neat for a given use case but that cable length kinda killed it for many data center use cases. Still, if folks can deal with those short distances there's lots of inexpensive equipment out there.
  - gnufx 6y ago
    Point taken, but for some time Mellanox has touted at least cross-campus-type IB. I never persuaded anyone to evaluate it for use between two machine rooms, but there were obvious non-HPC cases for it within them. I'd be interested to know how the long distance stuff actually works, not being a beginner with sales talk.
    - duxup 6y ago
      That's a good question. I haven't used it but extending infinaband ... I'm guessing there's some sort of intermediary devices doing some sort of piplining or emulation, that's usually how that works out.
GhettoMaestro 6y ago
> Things like hardware flow control, cut-through switching, jumbo frames, SR-IOV, RDMA, etc... are just a few checkboxes away and can give even small VMs crazy good bandwidth and latency. This comment's dismissive tone really leads me to believe you have never really dealt with the details of said technologies.

kijiki 6y ago

You can get 400Gbps bandwidth in commodity networking: https://www.fs.com/products/96982.html https://www.fs.com/products/96982.html Not going to get that sort of latency without extreme custom chips though.

jiggawatts 6y ago
Wow, that's not even that expensive! I generally don't get to play with hardware much these days, but my "dream" on-premises kit would be a hyperconverged cluster with AMD EPYC 2 CPUs, NVMe storage, and 400 Gbps networking. It would be interesting to see what kind of performance that would enable compared to the typical cloud platforms everyone is so excited about...
- briffle 6y ago
  Don't forget to add about ~$1000 per port for a 400GB transceiver.
gnufx 6y ago
Gbps isn't a unit of latency (or message rate).
exikyut 6y ago
($15K)