3 ms·

Do you get MPI ping-pong latencies on the order of a microsecond on a "normal" public cloud? No? Well, MPI applications that are sensitive to latency is one us

by jabl 9y ago

Do you get MPI ping-pong latencies on the order of a microsecond on a "normal" public cloud? No? Well, MPI applications that are sensitive to latency is one usecase where a "real" supercomputer can be useful.

KGIII 9y ago
Ah! This is even better. It led me to finding this: http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi I do believe I get it. Now to find out what kind of applications are greatly benefited from this. Thanks HN! You always make me dig in and learn new things! Edit: It was "MPI latency" that led me to that result, by the way.
- vmarsy 9y ago
  > Now to find out what kind of applications are greatly benefited from this. The common applications are scientific computing usually, here's a quick overview[1] of the kind of algorithms ran on supercomputers: PDEs, ODEs, FFTs, Sparse and Dense linear algebra, etc. These are usually used for scientific applications like weather forecasting, where you need to know about the result on time (i.e. before the hurricane reaches the coast!) [1] https://www.nap.edu/read/11148/chapter/7#125 https://www.nap.edu/read/11148/chapter/7#125
  - KGIII 9y ago
    I'll scrape the whole book and read it. Thanks! I know weather models still do it on supercomuters but understood the currently have plenty. I look forward to reading the book.
deleted 9y ago
[deleted]
XenophileJKO 9y ago
Public cloud, maybe not that far away. Linkedin's new data centers have sub 400ns switching and 100G interconnects. https://engineering.linkedin.com/blog/2016/03/project-altair--the-evolution-of-linkedins-data-center-network https://engineering.linkedin.com/blog/2016/03/project-altair...
- jabl 9y ago
  Yes, big hyperscalers are adopting CLOS networks as used in HPC for decades. 400ns switching, per se, is not that bad (IIRC 100G IB switches have around 100ns switching cost). Though I wonder when they are doing ECMP at the L3 level what the latency is when crossing multiple switches (for comparison, IB does ECMP at the L2 level)? Also, you need software to take advantage of this as well. MPI stacks tend to use RDMA with user-space networking. For ethernet, there is RoCE v1/v2 which is comparable to IB RDMA, though I'm not sure how mature that is at this point. Certainly it's true that ethernet has stepped up the game in recent years, while at the same time IB has more or less become a single-vendor play. So it'll be interesting to see what the future holds. Further, proprietary supercomputer networks have things like adaptive non-minimal routing, which enables efficient use of network topologies that are more affordable at scale, such as flattened butterfly, dragonfly etc. AFAIK neither IB nor ETH + IP-level ECMP support anything like that.
  - sitkack 9y ago
    For special topologies, one can put multiple IB nics in a single host. Nvidia has support for doing DMA transfers directly into GPU memory from 3d-party PCIe devices [0] [0] http://docs.nvidia.com/cuda/gpudirect-rdma/index.html http://docs.nvidia.com/cuda/gpudirect-rdma/index.html

by jabl 9y ago

KGIII 9y ago

Ah! This is even better. It led me to finding this: http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi I do believe I get it. Now to find out what kind of applications are greatly benefited from this. Thanks HN! You always make me dig in and learn new things! Edit: It was "MPI latency" that led me to that result, by the way.

vmarsy 9y ago
> Now to find out what kind of applications are greatly benefited from this. The common applications are scientific computing usually, here's a quick overview[1] of the kind of algorithms ran on supercomputers: PDEs, ODEs, FFTs, Sparse and Dense linear algebra, etc. These are usually used for scientific applications like weather forecasting, where you need to know about the result on time (i.e. before the hurricane reaches the coast!) [1] https://www.nap.edu/read/11148/chapter/7#125 https://www.nap.edu/read/11148/chapter/7#125
- KGIII 9y ago
  I'll scrape the whole book and read it. Thanks! I know weather models still do it on supercomuters but understood the currently have plenty. I look forward to reading the book.

deleted 9y ago

[deleted]

XenophileJKO 9y ago

Public cloud, maybe not that far away. Linkedin's new data centers have sub 400ns switching and 100G interconnects. https://engineering.linkedin.com/blog/2016/03/project-altair--the-evolution-of-linkedins-data-center-network https://engineering.linkedin.com/blog/2016/03/project-altair...

jabl 9y ago
Yes, big hyperscalers are adopting CLOS networks as used in HPC for decades. 400ns switching, per se, is not that bad (IIRC 100G IB switches have around 100ns switching cost). Though I wonder when they are doing ECMP at the L3 level what the latency is when crossing multiple switches (for comparison, IB does ECMP at the L2 level)? Also, you need software to take advantage of this as well. MPI stacks tend to use RDMA with user-space networking. For ethernet, there is RoCE v1/v2 which is comparable to IB RDMA, though I'm not sure how mature that is at this point. Certainly it's true that ethernet has stepped up the game in recent years, while at the same time IB has more or less become a single-vendor play. So it'll be interesting to see what the future holds. Further, proprietary supercomputer networks have things like adaptive non-minimal routing, which enables efficient use of network topologies that are more affordable at scale, such as flattened butterfly, dragonfly etc. AFAIK neither IB nor ETH + IP-level ECMP support anything like that.
- sitkack 9y ago
  For special topologies, one can put multiple IB nics in a single host. Nvidia has support for doing DMA transfers directly into GPU memory from 3d-party PCIe devices [0] [0] http://docs.nvidia.com/cuda/gpudirect-rdma/index.html http://docs.nvidia.com/cuda/gpudirect-rdma/index.html