10 ms·

But can't you make the same argument about bandwidth? Every second that the processing units are computing, terabits of bandwidth are unused.

• magicalhippo 6y ago
  Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.
• dooglius 6y ago
  No, computing is happening in parallel with data being sent/received
  • gnufx 6y ago
    What fraction of communication on a typical HPC system is overlapped with computation? My guess is rather little, despite the Mellanox sales pitch. I do wish people commonly made cluster-global profiling-type measurements to be able to answer that sort of question and do optimization.
    • fluffything 6y ago
      Heavily depends on the application. Many applications running on a Top 10 system of the Top 500 are required to show good single threaded performance and good weak and strong scalings up to a big part of the system. Checking all these 3 check boxes is hard to do if your communication is not fully overlapped by computation. The smaller your problem, the better your single threaded performance (e.g. at some point your problem fits in L3), and the larger fraction of execution time that communication dominates. Even the poorest applications on these systems use non-blocking MPI to overlap communication with computation nowadays.
• fluffything 6y ago
  Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument. If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful. A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities. So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D
• dekhn 6y ago
  Ultimately what everybody wants is a balanced system (that acts balanced on the workloads of interest). Put another way: you don't want any one part of the system to be massively overprovisioned in a way that causes expensive bottlenecks. The Cray T3E had fairly slow processors, but you coudl keep the entire supercomputer at very high utilization and get better throughput that machines with faster processors and a slower interconnect.