3 ms·

> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be

by tfgg 13y ago

> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implementation of the existing one. You still haven't explained how I do a parallel FFT without high speed interconnects -- I suspect the answer is "don't". I've tried running the stuff I do (DFT) over a 10GigE network on my cluster and it ran at about 10% of the speed of an Infiniband-enabled calculation. There are methods for getting better scaling, but they all (as I said) involve rather crass approximations which you don't always want to do. Those methods will probably increasingly be used more on large supercomputers due to their superior scaling characteristics, even with their nice high-speed interconnects, but you're still making a sacrifice in accuracy. It might be possible to get good low-communication scaling for some models, as you were in the case of your sampling system (which typically parallelizes nicely, but only if the individual sampling jobs can fit on a single node), but you can't extrapolate that to assume that everyone can. As I said, exchanging a poorly-scaling model for a different, non-equivalent well-scaling model is a scientific question with tradeoffs, not a programming one.

dekhn 13y ago
Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. but lots of effort has been put into making the MPI libraries work efficiently over it, compared to 10GBe, so codes run more effficiently (as you observed). DFT seems like the next thing that's going to not need supercomputers. I see some nice new algorithms coming on line designed for Amazon cloud infrastructure.
- deleted 13y ago
  [deleted]
- xtreme 13y ago
  > note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimized MPI for ethernet. In fact, standards like RoCE and iWarp have been devised by Ethernet working group to compete with infiniband on this particular metric.
  - dekhn 13y ago
    actually, wire latencies for 10GbE are much better than you think- they are basically the same. Anyway, my point was that infiniband is not a high speed interconnect, and it's cheaper than 10GbE. The challenge is to deliver scaled infiniband, which is far harder than scaled ethernet.

by tfgg 13y ago

dekhn 13y ago

Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. but lots of effort has been put into making the MPI libraries work efficiently over it, compared to 10GBe, so codes run more effficiently (as you observed). DFT seems like the next thing that's going to not need supercomputers. I see some nice new algorithms coming on line designed for Amazon cloud infrastructure.

deleted 13y ago
[deleted]
xtreme 13y ago
> note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimized MPI for ethernet. In fact, standards like RoCE and iWarp have been devised by Ethernet working group to compete with infiniband on this particular metric.
- dekhn 13y ago
  actually, wire latencies for 10GbE are much better than you think- they are basically the same. Anyway, my point was that infiniband is not a high speed interconnect, and it's cheaper than 10GbE. The challenge is to deliver scaled infiniband, which is far harder than scaled ethernet.