4 ms·

As far as I understand, you're saying that high-speed interconnects are largely a waste of time? (and apparently that doing protein MD is a waste of time, but t

by tfgg 13y ago

As far as I understand, you're saying that high-speed interconnects are largely a waste of time? (and apparently that doing protein MD is a waste of time, but that's a scientific question that I'm going to disagree with you on). I can imagine that might be the case for some simulations, but how I do, for example, a parallel FFT without significant communication? I personally see it more as there's a pretty limited set of computational problems which are low-communication, often requiring rather gross approximations, even with the "clever computer scientists" looking at the problem, and they're often not the problems scientists actually want to solve, and so dismissing supercomputing is rather ridiculous.

dekhn 13y ago
High speed interconnects are not worth the extra cost. They just compensate for poor programming. With regards to protein MD, no, I'm not saying protein MD is a waste of time. In fact, I run the Exacycle program at Google which has run the largest MD simulations (many milliseconds) ever done, and the results are quite good. You would never have gotten results as good as ours on a supercomputer- even the world's largest, with the most highly scaling MD codes. I speak from experience- I used to code for supercomputers, in fact working on protein (and DNA/RNA, my interest) structural dynamics, with some of the leading MD codes. I helped port a better approach to Google's infrastructure some time ago. It's not a clever hack or gross approximation: https://simtk.org/home/msmbuilder https://simtk.org/home/msmbuilder it's a distinctly better way of modelling protein dynamics than running long single trajectories on supercomputers.
- tfgg 13y ago
  > High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implementation of the existing one. You still haven't explained how I do a parallel FFT without high speed interconnects -- I suspect the answer is "don't". I've tried running the stuff I do (DFT) over a 10GigE network on my cluster and it ran at about 10% of the speed of an Infiniband-enabled calculation. There are methods for getting better scaling, but they all (as I said) involve rather crass approximations which you don't always want to do. Those methods will probably increasingly be used more on large supercomputers due to their superior scaling characteristics, even with their nice high-speed interconnects, but you're still making a sacrifice in accuracy. It might be possible to get good low-communication scaling for some models, as you were in the case of your sampling system (which typically parallelizes nicely, but only if the individual sampling jobs can fit on a single node), but you can't extrapolate that to assume that everyone can. As I said, exchanging a poorly-scaling model for a different, non-equivalent well-scaling model is a scientific question with tradeoffs, not a programming one.
  - dekhn 13y ago
    Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. but lots of effort has been put into making the MPI libraries work efficiently over it, compared to 10GBe, so codes run more effficiently (as you observed). DFT seems like the next thing that's going to not need supercomputers. I see some nice new algorithms coming on line designed for Amazon cloud infrastructure.
    - deleted 13y ago
      [deleted]
    - xtreme 13y ago
      > note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimized MPI for ethernet. In fact, standards like RoCE and iWarp have been devised by Ethernet working group to compete with infiniband on this particular metric.
      - dekhn 13y ago
        actually, wire latencies for 10GbE are much better than you think- they are basically the same. Anyway, my point was that infiniband is not a high speed interconnect, and it's cheaper than 10GbE. The challenge is to deliver scaled infiniband, which is far harder than scaled ethernet.

by tfgg 13y ago

dekhn 13y ago

High speed interconnects are not worth the extra cost. They just compensate for poor programming. With regards to protein MD, no, I'm not saying protein MD is a waste of time. In fact, I run the Exacycle program at Google which has run the largest MD simulations (many milliseconds) ever done, and the results are quite good. You would never have gotten results as good as ours on a supercomputer- even the world's largest, with the most highly scaling MD codes. I speak from experience- I used to code for supercomputers, in fact working on protein (and DNA/RNA, my interest) structural dynamics, with some of the leading MD codes. I helped port a better approach to Google's infrastructure some time ago. It's not a clever hack or gross approximation: https://simtk.org/home/msmbuilder https://simtk.org/home/msmbuilder it's a distinctly better way of modelling protein dynamics than running long single trajectories on supercomputers.

tfgg 13y ago
> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implementation of the existing one. You still haven't explained how I do a parallel FFT without high speed interconnects -- I suspect the answer is "don't". I've tried running the stuff I do (DFT) over a 10GigE network on my cluster and it ran at about 10% of the speed of an Infiniband-enabled calculation. There are methods for getting better scaling, but they all (as I said) involve rather crass approximations which you don't always want to do. Those methods will probably increasingly be used more on large supercomputers due to their superior scaling characteristics, even with their nice high-speed interconnects, but you're still making a sacrifice in accuracy. It might be possible to get good low-communication scaling for some models, as you were in the case of your sampling system (which typically parallelizes nicely, but only if the individual sampling jobs can fit on a single node), but you can't extrapolate that to assume that everyone can. As I said, exchanging a poorly-scaling model for a different, non-equivalent well-scaling model is a scientific question with tradeoffs, not a programming one.
- dekhn 13y ago
  Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. but lots of effort has been put into making the MPI libraries work efficiently over it, compared to 10GBe, so codes run more effficiently (as you observed). DFT seems like the next thing that's going to not need supercomputers. I see some nice new algorithms coming on line designed for Amazon cloud infrastructure.
  - deleted 13y ago
    [deleted]
  - xtreme 13y ago
    > note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimized MPI for ethernet. In fact, standards like RoCE and iWarp have been devised by Ethernet working group to compete with infiniband on this particular metric.
    - dekhn 13y ago
      actually, wire latencies for 10GbE are much better than you think- they are basically the same. Anyway, my point was that infiniband is not a high speed interconnect, and it's cheaper than 10GbE. The challenge is to deliver scaled infiniband, which is far harder than scaled ethernet.