3 ms·

Don't want to discourage you, but the problem is that super-large runs generally require super-well-debugged code to scale to such large core numbers. Hence it'

by lutorm 14y ago

Don't want to discourage you, but the problem is that super-large runs generally require super-well-debugged code to scale to such large core numbers. Hence it's pretty difficult to do such a run without gradually stepping up with progressively larger runs, ironing out the bugs, running a bigger run, etc. This is likely to be extra hard for hybrid GPU/distributed applications, and the cycle time for these very large runs becomes long so the process is slow. Our research group was determined to run the largest hydrodynamic cosmological simulation ever done, using a new code. Over a year ago the code was "basically ready" and had been used on "small" runs of ~hundreds of k CPU hours. They still haven't completed a big run...

jmt3 14y ago
I'm really curious what sorts of bugs you are referring to. Could you elaborate on what the hurdles are going from a small run to a large one?
- dalke 14y ago
  Classically speaking, it's hard to get linear scaling for a large number of cores. Eg, on a small, 128 core machine, one algorithm I did recently gives maximum throughput with 15 cores, and that only gives a 10x speedup. To make effective use of all 128 cores I would have to pay a lot more attention to how the data flow maps to the architecture. That's hard. For example, one of the older pieces of hardware used a ring-based topology, which was more appropriate for a force decomposition, while another network architecture made a spatial decomposition easier. And on that old ring-based topology system, the previous developers could program the network chip to handle I/O while the CPU was doing the dynamics, in order to eek out even more performance. In general, if you scale by more than one or two orders of magnitude then you're going to run into new, unexpected problems. In one project I worked on, the code was fast on all of the test sets, but when scaled up to the entire data set it was unexpectedly slow. We traced that down to a logger which we hadn't realized was enabled. In your code you might find that a minor piece of code used order n-squared time but because of data partitioning, it has minimal impact. Or you might find that you have a barrier, where everything is supposed to be synchronized. (Eg, once every 1,000 time steps you rebalance the system load.) A few stragglers might not affect small system sizes, but cause 50% slowdowns with large numbers of CPUs. You might even have to rewrite the code to remove the barrier and come up with another way to do dynamic load balancing. Bram Cohen, in his Stanford EE380/2005 talk on BitTorrent (linked from http://bramcohen.livejournal.com/11025.html?nojs=1 http://bramcohen.livejournal.com/11025.html?nojs=1 , at 8:45) mentioned how TCP's checksum isn't good enough once you start swapping terabytes of data over the Internet, so BitTorrent has to work around that. On the topic of flakiness, Google's own Big Table/MapReduce, etc. is designed to be able to handle machine faults, which can happen with large numbers of cores. How well does your code handle those sorts of rare failures? In another project I worked on, there was a bug in the code. It didn't conserve energy as well as we thought it should. That was eventually tracked down to a trigonometry error where we didn't compute one of the energy terms correctly when the angle was exactly 0.0. This doesn't happen that frequently, which is why we only saw it with large simulations run for a long time.
- lutorm 14y ago
  dalke mentioned some good examples. In our case, the most annoying problem we had was that the entire run would sometimes just hang. I'm not aware that the problem was tracked down, but the author suspected some race condition in the MPI library that caused some message to get lost and thus some task to wait indefinitely.

by lutorm 14y ago

jmt3 14y ago

I'm really curious what sorts of bugs you are referring to. Could you elaborate on what the hurdles are going from a small run to a large one?

dalke 14y ago
Classically speaking, it's hard to get linear scaling for a large number of cores. Eg, on a small, 128 core machine, one algorithm I did recently gives maximum throughput with 15 cores, and that only gives a 10x speedup. To make effective use of all 128 cores I would have to pay a lot more attention to how the data flow maps to the architecture. That's hard. For example, one of the older pieces of hardware used a ring-based topology, which was more appropriate for a force decomposition, while another network architecture made a spatial decomposition easier. And on that old ring-based topology system, the previous developers could program the network chip to handle I/O while the CPU was doing the dynamics, in order to eek out even more performance. In general, if you scale by more than one or two orders of magnitude then you're going to run into new, unexpected problems. In one project I worked on, the code was fast on all of the test sets, but when scaled up to the entire data set it was unexpectedly slow. We traced that down to a logger which we hadn't realized was enabled. In your code you might find that a minor piece of code used order n-squared time but because of data partitioning, it has minimal impact. Or you might find that you have a barrier, where everything is supposed to be synchronized. (Eg, once every 1,000 time steps you rebalance the system load.) A few stragglers might not affect small system sizes, but cause 50% slowdowns with large numbers of CPUs. You might even have to rewrite the code to remove the barrier and come up with another way to do dynamic load balancing. Bram Cohen, in his Stanford EE380/2005 talk on BitTorrent (linked from http://bramcohen.livejournal.com/11025.html?nojs=1 http://bramcohen.livejournal.com/11025.html?nojs=1 , at 8:45) mentioned how TCP's checksum isn't good enough once you start swapping terabytes of data over the Internet, so BitTorrent has to work around that. On the topic of flakiness, Google's own Big Table/MapReduce, etc. is designed to be able to handle machine faults, which can happen with large numbers of cores. How well does your code handle those sorts of rare failures? In another project I worked on, there was a bug in the code. It didn't conserve energy as well as we thought it should. That was eventually tracked down to a trigonometry error where we didn't compute one of the energy terms correctly when the angle was exactly 0.0. This doesn't happen that frequently, which is why we only saw it with large simulations run for a long time.
lutorm 14y ago
dalke mentioned some good examples. In our case, the most annoying problem we had was that the entire run would sometimes just hang. I'm not aware that the problem was tracked down, but the author suspected some race condition in the MPI library that caused some message to get lost and thus some task to wait indefinitely.