3 ms·

you would enjoy Dave Ackley's perspective on 'robust computing' vs 'efficient computing'. But his one of his ideas is allowing for error: which can speed up cal

you would enjoy Dave Ackley's perspective on 'robust computing' vs 'efficient computing'. But his one of his ideas is allowing for error: which can speed up calculations if you can also reduce the error as the calculation goes on.

dekhn 3y ago
Well... there's several things going wrong here and his perspective only seems to cover some of them. In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge. Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure. Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines). Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.

dekhn 3y ago

Well... there's several things going wrong here and his perspective only seems to cover some of them. In the case of a scientific simulation, we can find algorithmic speedups that are "lossy"- an example would be approximations to n-body systems, where you need to calculate n-squared interactions (between all pairs). You can calculate all n-squared interactions which produces teh completely correct result. But since atoms that are far away don't interact strongly (falls off as 1 over r squared or more). So you can maintain a neighborlist- all atoms within a distance R- cheaper than you can calculate n-squared interactions, with some tiny error that is unknown. It's assumed in many cases the errror is neglible and the speedup is huge. Or you can switch to using a particle-mesh method which involves taking a fourier transform, doing some work in fourier space, then an inverse transform. The results are nlogn, and the error is small (and known). Speedup is signfiicant but takes much, much more computational skill and infrastructure. Next, the case I'm referring to of an accelerator running a tensorflow job, it's a totally different scenario- here, some random subset of machines will repeatedly return the wrong result- say, for a matrix-vector operation. Maybe garbage numbers, maybe all zero, maybe some infs. When that gets summed into your gradient, it often causes blow-up and the entire job terminates. It's not clear whether it makes sense to make high-performance jobs have to be robust- I see them as special cases where you're working hard to make sure the computing substrate is effectively 100% reliable (by sending/fixing those machines). Other folks have observed that some amount of small noise injection to the gradient can help training, but the sorts of errors I've seen almost immediately terminate the training job. I don't mind intentional noise addition, but noise due to hardware that is provably, reliably, and repeatedly miscalculating results? Not so much.