This may not be true in the future. There is some work being done on distributed neural net training. I can't recall the name of the technique at the moment, bu

This may not be true in the future. There is some work being done on distributed neural net training. I can't recall the name of the technique at the moment, but a paper came out in the last year showing results comparable with backprop that only required local communication of information (whatever this technique's alternative to gradients is).