There are alternatives to backprop like direct feedback alignment (1). They specifically focus on making the computation more parallel and thus more scaleable f

There are alternatives to backprop like direct feedback alignment (1). They specifically focus on making the computation more parallel and thus more scaleable for the reasons you mention. (1) http://papers.nips.cc/paper/6441-direct-feedback-alignment-provides-learning-in-deep-neural-networks.pdf http://papers.nips.cc/paper/6441-direct-feedback-alignment-p...