>In the brain, neurons cannot wait for a sequential forward and backward sweep. I think that neurons hate when we animate them. Being a little more serious, i

>In the brain, neurons cannot wait for a sequential forward and backward sweep. I think that neurons hate when we animate them. Being a little more serious, is it possible to perform gradual back propagation? I.e., for iteration 1 train only last layer, for iteration 2 change two last layers, for iteration 4 train three last layers, etc. I also have to say that in my spare time I am investigating a way to train networks on the whole set as opposed to the single sample or batch back propagation and one result I've got so far is not quite uninteresting, in my opinion: https://github.com/thesz/higgs-logistic-regression https://github.com/thesz/higgs-logistic-regression The approximation of gradients is still use of gradients. It still computes updates which are need to be shared and still has some bottleneck on scalability.