>So computing a stochastic gradient over each example in parallel, and them aggregating them, isn't necessarily a bad idea. Wait a minute, I don't think that's

>So computing a stochastic gradient over each example in parallel, and them aggregating them, isn't necessarily a bad idea. Wait a minute, I don't think that's what the paper is suggesting. The communication costs of aggregating every micro-step of SGD would be extreme. I think Jimmy Lin is saying you should partition the data, train one model per partition using SGD, and then combine the models in an ensemble.