Actually, I don't quite understand their method. At the end of the day, it is not a gigantic shared memory, as in Hogwild! paper conveniently describes. You nee

Actually, I don't quite understand their method. At the end of the day, it is not a gigantic shared memory, as in Hogwild! paper conveniently describes. You need to synchronize slave's model with master model, in model parallelism case, you need to send responses to another machine for further processes. This sounds extremely complicated actually to implement in modern HPC environment (assuming MPI as the main implementation technique). Sorry if this sounds a bit defensive than it is meant to be. I just think that there are much more to the implementation than the simple Hogwild! approach (probably some clever pipelining to cover the communication / computation cost as the Hogwild! approach relaxed consistency of the model). It seems that this year's ImageNet classification competition result will be more interesting than what I expected.