Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs. 1:4 gives 1:2 on Torch and Caffe,

Yes it looks like they have architected for low communication overhead and true parallelization when you have multiple GPUs. 1:4 gives 1:2 on Torch and Caffe, and 1:4 = 1:4 on CNTK. No idea what the catch is :)