3 ms·
I'm a little confused by the architecture here. Keras already abstracts TensorFlow and PyTorch - and CNTK. Don't at least TF and CNTK already handle the distrib
by gaius 8y ago
I'm a little confused by the architecture here. Keras already abstracts TensorFlow and PyTorch - and CNTK. Don't at least TF and CNTK already handle the distributed training use cases natively? (I am not familiar with PyTorch). It's certainly one of CNTK's main selling points. What are the use cases for adding yet another layer to the stack? In what way is this simpler than tweaking to use the distributed features of the underlying packages natively? Which is what they are designed for.
- mbeex 8y ago> Keras already abstracts TensorFlow and PyTorch - and CNTK. PyTorch? You mean Theano, right?
- gaius 8y agoYes my mistake! I mainly use Keras and CNTK in R, I made a conscious decision to ignore most of the others to avoid getting bogged down.
- cromulen 8y agoI don't know about CNTK, but Tensorflow and I think PyTorch don't have _good_ distributed training. They use a distributed training model that utilizes parameter servers, which scales nowhere near Horovod's mpi solution. Even for single-machine-multi-gpu solutions, only now in Tensorflow 1.8 is pure tensorflow as fast as Horovod with it's estimator MirroredStrategy. If you watch Tensorflow dev days 2018, the devs say they're working on bringing something like Horovod to pure Tensorflow
- vinn124 8y ago> What are the use cases for adding yet another layer to the stack? in my limited experience with horovod, horovod is most useful when youre running large clusters of workers/ps. in those situations, you typically have to manually find the appropriate balance of workers/ps. (otherwise youd run into blocking or network saturation issues.) horovod addresses this issue with their ring allreduce implementation. having said all of that, im sticking with distributed tf for now.