3 ms·
That's a good question. There are a small number of people who have experience with it. The number is small because supercomputers don't really provide any sp
by dekhn 10y ago
That's a good question. There are a small number of people who have experience with it. The number is small because supercomputers don't really provide any speedup advantages, although this is mainly because the code developers of distributed DNNs have been using TCP/IP networks with standard RPC instead of fast interconnect.
I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI.
There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.