3 ms·
I guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-l
by dekhn 10y ago
I guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-latency systems. They are highly tolerant of async updates.
Most parallelism in DL uses coarse exchange, although I agree wrt to large models. That's similarly true for every supercomputer application (it's the only reason people use supercomputers now; it's much easier and faster if your system fits on a single system).