2 ms·
Yes, exaflop, I'm so used to writing "FLOPs" that I wrote it xD You may have a background in parallelizing simulations, but I beg to differ about parallelizing
by deepnotderp 10y ago
Yes, exaflop, I'm so used to writing "FLOPs" that I wrote it xD
You may have a background in parallelizing simulations, but I beg to differ about parallelizing deep learning training. There is a lot of communication involved. For one, your model is very likely to be enormous (perhaps even necessitating model parallelism) and second, there is a LOT of communication involved even with data parallelism.
- dekhn 10y agoI guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-latency systems. They are highly tolerant of async updates. Most parallelism in DL uses coarse exchange, although I agree wrt to large models. That's similarly true for every supercomputer application (it's the only reason people use supercomputers now; it's much easier and faster if your system fits on a single system).