4 ms·
You mean "20 ExaFLOP". "20 ExaFLOPs" is a per-second measure. FWIW, I work at Google, have a background in parallelizing simulations, and built an execution s
by dekhn 10y ago
You mean "20 ExaFLOP". "20 ExaFLOPs" is a per-second measure.
FWIW, I work at Google, have a background in parallelizing simulations, and built an execution system that runs huge hyperparameter combinations. We called it "exacycle" because it exceeded 1 exaflop / second (no communication between tasks) using only idle cycles.
I think you probably missed my point: it's now pretty well established that for any physical simulation process, you can train a net using far less energy and get an equivalent quality result. The training itself doesn't require lots of communication unless the model is enormous.
- deepnotderp 10y agoYes, exaflop, I'm so used to writing "FLOPs" that I wrote it xD You may have a background in parallelizing simulations, but I beg to differ about parallelizing deep learning training. There is a lot of communication involved. For one, your model is very likely to be enormous (perhaps even necessitating model parallelism) and second, there is a LOT of communication involved even with data parallelism.
- dekhn 10y agoI guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-latency systems. They are highly tolerant of async updates. Most parallelism in DL uses coarse exchange, although I agree wrt to large models. That's similarly true for every supercomputer application (it's the only reason people use supercomputers now; it's much easier and faster if your system fits on a single system).