3 ms·

Apparently perfect simulation will require zettaflops! We can't even achieve exascale right now! (But we'll get there)

dekhn 10y ago
There is no such thing as perfect simulation. And, it's more likely that when people realize how incredibly hard it is to build exascale machines, and how unproductive they will be, they'll just end up training deep learning models which approximate the perfect simulations well enough and cost-effectively enough on commodity GPUs that nobody will buy or build supercomputers any more.
- deepnotderp 10y ago
  Baidu estimates it takes ~20 ExaFLOPs to train a deep net. Consider that it's not uncommon to churn through ~100 hyperparamter combinations (let alone architectures), especially for something high end as this, and you realize that deep learning is no escape from the compute problem! There's a reason why the big DL players hire lots of HPC guys for deep learning. And also, parallelizing SGD is a highly non-trivial task, until very recently, you had to linearly lower the learning rate as you added new workers, and keep in mind that bandwidth is far more expensive (both economically, temporally and energy-wise) than compute. That being said, many people are already exploring deep learning for weather simulation (such as Yandex I believe) and it has worked very well, and is near, or beating SOTA iirc, so I definitely think there's a future there. For the record, I think exacsale is achievable, but not with the current architecture (admittedly I'm biased, as a deep learning chip and supercomputing chip startup founder), but I think my objective evidence is pretty strong that with a new architecture, exascale is possible. On the other hand, zettascale may be the end of supercomputer scaling. Moore's law, Dennard Scaling, et al are no saviors either, since communication and scheduling logic now dominates the cost of computation.
  - dekhn 10y ago
    You mean "20 ExaFLOP". "20 ExaFLOPs" is a per-second measure. FWIW, I work at Google, have a background in parallelizing simulations, and built an execution system that runs huge hyperparameter combinations. We called it "exacycle" because it exceeded 1 exaflop / second (no communication between tasks) using only idle cycles. I think you probably missed my point: it's now pretty well established that for any physical simulation process, you can train a net using far less energy and get an equivalent quality result. The training itself doesn't require lots of communication unless the model is enormous.
    - deepnotderp 10y ago
      Yes, exaflop, I'm so used to writing "FLOPs" that I wrote it xD You may have a background in parallelizing simulations, but I beg to differ about parallelizing deep learning training. There is a lot of communication involved. For one, your model is very likely to be enormous (perhaps even necessitating model parallelism) and second, there is a LOT of communication involved even with data parallelism.
      - dekhn 10y ago
        I guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-latency systems. They are highly tolerant of async updates. Most parallelism in DL uses coarse exchange, although I agree wrt to large models. That's similarly true for every supercomputer application (it's the only reason people use supercomputers now; it's much easier and faster if your system fits on a single system).

dekhn 10y ago

There is no such thing as perfect simulation. And, it's more likely that when people realize how incredibly hard it is to build exascale machines, and how unproductive they will be, they'll just end up training deep learning models which approximate the perfect simulations well enough and cost-effectively enough on commodity GPUs that nobody will buy or build supercomputers any more.

deepnotderp 10y ago
Baidu estimates it takes ~20 ExaFLOPs to train a deep net. Consider that it's not uncommon to churn through ~100 hyperparamter combinations (let alone architectures), especially for something high end as this, and you realize that deep learning is no escape from the compute problem! There's a reason why the big DL players hire lots of HPC guys for deep learning. And also, parallelizing SGD is a highly non-trivial task, until very recently, you had to linearly lower the learning rate as you added new workers, and keep in mind that bandwidth is far more expensive (both economically, temporally and energy-wise) than compute. That being said, many people are already exploring deep learning for weather simulation (such as Yandex I believe) and it has worked very well, and is near, or beating SOTA iirc, so I definitely think there's a future there. For the record, I think exacsale is achievable, but not with the current architecture (admittedly I'm biased, as a deep learning chip and supercomputing chip startup founder), but I think my objective evidence is pretty strong that with a new architecture, exascale is possible. On the other hand, zettascale may be the end of supercomputer scaling. Moore's law, Dennard Scaling, et al are no saviors either, since communication and scheduling logic now dominates the cost of computation.
- dekhn 10y ago
  You mean "20 ExaFLOP". "20 ExaFLOPs" is a per-second measure. FWIW, I work at Google, have a background in parallelizing simulations, and built an execution system that runs huge hyperparameter combinations. We called it "exacycle" because it exceeded 1 exaflop / second (no communication between tasks) using only idle cycles. I think you probably missed my point: it's now pretty well established that for any physical simulation process, you can train a net using far less energy and get an equivalent quality result. The training itself doesn't require lots of communication unless the model is enormous.
  - deepnotderp 10y ago
    Yes, exaflop, I'm so used to writing "FLOPs" that I wrote it xD You may have a background in parallelizing simulations, but I beg to differ about parallelizing deep learning training. There is a lot of communication involved. For one, your model is very likely to be enormous (perhaps even necessitating model parallelism) and second, there is a LOT of communication involved even with data parallelism.
    - dekhn 10y ago
      I guarantee that communication is small compared to what the weather simulators require. Deep learning systems typically work fine on large-bandwidth, medium-latency systems. They are highly tolerant of async updates. Most parallelism in DL uses coarse exchange, although I agree wrt to large models. That's similarly true for every supercomputer application (it's the only reason people use supercomputers now; it's much easier and faster if your system fits on a single system).