5 ms·

PyTorch is becoming much more popular among researchers, even if TensorFlow use is widespread among data scientists. Would it be bad if TensorFlow became the d

PyTorch is becoming much more popular among researchers, even if TensorFlow use is widespread among data scientists. Would it be bad if TensorFlow became the default library for deep learning? For companies that aren't Google, yes. Many organizations don't like how Google's controlled Android development. But I think an important part of Dean's lecture are the slides that talk about making ML/DL expertise obsolete. That's the future, whether you use TensorFlow or not.

deepnotderp 9y ago
It's cute, but isn't going to completely obsolete ML/DL expertise. (And believe me, as a prospective supplier of DL compute, I would love for the answer to be "more compute"). To give you an idea of how much compute it took, they spent two million dollars for one run on a relatively toy dataset, CIFAR-10. Imagine how much it would take on an imagenet sized dataset! Can your company afford ~$20 million+ per dataset? I do think that hyperparameter twiddling aspect might get automated, but believe me, that is much welcomed by the DL research community! I would much rather spend my time on new ideas rather than trying ten different initializations :)
- blueyes 9y ago
  Sure, but that's $2m of today's compute on today's chips. The constraints on chips and cost are moving, and Google is pushing them.
  - deepnotderp 9y ago
    Sure, so it becomes ~$2 million for an imagenet scale dataset with TPUs and ~$200K with our chips. Still pretty expensive :)
londons_explore 9y ago
Android is a very closed opensource project. For example, if you write a pull request to add a feature as an outsider, the chances of it getting into the main project is very low. Chromium is very open. In many cases, Google has to ask permission from outside contributors to commit code, since the outside contributors are the "boss" of part of the codebase. Tensorflow is closer to chromiums model. There are a lot of external contributors.
svara 9y ago
> PyTorch is becoming much more popular among researchers, even if TensorFlow use is widespread among data scientists. Could you maybe comment on why this is the case? Are there technical reasons for that?
- agibsonccc 9y ago
  A big reason is the dynamic computation graphs. Tensorflow fold is supposed to be the response to that, but overall people are finding pytorch to be more flexible. (Warning: I'm biased. I compete VERY heavily with anything google including cloud and their deep learning framework.)

deepnotderp 9y ago

It's cute, but isn't going to completely obsolete ML/DL expertise. (And believe me, as a prospective supplier of DL compute, I would love for the answer to be "more compute"). To give you an idea of how much compute it took, they spent two million dollars for one run on a relatively toy dataset, CIFAR-10. Imagine how much it would take on an imagenet sized dataset! Can your company afford ~$20 million+ per dataset? I do think that hyperparameter twiddling aspect might get automated, but believe me, that is much welcomed by the DL research community! I would much rather spend my time on new ideas rather than trying ten different initializations :)

blueyes 9y ago
Sure, but that's $2m of today's compute on today's chips. The constraints on chips and cost are moving, and Google is pushing them.
- deepnotderp 9y ago
  Sure, so it becomes ~$2 million for an imagenet scale dataset with TPUs and ~$200K with our chips. Still pretty expensive :)

londons_explore 9y ago

Android is a very closed opensource project. For example, if you write a pull request to add a feature as an outsider, the chances of it getting into the main project is very low. Chromium is very open. In many cases, Google has to ask permission from outside contributors to commit code, since the outside contributors are the "boss" of part of the codebase. Tensorflow is closer to chromiums model. There are a lot of external contributors.

svara 9y ago

> PyTorch is becoming much more popular among researchers, even if TensorFlow use is widespread among data scientists. Could you maybe comment on why this is the case? Are there technical reasons for that?

agibsonccc 9y ago
A big reason is the dynamic computation graphs. Tensorflow fold is supposed to be the response to that, but overall people are finding pytorch to be more flexible. (Warning: I'm biased. I compete VERY heavily with anything google including cloud and their deep learning framework.)