8 ms·

I've used tensorflow in 2016-2018 and switched to Pytorch a few months ago. My theory is that TF is used mainly because it is supported by Google, even if it is

I've used tensorflow in 2016-2018 and switched to Pytorch a few months ago. My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. In fact, TF was not meant to be public but was at first some internal tool at Google. This would explain why people at Google itself started to develop Keras or Sonnet (deepmind) at some point. For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.

me2too 8y ago
But why? I do use Tensorflow form the 0.x release and I'm still using it right now, migrating all the codebase to the 2.x version. I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier. Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow. Both frameworks are converging to something really similar. I don't see a reason to switch (right now)
- CretinDesAlpes 8y ago
  At least 1) debugging in a numpy-like env and 2) multi GPU training on a single machine with multiple GPU (last time I was using TF it was such a nightmare that people had to use horovod, a wrapper library developed by... Uber).
  - me2too 8y ago
    Point 1 is a good point, although tfdebug exists and it works. For point 2: I don't see the problem. I use almost daily a wrapper (defined inside tf.contrib, that in version 2.x will go in core [I hope]) around the optimizer that in 2 lines allows me to distribute the training on multiple GPUs on the same machine
    - lostmsu 8y ago
      Which wrapper are you talking about?
      - me2too 8y ago
        https://www.tensorflow.org/api_docs/python/tf/contrib/distribute/MirroredStrategy https://www.tensorflow.org/api_docs/python/tf/contrib/distri...
  - jamesblonde 8y ago
    For a comparison of Horovod and TensorFlow's new CollectiveAllReduce, read here: https://www.logicalclocks.com/goodbye-horovod-hello-tensorflow-collectiveallreduce/ https://www.logicalclocks.com/goodbye-horovod-hello-tensorfl...
- samstave 8y ago
  As someone on the sidelines with no explanation in either, but interest in where all this is going, can you share what exactly you do with TF or pytorch? What actual product(ive) things?
  - CardenB 8y ago
    Machine learning pipelines are brittle and hard to debug as the data representation is condensed to numbers in matrices instead of expressive data structures with semantically meaningful variable names. To mitigate this complexity, PyTorch and Tensorflow are heavy weight machine learning frameworks that give your software a lot of structure as well as tooling to monitor the progress of training and debug your models as well as some deployment tooling. Any neural net based component of software will likely be developed in one of these frameworks.
  - me2too 8y ago
    Computer Vision tasks: I train models to to object detection, image classification, semantic segmentation... export the model, load the trained model in C++ and run inference. Or generative models: same workflow (python definition and train. Export and use in other languages). I use Tensorflow basically everyday. I use pytorch only when I read a model implemented in this framework to reimplement it in Tensorflow (in order to use all the tools I developed to simplify the train to shipping to production phases)
  - thfuran 8y ago
    Train neural networks for medical imaging tasks like organ segmentation or lesion detection.
  - noelsusman 8y ago
    There are lots of applications, theoretically, but the primary use cases are image classification and natural language processing. It's not worth it for other problem types. I'm constantly surprised by how much attention it gets given how narrow the scope is. I guess a lot of people need to classify images.
kajecounterhack 8y ago
Keras wasn't developed by Google, it was adopted by TF. > My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google). It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things. Well-designed primitives / abstractions are important; Tensorflow does this well.
- deleted 8y ago
  [deleted]
- buboard 8y ago
  for something to be "industrial" it has to be at least stable and well documented. tf doesnt feel like that.
  - kajecounterhack 8y ago
    Care to point out some examples? (I use tf every day and it feels fine to me but am sure the tf team is curious where others see areas for improvement).
- qnsi 8y ago
  I thought Keras was developed by Francis Chollet, who works at Google? Not sure if he develops and maintains during company time, but wouldnt be surprised
  - dekhn 8y ago
    He was hired after he made keras
    - qnsi 8y ago
      Thanks! Sorry for misleading comment above, then.
adamnemecek 8y ago
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. Sounds like every Google developed technology. Angular, Go, Android, the list is infinite.
solomatov 8y ago
Keras was an independent product Sonnet is based on TF
sseveran 8y ago
I am probably an outlier (I have experience with both compiler development and lazy languages) but I found the graph model to be very natural. Much of the struggles I saw people having conceptually I had already had years before. The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep. I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed. I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been. All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.
- lostmsu 8y ago
  What would you prefer to see instead of Python? My company is working on a full proprietary .NET binding for TensorFlow, and we are considering making Java version too, if there is a market for it.
  - sseveran 8y ago
    Honestly Haskell. But on .net F# could be interesting. I don't think these bindings are useful to me in isolation. The reason I have not really gone down that path is that one would need to clone all the higher level TF code that has been written in python.
cs702 8y ago
I had a similar experience at work. We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow. Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision. [a] https://pytorch.org/2018/05/02/road-to-1.0.html https://pytorch.org/2018/05/02/road-to-1.0.html
qwerty456127 8y ago
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.

me2too 8y ago

But why? I do use Tensorflow form the 0.x release and I'm still using it right now, migrating all the codebase to the 2.x version. I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier. Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow. Both frameworks are converging to something really similar. I don't see a reason to switch (right now)

CretinDesAlpes 8y ago
At least 1) debugging in a numpy-like env and 2) multi GPU training on a single machine with multiple GPU (last time I was using TF it was such a nightmare that people had to use horovod, a wrapper library developed by... Uber).
- me2too 8y ago
  Point 1 is a good point, although tfdebug exists and it works. For point 2: I don't see the problem. I use almost daily a wrapper (defined inside tf.contrib, that in version 2.x will go in core [I hope]) around the optimizer that in 2 lines allows me to distribute the training on multiple GPUs on the same machine
  - lostmsu 8y ago
    Which wrapper are you talking about?
    - me2too 8y ago
      https://www.tensorflow.org/api_docs/python/tf/contrib/distribute/MirroredStrategy https://www.tensorflow.org/api_docs/python/tf/contrib/distri...
- jamesblonde 8y ago
  For a comparison of Horovod and TensorFlow's new CollectiveAllReduce, read here: https://www.logicalclocks.com/goodbye-horovod-hello-tensorflow-collectiveallreduce/ https://www.logicalclocks.com/goodbye-horovod-hello-tensorfl...
samstave 8y ago
As someone on the sidelines with no explanation in either, but interest in where all this is going, can you share what exactly you do with TF or pytorch? What actual product(ive) things?
- CardenB 8y ago
  Machine learning pipelines are brittle and hard to debug as the data representation is condensed to numbers in matrices instead of expressive data structures with semantically meaningful variable names. To mitigate this complexity, PyTorch and Tensorflow are heavy weight machine learning frameworks that give your software a lot of structure as well as tooling to monitor the progress of training and debug your models as well as some deployment tooling. Any neural net based component of software will likely be developed in one of these frameworks.
- me2too 8y ago
  Computer Vision tasks: I train models to to object detection, image classification, semantic segmentation... export the model, load the trained model in C++ and run inference. Or generative models: same workflow (python definition and train. Export and use in other languages). I use Tensorflow basically everyday. I use pytorch only when I read a model implemented in this framework to reimplement it in Tensorflow (in order to use all the tools I developed to simplify the train to shipping to production phases)
- thfuran 8y ago
  Train neural networks for medical imaging tasks like organ segmentation or lesion detection.
- noelsusman 8y ago
  There are lots of applications, theoretically, but the primary use cases are image classification and natural language processing. It's not worth it for other problem types. I'm constantly surprised by how much attention it gets given how narrow the scope is. I guess a lot of people need to classify images.

kajecounterhack 8y ago

Keras wasn't developed by Google, it was adopted by TF. > My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google). It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things. Well-designed primitives / abstractions are important; Tensorflow does this well.

deleted 8y ago
[deleted]
buboard 8y ago
for something to be "industrial" it has to be at least stable and well documented. tf doesnt feel like that.
- kajecounterhack 8y ago
  Care to point out some examples? (I use tf every day and it feels fine to me but am sure the tf team is curious where others see areas for improvement).
qnsi 8y ago
I thought Keras was developed by Francis Chollet, who works at Google? Not sure if he develops and maintains during company time, but wouldnt be surprised
- dekhn 8y ago
  He was hired after he made keras
  - qnsi 8y ago
    Thanks! Sorry for misleading comment above, then.

adamnemecek 8y ago

> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. Sounds like every Google developed technology. Angular, Go, Android, the list is infinite.

solomatov 8y ago

Keras was an independent product Sonnet is based on TF

sseveran 8y ago

I am probably an outlier (I have experience with both compiler development and lazy languages) but I found the graph model to be very natural. Much of the struggles I saw people having conceptually I had already had years before. The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep. I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed. I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been. All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.

lostmsu 8y ago
What would you prefer to see instead of Python? My company is working on a full proprietary .NET binding for TensorFlow, and we are considering making Java version too, if there is a market for it.
- sseveran 8y ago
  Honestly Haskell. But on .net F# could be interesting. I don't think these bindings are useful to me in isolation. The reason I have not really gone down that path is that one would need to clone all the higher level TF code that has been written in python.

cs702 8y ago

I had a similar experience at work. We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow. Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision. [a] https://pytorch.org/2018/05/02/road-to-1.0.html https://pytorch.org/2018/05/02/road-to-1.0.html

qwerty456127 8y ago

> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners. I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.