5 ms·

At work, we switched over from TensorFlow to PyTorch when 1.0 was released, both for R&D and production... and our productivity and happiness with PyTorch notic

by cs702 7y ago

At work, we switched over from TensorFlow to PyTorch when 1.0 was released, both for R&D and production... and our productivity and happiness with PyTorch noticeably, significantly improved. Back when we were using TensorFlow, whenever we wanted to try something new that wasn't already provided out-of-the-box by existing APIs, sooner or later we would find ourselves wrestling with its machinery, especially for models with more complex control flow. TensorFlow feels like it was built from the ground up to scale up to billions of users and all kinds of devices, with developer productivity and happiness a secondary priority. PyTorch feels like it was built the other way around, prioritizing developer productivity and happiness; other considerations were secondary. That said, we are keeping an eye on Swift + MLIR + TensorFlow. We think it could unseat PyTorch for R&D and eventually, production, due to (a) the promise of automatic creation of high-performance GPU/TPU kernels without hassle, (b) Swift's easy learning curve, and (c) Swift's fast performance and type safety. Jeremy Howard has a good post about this: https://www.fast.ai/2019/03/06/fastai-swift/ https://www.fast.ai/2019/03/06/fastai-swift/

RcouF1uZ4gsC 7y ago
If you are looking for more performance and type safety versus python, PyTorch does have a C++ frontend. https://pytorch.org/cppdocs/frontend.html#end-to-end-example https://pytorch.org/cppdocs/frontend.html#end-to-end-example
jorlow 7y ago
> TensorFlow feels like it was built from the ground up to scale up to billions of users and all kinds of devices, with developer productivity and happiness a secondary priority. PyTorch feels like it was built the other way around, prioritizing developer productivity and happiness; other considerations were secondary. I recently moved from Google to Facebook and this is how I'd characterize most of the differences I see: Facebook optimizes for your ability to make progress above everything else. Google, not so much.
- kabes 7y ago
  Seems to hold up for most google stuff I tried. E.g. kubernetes and angular.
option 7y ago
If you want type safety on top of Pytorch, checkout this project https://github.com/NVIDIA/NeMo https://github.com/NVIDIA/NeMo
- MiroF 7y ago
  It'd be nice to have the same pytorch abstractions but in a more statically typed language than pytorch.
  - jhj 7y ago
    there's a C++14 frontend https://pytorch.org/cppdocs/frontend.html#end-to-end-example https://pytorch.org/cppdocs/frontend.html#end-to-end-example though the typing doesn't dive into tensor dimensions etc.
  - xvilka 7y ago
    You can use PyTorch Rust bindings [1] or OCaml bindings [2]. [1] https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs [2] https://github.com/LaurentMazare/ocaml-torch https://github.com/LaurentMazare/ocaml-torch
amznthrowaway5 7y ago
Chris Lattner left Google a few days ago. Swift for TensorFlow is likely dead.
- physicsyogi 7y ago
  I'd be surprised if it's dead. At least two of the people on the Swift for Tensorflow team were building a deep learning library in Swift before Swift for Tensorflow was a thing. Much of their work is a part of Swift for Tensorflow.
  - melling 7y ago
    What about autodiff? Is that a separate effort?
  - amznthrowaway5 7y ago
    That's good to hear, but S4TF has failed to gain any real traction among ML users and was started over two years ago. I've heard not many care about it apart from the core team and now that Lattner is gone it's likely Google will send it to the graveyard soon.
    - byt143 7y ago
      https://twitter.com/clattner_llvm/status/1222032740897284097 https://twitter.com/clattner_llvm/status/1222032740897284097 Jeff Dean seems to disagree. Also, I think it hasn't picked up steam because it just isn't mature enough yet
      - nl 7y ago
        That's a very non-committed response from Jeff Dean! I suspect it will drift along like the Java TensorFlow wrapper. Useful in some specific circumstances, but unused outside that.
        byt143 7y ago
jostmey 7y ago
The problem is not that Tensorflow is based around python instead of swift. The Tensorflow API is a mess. The programming language is not the problem, and switching to swift will only add to the API mess.
mark_l_watson 7y ago
Thanks for your comments. I have been using TF just about since the first public version and it is what I am used to. I have been thinking of switching to PyTorch, and I think I will take the plunge and tackle the PyTorch learning curve.
- cs702 7y ago
  Having transitioned from TF to PyTorch for both research and production, I can tell you the learning curve is pretty mild. At first, PyTorch will feel like "Numpy on accelerated hardware with built-in backprop and DL/ML facilities." And like Numpy, it will quickly become second nature and get out of your way. Good luck with the transition!
bonoboTP 7y ago
I first learned TensorFlow and built a lot of very custom stuff with its "Slim" API and have been very happy about it. The only thing I always struggle with is the grunt work of wrangling tensor shapes and indices, e.g. broadcasting, tf.tile, tf.gather, tf.scatter_nd, tf.where etc. These are a real headache to use, while numpy for example has its so called "fancy indexing" and mask-based indexing, which are not available in TF. Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. Maybe someone who just uses PyTorch doesn't miss this because they never used it. My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That's not the best design in terms of DRY. I understand why it's technically needed due to the working principles of PyTorch (especially for weight sharing in different parts of the model, which in TF may need some getting used with its variable scopes and reuse flags), it's just inconvenient.
- cs702 7y ago
  > Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. That's one of the things we realized we disliked most about TensorFlow: Whenever we tried to do anything that was out-of-the-box, we would have to write Python code that would then construct the TensorFlow graph that would actually specify and eventually run the computations. It felt like metaprogramming from a nice, general-purpose language (Python) in an inflexible, non-general purpose one (TF ops). > My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That is not true. You can create graphs, compute losses, and backpropagate them on the fly, because autograd records all operations on the fly.[a] The nn.Module class is a convenient, nice, Pythonic wrapper with lots of functionality, but you don't have to use it. You can create your own wrapper if you'd like. [a] https://pytorch.org/docs/stable/notes/autograd.html https://pytorch.org/docs/stable/notes/autograd.html
dchichkov 7y ago
I'm kinda looking forward to see Bazel gone as well, in a few years. The Tensorflow of build systems ;). It also seems there is something strange going on with open source from Google. Maybe it is as simple as a mismatch of impedance between their monorepo and the outside world, which they don't maintain. Or maybe it is something else.
317070 7y ago
Swift for tensorflow is probably dead in the water though, now that Chris Lattner left Google this week: https://www.businesswire.com/news/home/20200127005141/en/Google-Tesla-Engineer-Chris-Lattner-Lead-SiFive https://www.businesswire.com/news/home/20200127005141/en/Goo...

by cs702 7y ago

RcouF1uZ4gsC 7y ago

If you are looking for more performance and type safety versus python, PyTorch does have a C++ frontend. https://pytorch.org/cppdocs/frontend.html#end-to-end-example https://pytorch.org/cppdocs/frontend.html#end-to-end-example

jorlow 7y ago

> TensorFlow feels like it was built from the ground up to scale up to billions of users and all kinds of devices, with developer productivity and happiness a secondary priority. PyTorch feels like it was built the other way around, prioritizing developer productivity and happiness; other considerations were secondary. I recently moved from Google to Facebook and this is how I'd characterize most of the differences I see: Facebook optimizes for your ability to make progress above everything else. Google, not so much.

kabes 7y ago
Seems to hold up for most google stuff I tried. E.g. kubernetes and angular.

option 7y ago

If you want type safety on top of Pytorch, checkout this project https://github.com/NVIDIA/NeMo https://github.com/NVIDIA/NeMo

MiroF 7y ago
It'd be nice to have the same pytorch abstractions but in a more statically typed language than pytorch.
- jhj 7y ago
  there's a C++14 frontend https://pytorch.org/cppdocs/frontend.html#end-to-end-example https://pytorch.org/cppdocs/frontend.html#end-to-end-example though the typing doesn't dive into tensor dimensions etc.
- xvilka 7y ago
  You can use PyTorch Rust bindings [1] or OCaml bindings [2]. [1] https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs [2] https://github.com/LaurentMazare/ocaml-torch https://github.com/LaurentMazare/ocaml-torch

amznthrowaway5 7y ago

Chris Lattner left Google a few days ago. Swift for TensorFlow is likely dead.

physicsyogi 7y ago
I'd be surprised if it's dead. At least two of the people on the Swift for Tensorflow team were building a deep learning library in Swift before Swift for Tensorflow was a thing. Much of their work is a part of Swift for Tensorflow.
- melling 7y ago
  What about autodiff? Is that a separate effort?
- amznthrowaway5 7y ago
  That's good to hear, but S4TF has failed to gain any real traction among ML users and was started over two years ago. I've heard not many care about it apart from the core team and now that Lattner is gone it's likely Google will send it to the graveyard soon.
  - byt143 7y ago
    https://twitter.com/clattner_llvm/status/1222032740897284097 https://twitter.com/clattner_llvm/status/1222032740897284097 Jeff Dean seems to disagree. Also, I think it hasn't picked up steam because it just isn't mature enough yet
    - nl 7y ago
      That's a very non-committed response from Jeff Dean! I suspect it will drift along like the Java TensorFlow wrapper. Useful in some specific circumstances, but unused outside that.
      - byt143 7y ago

jostmey 7y ago

The problem is not that Tensorflow is based around python instead of swift. The Tensorflow API is a mess. The programming language is not the problem, and switching to swift will only add to the API mess.

mark_l_watson 7y ago

Thanks for your comments. I have been using TF just about since the first public version and it is what I am used to. I have been thinking of switching to PyTorch, and I think I will take the plunge and tackle the PyTorch learning curve.

cs702 7y ago
Having transitioned from TF to PyTorch for both research and production, I can tell you the learning curve is pretty mild. At first, PyTorch will feel like "Numpy on accelerated hardware with built-in backprop and DL/ML facilities." And like Numpy, it will quickly become second nature and get out of your way. Good luck with the transition!

bonoboTP 7y ago

I first learned TensorFlow and built a lot of very custom stuff with its "Slim" API and have been very happy about it. The only thing I always struggle with is the grunt work of wrangling tensor shapes and indices, e.g. broadcasting, tf.tile, tf.gather, tf.scatter_nd, tf.where etc. These are a real headache to use, while numpy for example has its so called "fancy indexing" and mask-based indexing, which are not available in TF. Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. Maybe someone who just uses PyTorch doesn't miss this because they never used it. My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That's not the best design in terms of DRY. I understand why it's technically needed due to the working principles of PyTorch (especially for weight sharing in different parts of the model, which in TF may need some getting used with its variable scopes and reuse flags), it's just inconvenient.

cs702 7y ago
> Other than that, I'm very happy about TF's graph structure. It means, I can just build the graph (a bit like a declarative style) and in each iteration (session run) I can specify which tensors I'd like to fetch. Then TF takes into account the data dependencies and lazily computes only what is needed. Very convenient and I got very used to this. That's one of the things we realized we disliked most about TensorFlow: Whenever we tried to do anything that was out-of-the-box, we would have to write Python code that would then construct the TensorFlow graph that would actually specify and eventually run the computations. It felt like metaprogramming from a nice, general-purpose language (Python) in an inflexible, non-general purpose one (TF ops). > My other problem with pytorch is that you have to define every model / layer twice: once in the constructor and once in the forward() function. That is not true. You can create graphs, compute losses, and backpropagate them on the fly, because autograd records all operations on the fly.[a] The nn.Module class is a convenient, nice, Pythonic wrapper with lots of functionality, but you don't have to use it. You can create your own wrapper if you'd like. [a] https://pytorch.org/docs/stable/notes/autograd.html https://pytorch.org/docs/stable/notes/autograd.html

dchichkov 7y ago

I'm kinda looking forward to see Bazel gone as well, in a few years. The Tensorflow of build systems ;). It also seems there is something strange going on with open source from Google. Maybe it is as simple as a mismatch of impedance between their monorepo and the outside world, which they don't maintain. Or maybe it is something else.

317070 7y ago

Swift for tensorflow is probably dead in the water though, now that Chris Lattner left Google this week: https://www.businesswire.com/news/home/20200127005141/en/Google-Tesla-Engineer-Chris-Lattner-Lead-SiFive https://www.businesswire.com/news/home/20200127005141/en/Goo...