5 ms·
But _why_ use TF when you have PyTorch which is just as powerful, runs noticeably faster for most workloads, _and_ is easy to understand? What are you gaining b
by 0xbear 9y ago
But _why_ use TF when you have PyTorch which is just as powerful, runs noticeably faster for most workloads, _and_ is easy to understand? What are you gaining by using TF these days?
- programnature 9y agoIf you want to run in production. Even FB doesn't use PyTorch in production, and instead uses Caffe2.
- Q6T46nT668w6i3m 9y agoPyTorch was certainly used in production at Facebook (and elsewhere). It might’ve changed in the past few months as Caffe 2 matured, but there were a few internal applications that used PyTorch.
- programnature 9y agoFwiw this is what Soumith has said: "Internally at Facebook, we have a unified strategy. We say PyTorch is used for all of research and Caffe 2 is used for all of production." https://www.oreilly.com/ideas/why-ai-and-machine-learning-researchers-are-beginning-to-embrace-pytorch https://www.oreilly.com/ideas/why-ai-and-machine-learning-re... Its not exactly a secret that PyTorch's tradeoffs favor research and not production.
- Q6T46nT668w6i3m 9y agoTotally. And I think it’s a solid strategy. However, there’s certainly an interest (internally and externally) in providing a better inter-operability story between the two. I imagine something like using Keras for model creation (and possibly training) and running (either on mobile or the cloud) on some Caffe 2 deployment.
- programnature 9y agoIts a good strategy. But its no silver bullet either. If you're exporting to a "static graph" platform, your losing a major benefit of PyTorch. If you mostly just care about shipping to production, a case can be made to just use tf/caffe2/mxnet etc from the start. While PyTorch is extremely cool, the fanboyism is out of hand, thinking that what's good for their corner of the universe must be awesome for every use case and therefore TF is a overcomplex turd. Its not like the people designing these systems are stupid.
- Q6T46nT668w6i3m 9y agoI agree. PyTorch’s dynamism is fantastic. However, I have no idea how you’d manage to recompile PyTorch code to Caffe 2 in a satisfying way. If something is released, I suspect it’d be limited to a subset of PyTorch features (I’d also bet that subset doesn’t include the features that make PyTorch compelling).
- kuschku 9y agoIs PyTorch actually faster than TensorFlow with the C++ static compilation?
- 0xbear 9y agoI've tried XLA and observed no measurable benefit for most models, small benefit for some, and worse performance for others. That was a few months ago, so maybe things have changed since then. It won't be a huge difference no matter what they do, because the vast majority of time is spent in CUDNN and gemm/gemv anyway
- jjawssd 9y agoTensorflow is more documented and talked about right now which is why more people will be interested in it
- 0xbear 9y agoI'm not actually sure it's "more documented". It has more documentation, that's true, but that's not the same thing as better documentation, and after some point the mountain of documentation becomes a daunting obstacle. TF IMO is far beyond that point.
- SimbaOnSteroids 9y agoI find the documentation daunting but I had assumed that was because I didn't really understand how AI works and that as my understanding grew the docs would start to make more sense.
- 0xbear 9y agoTake a look at PyTorch. Much easier to understand, fewer bugs, much less obscure all around. I've spent years working with TF and have come to dislike its inflexible, verbose, and convoluted way of doing things. I'm 80% on PyTorch now.
- Q6T46nT668w6i3m 9y agoI like and use both PyTorch and TensorFlow. You’re entirely correct that TensorFlow is far more verbose than PyTorch, but Keras remedies this (for many problems). I’d also add that PyTorch has a narrower scope than PyTorch and because of this introduces problems when scaling or distributing (PyTorch’s flexible broadcasting isn’t without trade-offs). Maybe there’ll be a convergence of Caffe 2 and PyTorch to remedy that problem.
- dr1337 9y agoFor me, Keras is the least verbose of Tensorflow and PyTorch, however I find it too sparse and there's too much "magic" happening under the nicely done API. I have found PyTorch to be the happy half-way point between Tensorflow and Keras in terms of verbosity.
- jamesmishra 9y agoI think at the moment, TensorFlow has a better tooling and deployment story with tools like TensorBoard[1], Tensor Processing Units in Google Cloud[2], distributed training[3], and deploying to mobile apps[4]. As time passes, PyTorch will probably get better at all of these things, but the core team and surrounding community need to make it a priority. Meanwhile, Google's investment in TensorFlow will continue, and there's no sign that it will be a bad bet for large-scale deployments... even if PyTorch becomes the favored system for prototyping. [1]: https://www.tensorflow.org/get_started/summaries_and_tensorboard https://www.tensorflow.org/get_started/summaries_and_tensorb... [2]: https://www.servethehome.com/google-cloud-tpu-details-revealed/ https://www.servethehome.com/google-cloud-tpu-details-reveal... [3]: https://www.tensorflow.org/deploy/distributed https://www.tensorflow.org/deploy/distributed [4]: https://www.tensorflow.org/mobile/ https://www.tensorflow.org/mobile/
- 0xbear 9y agoTPU will be everywhere next year in the form of NVIDIA Tesla V100. And FB is clearly getting much better ROI. It's not even close.
- deleted 9y ago[deleted]
- mamp 9y agoIt's nice to have the Keras front-end for TF - less boilerplate etc. I don't think there would be a reason why Keras couldn't sit on PyTorch but I can't see any work on this. Keras is very productive unless you are doing NN research. Also you can also run it using CNTK if you want speed up RNNs which are slower on TF compared with CNTK.
- Q6T46nT668w6i3m 9y agoI use Keras for machine learning research and find that it works extremely well. The one advantage of PyTorch is more flexible broadcasting, but that introduces other problems. A major advantage of Keras is that if you’re using a standard training scheme (e.g. training a convolutional neural network for image classification), your research will be entirely focused on either the underlying architecture (that’s easily summarized and serialized), your custom layers (my favorite abstraction for convolutional neural networks), and losses. Keras’ abstractions eliminate stuff like IO that I find extremely distracting.
- make3 9y agoI really don't think the speed thing is as clearcut as you let it out to be
- make3 9y agoTensorflow was the first modern tool supported by a large ai company, so probably mostly historical reasons. Also, Google is likely still the company with the largest AI departments, and the company that gives them the most visibility.
- jorgemf 9y agoI think the are 2 main reasons: TF is backed by Google and you can use for research and production (server & mobile) with minor changes. You can also train your TF models in the cloud without having to setup anything (as in Google Cloud[1] or TensorPort[2]) [1] https://cloud.google.com/ml-engine/ https://cloud.google.com/ml-engine/ [2] https://tensorport.com/ https://tensorport.com/
- Q6T46nT668w6i3m 9y agoWhile I’ve observed PyTorch running faster for my (convolutional-based) research, it’s within a few milliseconds of TensorFlow and Keras. If this type of difference mattered (it might for some uses), I would imagine you’d use CuDNN directly. I guess that’s the point, the libraries are all wrapping the same library. It’s like measuring the IO performance between programming language standard libraries (they should all be close to the speed of the underlying system call).
- 0xbear 9y agoWe just plain can't do data augmentation quickly enough with TF. Queues-schmeyes, doesn't matter. Still tops out at about 35MB/sec on MS COCO and starves even a single Titan Xp. On the same hardware, with the same data augmentation steps, PyTorch gets ~50MB/s or so and saturates the GPU, since it never has to wait for data. In fact it can even read faster than that, and automatically parallelize the forward pass across several GPUs. You do still retain full control over placement, however. Super slick.
- tinyrick2 9y agoSome people (like me) use Windows. PyTorch is not supported by Windows currently.
- H1Supreme 9y agoSpin up a virtual machine? It's worth getting into Linux for machine learning.
- CodesInChaos 9y agoPassing GPUs to VMs is tricky, especially with nvidia consumer GPUs. It's certainly not "just spin up a VM".