7 ms·

PyTorch vs. TensorFlow in Academic Papers

103 points by m3at 5y ago

jitl 5y ago
What about JAX?
- kavalg 5y ago
  JAX is really cool, but still somewhat immature. I would love to see it taking more ground and improving wrt e.g. integration with tensorboard and getting all the goodies we have in tensorflow. If you are looking for a higher level framework, I would recommend elegy [0] which is very close to the keras API. [0] https://github.com/poets-ai/elegy https://github.com/poets-ai/elegy
- PartiallyTyped 5y ago
  Jax is great, but there are some rough edges. I am using Jax for differentiable programming, and in many cases, I saw enormous speedup after jit, sometimes in the ballpark of 1e4. For Neural Networks, I use Equinox, and/or Elegy.
  - patrickkidger 5y ago
    I'm happy to hear about Equinox being used! (I'm the author.) I'm curious what your workloads are that you're seeing speedups of as much as 1e4? Greatest I've heard of before was ~1e2 on some differential equation solving.
    - chillee 5y ago
      If you’re massively dominated by overhead I can see it. I’ve definitely done microbenchmarks before where compilation gets you 1000x improvement.
    - PartiallyTyped 5y ago
      Mainly optimizers/solvers for multivariate functions. The 1e4 speedup was on a Trust Region optimizer. The algorithm was implemented to solve "the hard case"[1] involves multiple Cholesky factorizations, a matrix inversion, an eigenvalue decomposition on each step, and a call to scipy.linalg.solve_triangular. Part of the speedup is likely from caching/avoiding recomputations things. Granted, I had to rewrite a lot of the code to accommodate jax's peculiarities around python semantics, and made extensive use of jax.lax.{fori_loop, while_loop, scan, cond}. [1] http://www.apmath.spbu.ru/cnsa/pdf/monograf/Numerical_Optimization2006.pdf http://www.apmath.spbu.ru/cnsa/pdf/monograf/Numerical_Optimi..., see page 87.
brutus1213 5y ago
I'm a professional scientist, so let me give my two cents on this matter. Being able to compare your work against SOTA (state of the art) is pretty critical in academic publications. If everyone else in your area uses framework X, it makes a lot of sense for you to do it too. For the last few years, Pytorch has been king for the topics I care about. However .. one area where Tensorflow shined was the static graph. As our models get even more intensive and needs different parts to execute in parallel, we are seeing some challenges in PyTorch's execution model. For example: https://pytorch.org/docs/stable/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel https://pytorch.org/docs/stable/notes/cuda.html#use-nn-paral... It appears to me that high performance model execution is a bit tricky if you want to do lots of things in parallels. TorchServe also seems quite simple compared to offerings from Tensorflow. So in summary, I think Tensorflow still has some features unmatched by others. It really depends on what you are doing.
- erwincoumans 5y ago
  Indeed, Google/Alphabet is gradually making the shift to JAX but also to ML Pathways towards models that support multiple tasks and multiple sensory inputs and sparse instead of dense: See https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-... and Jeff Dean's TED talk: https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be?language=en https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you...
  - The_rationalist 5y ago
    what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
    - erwincoumans 5y ago
      For Academic Papers (the context of this HN topic), JAX and PyTorch makes more sense. A new runtime could be useful in production.
kylebgorman 5y ago
If I may, there's no real reason to break out ACL vs. NAACL vs. EMNLP, since they're all run by the ACL and one would be hard-pressed to say how the EMNLP community might differ from the ACL community at this point. And if you're doing NAACL you might want to do EACL and IJCNLP too.
cweill 5y ago
Can someone please share the current state of deploying Pytorch models to productions? TensorFlow has TF serving which is excellent and scalable. Last I checked there wasn't a PyTorch equivalent. I'm curious how these charts look for companies that are serving ML in production, not just research. Research is biased towards flexibility and ease of use, not necessarily scalability or having a production ecosystem.
- TheGuyWhoCodes 5y ago
  Honestly today there are way too many options. There is TorchServe but I haven't used it so I'm not sure how production ready it is. You have Nvidia's triton server which support cpu and gpu with tf1,tf2,pytorch,onnx and tensorRT. You have onnx runtime which can run on cpu and gpu and there are convertors from tf and pytorch to onnx. Then you have cloud based solutions like AWS sagemaker, elastic inference endpoints and even Inf1 instances that use AWS Inferentia chips which you would run with the Neuron SDK, they even have TensorFlow serving containers with built it support for Inferentia. End of the day it really depends on your model, size, latency, inference runtime and the cost obviously. And that's before optimizations like FP16, BFLOAT16, TF32, INT8, pruning, layers rewrite, getting rid of batch normalization etc. Then you have up and coming solutions like Neural Magic (not associated) deepsparse to create sparse models for inference. And that's just for cloud if you are talking about edge ml it's even more down the rabbit hole..
savant_penguin 5y ago
A big mistake on the side of tensorflow was trying to copy theano including those dreadful functional loops whereas in pytorch for loops are not pain to use and very well integrated with the language
siver_john 5y ago
The graph on here seems similar to what I've noticed. My lab mainly uses Tensorflow mainly driven by my knowledge. And the only reason why I learned Tensorflow initially was that PyTorch was just starting when I was choosing a framework and the documentation wasn't as established. However, recently I recommended a student who was asking me which framework to probably use PyTorch due to the ease of implementation comparatively.
- rg111 5y ago
  That’s a good decision. I know both Keras and PyTorch, and I will recommend PyTorch any day. I haven’t used TF/Keras in the last 2.5 years outside of Edge AI projects- neither at work nor at personal projects. JAX is interesting, though. But definitely not advisable as someone's first DL framework.
fartcannon 5y ago
Tensorflow was too hard. Keras wasn't hard enough. Pytorch was juuuuust right.
jankaul 5y ago
I mostly used Tensorflow and I'm curious what makes PyTorch models easier to implement? With Tensorflow 2 you get the Keras API which is really easy to use.

103 points by m3at 5y ago

jitl 5y ago

What about JAX?

kavalg 5y ago
JAX is really cool, but still somewhat immature. I would love to see it taking more ground and improving wrt e.g. integration with tensorboard and getting all the goodies we have in tensorflow. If you are looking for a higher level framework, I would recommend elegy [0] which is very close to the keras API. [0] https://github.com/poets-ai/elegy https://github.com/poets-ai/elegy
PartiallyTyped 5y ago
Jax is great, but there are some rough edges. I am using Jax for differentiable programming, and in many cases, I saw enormous speedup after jit, sometimes in the ballpark of 1e4. For Neural Networks, I use Equinox, and/or Elegy.
- patrickkidger 5y ago
  I'm happy to hear about Equinox being used! (I'm the author.) I'm curious what your workloads are that you're seeing speedups of as much as 1e4? Greatest I've heard of before was ~1e2 on some differential equation solving.
  - chillee 5y ago
    If you’re massively dominated by overhead I can see it. I’ve definitely done microbenchmarks before where compilation gets you 1000x improvement.
  - PartiallyTyped 5y ago
    Mainly optimizers/solvers for multivariate functions. The 1e4 speedup was on a Trust Region optimizer. The algorithm was implemented to solve "the hard case"[1] involves multiple Cholesky factorizations, a matrix inversion, an eigenvalue decomposition on each step, and a call to scipy.linalg.solve_triangular. Part of the speedup is likely from caching/avoiding recomputations things. Granted, I had to rewrite a lot of the code to accommodate jax's peculiarities around python semantics, and made extensive use of jax.lax.{fori_loop, while_loop, scan, cond}. [1] http://www.apmath.spbu.ru/cnsa/pdf/monograf/Numerical_Optimization2006.pdf http://www.apmath.spbu.ru/cnsa/pdf/monograf/Numerical_Optimi..., see page 87.

brutus1213 5y ago

I'm a professional scientist, so let me give my two cents on this matter. Being able to compare your work against SOTA (state of the art) is pretty critical in academic publications. If everyone else in your area uses framework X, it makes a lot of sense for you to do it too. For the last few years, Pytorch has been king for the topics I care about. However .. one area where Tensorflow shined was the static graph. As our models get even more intensive and needs different parts to execute in parallel, we are seeing some challenges in PyTorch's execution model. For example: https://pytorch.org/docs/stable/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel https://pytorch.org/docs/stable/notes/cuda.html#use-nn-paral... It appears to me that high performance model execution is a bit tricky if you want to do lots of things in parallels. TorchServe also seems quite simple compared to offerings from Tensorflow. So in summary, I think Tensorflow still has some features unmatched by others. It really depends on what you are doing.

erwincoumans 5y ago
Indeed, Google/Alphabet is gradually making the shift to JAX but also to ML Pathways towards models that support multiple tasks and multiple sensory inputs and sparse instead of dense: See https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-... and Jeff Dean's TED talk: https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be?language=en https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you...
- The_rationalist 5y ago
  what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
  - erwincoumans 5y ago
    For Academic Papers (the context of this HN topic), JAX and PyTorch makes more sense. A new runtime could be useful in production.

kylebgorman 5y ago

If I may, there's no real reason to break out ACL vs. NAACL vs. EMNLP, since they're all run by the ACL and one would be hard-pressed to say how the EMNLP community might differ from the ACL community at this point. And if you're doing NAACL you might want to do EACL and IJCNLP too.

cweill 5y ago

Can someone please share the current state of deploying Pytorch models to productions? TensorFlow has TF serving which is excellent and scalable. Last I checked there wasn't a PyTorch equivalent. I'm curious how these charts look for companies that are serving ML in production, not just research. Research is biased towards flexibility and ease of use, not necessarily scalability or having a production ecosystem.

TheGuyWhoCodes 5y ago
Honestly today there are way too many options. There is TorchServe but I haven't used it so I'm not sure how production ready it is. You have Nvidia's triton server which support cpu and gpu with tf1,tf2,pytorch,onnx and tensorRT. You have onnx runtime which can run on cpu and gpu and there are convertors from tf and pytorch to onnx. Then you have cloud based solutions like AWS sagemaker, elastic inference endpoints and even Inf1 instances that use AWS Inferentia chips which you would run with the Neuron SDK, they even have TensorFlow serving containers with built it support for Inferentia. End of the day it really depends on your model, size, latency, inference runtime and the cost obviously. And that's before optimizations like FP16, BFLOAT16, TF32, INT8, pruning, layers rewrite, getting rid of batch normalization etc. Then you have up and coming solutions like Neural Magic (not associated) deepsparse to create sparse models for inference. And that's just for cloud if you are talking about edge ml it's even more down the rabbit hole..

savant_penguin 5y ago

A big mistake on the side of tensorflow was trying to copy theano including those dreadful functional loops whereas in pytorch for loops are not pain to use and very well integrated with the language

siver_john 5y ago

The graph on here seems similar to what I've noticed. My lab mainly uses Tensorflow mainly driven by my knowledge. And the only reason why I learned Tensorflow initially was that PyTorch was just starting when I was choosing a framework and the documentation wasn't as established. However, recently I recommended a student who was asking me which framework to probably use PyTorch due to the ease of implementation comparatively.

rg111 5y ago
That’s a good decision. I know both Keras and PyTorch, and I will recommend PyTorch any day. I haven’t used TF/Keras in the last 2.5 years outside of Edge AI projects- neither at work nor at personal projects. JAX is interesting, though. But definitely not advisable as someone's first DL framework.

fartcannon 5y ago

Tensorflow was too hard. Keras wasn't hard enough. Pytorch was juuuuust right.

jankaul 5y ago

I mostly used Tensorflow and I'm curious what makes PyTorch models easier to implement? With Tensorflow 2 you get the Keras API which is really easy to use.