5 ms·

I'm a professional scientist, so let me give my two cents on this matter. Being able to compare your work against SOTA (state of the art) is pretty critical in

I'm a professional scientist, so let me give my two cents on this matter. Being able to compare your work against SOTA (state of the art) is pretty critical in academic publications. If everyone else in your area uses framework X, it makes a lot of sense for you to do it too. For the last few years, Pytorch has been king for the topics I care about. However .. one area where Tensorflow shined was the static graph. As our models get even more intensive and needs different parts to execute in parallel, we are seeing some challenges in PyTorch's execution model. For example: https://pytorch.org/docs/stable/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel https://pytorch.org/docs/stable/notes/cuda.html#use-nn-paral... It appears to me that high performance model execution is a bit tricky if you want to do lots of things in parallels. TorchServe also seems quite simple compared to offerings from Tensorflow. So in summary, I think Tensorflow still has some features unmatched by others. It really depends on what you are doing.

erwincoumans 5y ago
Indeed, Google/Alphabet is gradually making the shift to JAX but also to ML Pathways towards models that support multiple tasks and multiple sensory inputs and sparse instead of dense: See https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-... and Jeff Dean's TED talk: https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be?language=en https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you...
- The_rationalist 5y ago
  what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
  - erwincoumans 5y ago
    For Academic Papers (the context of this HN topic), JAX and PyTorch makes more sense. A new runtime could be useful in production.
  - dekhn 5y ago
    tfrt underlies JAX. In many ways it's the system TF was supposed to have a 5 years ago.
    - erwincoumans 5y ago
      Yes indeed: https://blog.tensorflow.org/2022/02/tfrt-progress-update.html?m=1 https://blog.tensorflow.org/2022/02/tfrt-progress-update.htm... The ML Pathways infra is more complicated, and about more efficient scheduling many ops (in a graph) across many accelerators (rather than a small model on a single local GPU)
      - dekhn 5y ago
        sure, I know :) I used to work on TPUs at Google in Platforms, much of my time was spent working with Jax and other teams/researchers who found novel ways to break TPU hardware. What continues to surprise me is that for all his cleverness, Jeff Dean and the rest of the TF leadership spend the last 10 years basically recreating MPI-style high performance computing, but threw away all the learning and rebuilt every bit (except the matrix libraries) from scratch. TF started with parameter servers (every machine has its own copy of weights and periodically contributes them to a common model, asynchronously) to models that are sharded by data input and model structure that is mapped to the TPU topology (TPUv4 is a 3D wrapped torus). Really not that different from the T3E I used in the 90s.
probably_wrong 5y ago
I think Tensorflow made a bad move in academia by being so damn difficult to use on their earlier versions. Sure, their performance was always better than PyTorch's, but when you are an overworked PhD student you care less about your code being efficient and more about your code working at all. Word got around that debugging PyTorch was relatively painless, those earlier models made it into publications, and now here we are.
- p1esk 5y ago
  But the funny thing is - TF has never been faster than Pytorch. Even when Pytorch just came out they were roughly the same in terms of speed.
  - sillysaurusx 5y ago
    Not even remotely true. Or rather, our experiences differ greatly. The #1 problem with PyTorch is that it’s great if you want to use one videocard for training. Facebook has completely failed to support research scientists that want to do more than this. It’s no secret that I’m a jax fanboy. But I drink the koolaid because it tastes better than anyone else’s. PyTorch is gonna have a rude wake up call in about… oh, four years. They’ll wake up and hear everyone else comparing them to tensorflow, and it won’t be for the rosy reasons they currently enjoy. PyTorch devs are living in the dark ages without even realizing how much better it is when you have actual control over which parts of your program are JITed, along with an actual execution graph that you can walk and macroexpand lisp-style. https://jax.readthedocs.io/en/latest/autodidax.html https://jax.readthedocs.io/en/latest/autodidax.html should be required reading for every ML dev, and I can hardly get anyone to look at it. Sometimes I wonder if people just don’t see the steamroller coming for PyTorch. Probably — jax still reads to outsiders as a toy.
    - forgotmyoldacc 5y ago
      > The #1 problem with PyTorch is that it’s great if you want to use one videocard for training Incorrect information so confidently stated here. Tons of research papers that use more than one GPU for training, not sure what you're referring to? Standard DDP works fine, for starters.

erwincoumans 5y ago

Indeed, Google/Alphabet is gradually making the shift to JAX but also to ML Pathways towards models that support multiple tasks and multiple sensory inputs and sparse instead of dense: See https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-... and Jeff Dean's TED talk: https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you_think_but_it_could_be?language=en https://www.ted.com/talks/jeff_dean_ai_isn_t_as_smart_as_you...

The_rationalist 5y ago
what are your thoughts on https://github.com/tensorflow/runtime https://github.com/tensorflow/runtime ?
- erwincoumans 5y ago
  For Academic Papers (the context of this HN topic), JAX and PyTorch makes more sense. A new runtime could be useful in production.
- dekhn 5y ago
  tfrt underlies JAX. In many ways it's the system TF was supposed to have a 5 years ago.
  - erwincoumans 5y ago
    Yes indeed: https://blog.tensorflow.org/2022/02/tfrt-progress-update.html?m=1 https://blog.tensorflow.org/2022/02/tfrt-progress-update.htm... The ML Pathways infra is more complicated, and about more efficient scheduling many ops (in a graph) across many accelerators (rather than a small model on a single local GPU)
    - dekhn 5y ago
      sure, I know :) I used to work on TPUs at Google in Platforms, much of my time was spent working with Jax and other teams/researchers who found novel ways to break TPU hardware. What continues to surprise me is that for all his cleverness, Jeff Dean and the rest of the TF leadership spend the last 10 years basically recreating MPI-style high performance computing, but threw away all the learning and rebuilt every bit (except the matrix libraries) from scratch. TF started with parameter servers (every machine has its own copy of weights and periodically contributes them to a common model, asynchronously) to models that are sharded by data input and model structure that is mapped to the TPU topology (TPUv4 is a 3D wrapped torus). Really not that different from the T3E I used in the 90s.

probably_wrong 5y ago

I think Tensorflow made a bad move in academia by being so damn difficult to use on their earlier versions. Sure, their performance was always better than PyTorch's, but when you are an overworked PhD student you care less about your code being efficient and more about your code working at all. Word got around that debugging PyTorch was relatively painless, those earlier models made it into publications, and now here we are.

p1esk 5y ago
But the funny thing is - TF has never been faster than Pytorch. Even when Pytorch just came out they were roughly the same in terms of speed.
- sillysaurusx 5y ago
  Not even remotely true. Or rather, our experiences differ greatly. The #1 problem with PyTorch is that it’s great if you want to use one videocard for training. Facebook has completely failed to support research scientists that want to do more than this. It’s no secret that I’m a jax fanboy. But I drink the koolaid because it tastes better than anyone else’s. PyTorch is gonna have a rude wake up call in about… oh, four years. They’ll wake up and hear everyone else comparing them to tensorflow, and it won’t be for the rosy reasons they currently enjoy. PyTorch devs are living in the dark ages without even realizing how much better it is when you have actual control over which parts of your program are JITed, along with an actual execution graph that you can walk and macroexpand lisp-style. https://jax.readthedocs.io/en/latest/autodidax.html https://jax.readthedocs.io/en/latest/autodidax.html should be required reading for every ML dev, and I can hardly get anyone to look at it. Sometimes I wonder if people just don’t see the steamroller coming for PyTorch. Probably — jax still reads to outsiders as a toy.
  - forgotmyoldacc 5y ago
    > The #1 problem with PyTorch is that it’s great if you want to use one videocard for training Incorrect information so confidently stated here. Tons of research papers that use more than one GPU for training, not sure what you're referring to? Standard DDP works fine, for starters.