19 ms·

PyTorch vs. TensorFlow in 2022

350 points by SleekEagle 5y ago

minimaxir 5y ago
At this point, there's so much conceptual and syntactical overlap between the two frameworks that there isn't a "winner." Like most DS/ML tools, use which one fits your use case.
- SleekEagle 5y ago
  I totally agree! I think their technical differences are superseded by other practical considerations at this point. It's interesting to see how the landscape for which is better given a use case is changing though - PyTorch has made a lot of effort to grab more of the industry sector with TorchServe and PyTorch Live. Do you think PyTorch can catch up here? I think Google's Coral devices give them a lock on embedded devices in the coming years
  - dontreact 5y ago
    I tried and failed to use Google Coral about 8months ago. The dev experience was terrible. Our company just went with deploying using openvino on cpu, which was fast enough. I’m not sure coral has enough of an edge to make it worthwhile relative to simpler edge deployment options like cpu
tehf0x 5y ago
As a researcher in RL & ML in a big industry lab, I would say most of my colleagues are moving to JAX [https://github.com/google/jax https://github.com/google/jax], which this article kind of ignores. JAX is XLA-accelerated NumPy, it's cool beyond just machine learning, but only provides low-level linear algebra abstractions. However you can put something like Haiku [https://github.com/deepmind/dm-haiku https://github.com/deepmind/dm-haiku] or Flax [https://github.com/google/flax https://github.com/google/flax] on top of it and get what the cool kids are using :)
- SleekEagle 5y ago
  JAX is really exciting! JAX is mentioned in the Research subsection of the "Which should I pick" section. Do you think that the fundamental under-the-hood differences of JAX compared to TensorFlow and PyTorch will affect its adoption? Haiku is really cool - I haven't used Flax. It'll be really interested to see the development of JAX as time goes on. I also saw some benchmarks that show its neck-and-neck with PyTorch as the fastest of the three, but I think with more optimization its ceiling is higher than PyTorch's.
  - sillysaurusx 5y ago
    > Do you think that the fundamental under-the-hood differences of JAX compared to TensorFlow and PyTorch will affect its adoption? Of course. It's the only library that can be explained from first principles: https://jax.readthedocs.io/en/latest/autodidax.html https://jax.readthedocs.io/en/latest/autodidax.html
    - SleekEagle 5y ago
      Wow that's a really cool resource. Thanks for linking! Even still, do you think researchers will want to take the time to learn all of that when PyTorch gives them no real reason to switch? Every day spent learning JAX is another day spent not reviewing literature, writing papers, or developing new models.
      - 6gvONxR4sf7o 5y ago
        > Every day spent learning JAX is another day spent not reviewing literature, writing papers, or developing new models. Every day spent learning JAX is also another day spent not trying to fit a round peg into a square hole of other libraries. I made the leap when I was doing things that were painful in pytorch. In terms of time, I think I came out ahead. Not everything is a nail, and pytorch is better for some things, an jax is better for others. "Every day spent learning the screwdriver is a day spent not using your hammer."
cfcf14 5y ago
For any kind of research or experimental work, I cannot imagine using anything other than PyTorch, with the caveat that I do think JAX is extremely impressive and I've been meaning to learn more about it for a while. Even though I've been working with Tensorflow for a few years now and I feel like I do understand the API pretty well, to some extent that just means I'm _really_ good at navigating the documentation, because there's no way to intuit the way things work. And I still run into bizarre performance issues when profiling graphs pretty much all the time. Some ops are just inefficient - oh but it was fixed in 2.x.yy! Oh but then it broke again in 2.x.yy+1! Sigh. However - and I know this is a bit of a tired trope, but any kind of industrial deployment is just vastly, vastly easier with Tensorflow. I'm currently working with ultra-low-latency model development targeting a Tensorflow-Lite inference engine (C-API, wrapped via Rust) and it's just incredibly easy. With some elbow grease and willingness to dive into low level TF-Lite optimisations, one can see end to end model inference times in the order of 10-100us for simple models (say, a fully connected dnn with a few million parameters), and between 100us-1ms for fairly complex models utilising contemporary architectures in computer vision or NLP. Memory overhead and control over inference computation semantics are easy. As a nice cherry on top, we can take the same Tensorflow SavedModels that get compiled to TF-Lite files and instead compile them to tensorflow-js for easy web deployment, which is a great portability upside. However, I know there's some incredible progress being made on what one might call 'environmental agnostic computational graph ILs' (on second thought, let's not keep that name) which should open up more options for inference engines and graph optimisations (operator fusion, rollups, hardware dependant stuff, etc). Overall I feel like things have been continuously getting better for the last 5 years or so. I'm pleased to see so many more options.
- SleekEagle 5y ago
  Agreed - JAX is really cool. It will be interesting to see how TF & JAX develop considering they're both made by Google. I also think JAX has the potential to be the fastest, although right now it's neck-and-neck with PyTorch. Yes - a lot of TF users don't realize that knowing the "tricks of the trade" for wrangling TF don't apply in PT because it just works more easily. I agree that industry-centric applications should probably use TF. TFX is just invaluable. Have you checked out Google's Coral devices? TFLite + Coral = revolution for a lot of industries. Thanks for all your comments - I'm also really excited to see what the coming years bring. While we might debate if PT or TF is better, they're both undoubtedly improving very rapidly! So excited to see how ML/DL applications start permeating other industries
Kalanos 5y ago
Post doesn't talk about the actual libraries, just the ecosystems surrounding them. TF has more layer types, parallelizes better, is easier to assemble w keras, and you don't have to recreate the optimizer when loading from disk. pytorch doesn't have metrics out of the box. TF all the way. github.com/aiqc/aiqc
- SleekEagle 5y ago
  Do you think it's a bit easier to build custom layers in PyTorch though? Also, I think Lightning handles the issue of loading optimizers, but I'm not sure about that. It's nice to see TF get some love, but I still think PyTorch has easier debugging and is more pythonic which lowers the barrier to entry for a lot of people.
- dbish 5y ago
  Curious since I’ve been looking at this recently. What out of the box metrics would matter most to you? There are lots of libraries in the ecosystem for metrics but I’ve seen the request for built in metrics a few times now so it must be a clear need.
  - Kalanos 5y ago
    mostly just accuracy and r squared. i use lightning's torchmetrics.
- Kalanos 5y ago
  challenging to get multi-dimensional layer sizes right. torch is not compatible numpy as input, which is painful for pre and post processing. no metrics means no History object. no activation attribute for Linear layers and Softmax layer behaves weird.
303bookworm 5y ago
Mumble grumble Keras. ("Shakes old man's fist at sky, and at people who seems to enjoy boilerplate code too much")
- yboris 5y ago
  Keras is wonderful. I can't understand whether your comment is in favor or against it. Either way, could you share why?
  - oofbey 5y ago
    Keras is great for easy problems. But as soon as you want to color outside the lines its opinionated simple model just gets in the way. Every time I've tried to use keras for real work, it was a total pain in the butt. Do your homework in keras. Use something else for real work.
    - yboris 5y ago
      As the originally-posted-link suggests, Keras is a great way to start in ML (this is how I started, with the book Deep Learning with Python by François Chollet, the creator of Keras). And after one is comfortable with the concepts and the general flow of things, and if one needs to do more advanced stuff (as you say), then move on to something like PyTorch (I love that they mention PyTorch Lightning in the article).
    - otacust 5y ago
      I used fastai quite a bit with PyTorch and ended up feeling the same way. Great for spinning up an effective model in a few lines, but really tough to make it do exactly what you want it to do.
dash2 5y ago
"PyTorch and TensorFlow are far and away the two most popular Deep Learning frameworks today. The debate over whether PyTorch or TensorFlow is superior is a longstanding point of contentious debate, with each camp having its share of fervent supporters. Both PyTorch and TensorFlow..." Can an article really be any good if it starts off with such obvious SEO spam?
- breezeTrowel 5y ago
  > Can an article really be any good if it starts off with such obvious SEO spam? That's an interesting take. I fail to see how such mentions, in an article that compares two things and, thus, mentions both things together, are in any way SEO spam? For example, in an article comparing apples and oranges I would expect to see a rather high number of mentions of "apples and oranges". After all, that is the topic.
  - ojosilva 5y ago
    It's SEO for the OP site. The parent comment is saying the words TF and PyTorch are repeated exhaustively, I think sometimes senselessly, throughout the entire article. It doesn't mean the content is not of value.
  - edgyquant 5y ago
    That may be the topic but that first sentence reads like they are aiming to have a high number of mentions not a quality of content. For instance you can replace the second and third mentions of “PyTorch and Tensorflow” with something like “the two” or “they both.”
    - deleted 5y ago
      [deleted]
  - arrakis2021 5y ago
    This is keyword stuffing, a very old SEO technique which is obsolete. In fact such behavior works against you these days.
- 5y ago
axegon_ 5y ago
At one point I lost interest in both and ML/AI in general. I think eventually I got frustrated with the insane amounts of abuses for marketing purposes and never truly delivering what was promised(I know, I know, fake it till you make it). For better or worse far too few managed to make it, so most stuck with faking. I think I lost interest completely around 2019. But even back then they were starting to seem like twins - practically identical with some minor but sometimes subtle differences. Looking at random sections of the documentation, all you gotta do is ignore the semantics... > tf.nn.conv2d(input, filters, strides, padding, data_format='NHWC', dilations=None, name=None) > torch.nn.functional.conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1)
- hwers 5y ago
  Personally I wish we'd get beyond these low level abstractions by now. Machine learning is so wonderfully mathematical that making abstractions from the details should be incredibly easy and powerful. I can't believe like 8 years after the big ML wave e.g. frontend javascript developers aren't enjoying the fruits of these labors (despite there being no good reason for them not to be able to).
  - dbish 5y ago
    What do you envision that would help JavaScript devs take advantage of ML? There is tensorflow.js. Are you thinking completely different ‘building blocks’ that provide higher level apis like fast.ai geared towards frontend devs or something else?
    - SleekEagle 5y ago
      They actually made ml5.js for this reason, not sure if you've seen it https://ml5js.org/ https://ml5js.org/ Pretty cool imo
  - SleekEagle 5y ago
    Have you checked out TensorFlow.js? May be what you're looking for...
tadeegan 5y ago
Tensorflow is just such a classic clusterfuck google project. V2 had huge breaking changes (reminiscent of angular) and tons of the apis are clunky and don’t work well together. There are like 3 different ways to save a model. It’s almost like a bunch of teams built features with no oversight. I’m pretty sure tf is considered in maintenance mode within google as Brain and the tf creators themselves have moved to Jax. I do think Google learned a lot from tensorflow and am excited to see Jax pan out. Pytorch is a pleasure to debug. I think pytorch jit could close the deployment gap.
- dbish 5y ago
  Any other deployment issues for PyTorch you’re aware of that would help ‘close the gap’?
  - tadeegan 5y ago
    I think PyTorch c++ api is less mature and harder to compile into other projects. Tensorflow started with the c++ api exposed which is why the graph format is so stable and favorable to deployment in heterogeneous environments.
- stathibus 5y ago
  The article gives credit to TF server and TFLite and so forth as being better for deployment, but leaves out the fact that those systems don't fucking work most of the time, and support is pretty much over at this point. The same goes for model support; even the models in TF's own repository are sometimes broken or don't follow the API conventions set forth in the documentation. I honestly don't know how anyone uses TF in production at this point, unless they are frozen on a specific old version and have figured out an environment that works with their specific models already.
  - SleekEagle 5y ago
    Yeah, TensorFlow's API has definitely gotten convoluted and confusing. I think the shift from TF1 to TF2 and then later wrapping Keras in TF just caused a lot of problems. TensorFlow seems to be spreading itself pretty thin. Maintaining so many language bindings, TensorFlow.js, TFlite, Server, etc. seem like they could all use some focus, BUT, and this is a big but, do you think if they can get each part of their ecosystem to an easily usable point that they'll have cornered the industry sector? PyTorch is taking a much more targeted approach as seen with PyTorch Live, but I truly think that TFLite + Coral will be a game-changer for a lot of industries (and Google will make a fortune in the process). To me it seems like this is where Google's focus has lain in the AI space for the past couple of years. What do you think?
lettergram 5y ago
The thing is, tensorflow has more ability to run cross platform. I help maintain https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run distributed and federated use cases with custom layers. All of which are improved in tensorflow. That said, I’d use pytorch if I could. Simply put, it has a better user experience.
- SleekEagle 5y ago
  Yeah, it's pretty funny how reluctantly some people use TF because they have too, lol. The fact that PyTorch is pythonic and easier to debug makes it better for a ton of users, but TensorFlow keeps the entire DL process in mind more, not just modeling.
- ojosilva 5y ago
  I'm on that boat. Tensorflow.js is the only decent ML library for JS. Google support for TF.js has been dwindling, but new versions are still coming out AFAIK and we've just got Apple Silicon (M1) support.
astroalex 5y ago
No mention of Google MediaPipe (https://mediapipe.dev/ https://mediapipe.dev/), which is a mobile/edge framework for deploying TFLite models. MediaPipe has the advantage of letting you stitch together multiple models, transformations, and heuristics into a higher level computational graph. I'm not aware of any equivalent for PyTorch, although PyTorch Live seems like baby steps in that direction.
- justinlloyd 5y ago
  I am using mediapipe extensively in my day-to-day job. I've been impressed with it so far, and the ability to patch together multiple models in to, well... a media pipe, has been impressively useful. But I am also waiting for the penny to drop and google to announce they've abandoned it, or that their API has completely changed.
  - astroalex 5y ago
    Do you work at my company? Because that's our biggest fear too. I asked a friend of mine at Google to sleuth around internally and get a sense for the health of the project. He said that it's used on some internal projects and seems to have a pretty healthy internal website. So hopefully it won't be cancelled soon.
    - justinlloyd 5y ago
      > Do you work at my company? Maybe. But I only get paid in whatever snacks I can scrounge from the employee fridge and pickings have been slim of late since so few people come into the office these days. I am down to the "expired mystery mozzarella cheese sticks" and the leftover ketchup packages from Woodranch BBQ & Grill that are in the drawer with all the unused chopsticks and plastic forks. I too spoke to a friend at Google that is part of the team, and whilst he said there were no plans to cancel it, or make radical changes, when I asked about unplanned plans, he kinda just shrugged and said "You know Google..." I have a dual solution approach, Mediapipe for "in use now" and OpenPose for validation, slower processing and the "Google just **ed us" moment we're both anticipating. I need to build my own pose analysis system, but right now I don't have the bandwidth. On the last day of Christmas the CEO sent to me: Thirty-two Manfrotto Tripod extenders Sixteen Manfrotto tripods Sixteen high speed cables Sixteen Manfrotto C-clamps Sixteen Manfrotto 3/8 to 1/4-20 reducers Sixteen Quick release mounts Sixteen 4K cameras Fooouuuurrrrr high-speeeeeed PCIe capture cards Three days to hit deadline Two triggered circuit breakers One really huge headache And a new VR H.M.D.
nmca 5y ago
This article says that Google and DeepMind research use TF - but they don't. DeepMind use JAX almost exclusively, and many brain researchers use JAX too. ML eng is my area of expertise, and I would advise strongly against tensorflow.
- artdigital 5y ago
  What would you recommend for new comers that want to get into the space?
cinntaile 5y ago
In TF you don't have to manually calculate the number of parameters of a fully connected layer. That's kind of nice. I don't really understand why PyTorch requires me to do that. Surely this can be infered from the previous layer.
- SleekEagle 5y ago
  "same" padding for convolutional layers is super convenient too
  - cinntaile 5y ago
    Yeah true, that's also a bit annoying in PyTorch compared to TF. I remember reading on the PyTorch forums that this was a bit hard to implement for some reason, but I can't recall any details if there were any.
  - jbschlosser 5y ago
    FYI PyTorch now supports "same" padding for convolutions with stride=1; see the docs here: https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.ht... The stride>1 case has been a bit more controversial within TensorFlow, and there is ongoing discussion on the correct way to implement it within PyTorch on the issue: https://github.com/pytorch/pytorch/issues/3867 https://github.com/pytorch/pytorch/issues/3867
    - SleekEagle 5y ago
      Thanks for letting me know!
- jbschlosser 5y ago
  This is supported by PyTorch as well with nn.LazyLinear: https://pytorch.org/docs/stable/generated/torch.nn.LazyLinear.html https://pytorch.org/docs/stable/generated/torch.nn.LazyLinea....
  - 5y ago
strzibny 5y ago
This looks like a good comparison. A lot of comparisons online are way superficial or plain wrong.
- SleekEagle 5y ago
  I think the checklist of technical differences is a bit outdated at this point!
justinlloyd 5y ago
I will preface with the statement that my knowledge may be slightly out of date as I don't keep up on every nuanced change. I use PyTorch and TensorFlow, and the article is spot-on in regard to "mystery operations that take a long time" with no real rhyme or reason behind them with regard to TensorFlow. That said, on the whole, I skew more towards TensorFlow because it is generally easier to reason about the graph and how it connects. I also find the models that are available to usually be more refined, robust and useful straight out of the box. With PyTorch I am usually fighting a slew of version incompatibilities in the API between even more point releases. The models often feel more slap-dash thrown together, research like projects or toy projects, and whilst the article points out the number of papers that use PyTorch far exceeds those that use TensorFlow, and the number of models for PyTorch dwarfs that of TensorFlow, there isn't a lot of quality in the quantity. "90% of everything is crap." Theodore Sturgeon. And that goes double for PyTorch models. A lot of the models, and even some datasets, just feel like throwaway projects that someone put up online. If you are on macOS or Linux and using Python, PyTorch works fine, but don't step outside of that boundary. PyTorch and TensorFlow work with other operating systems, and other languages besides Python, but working with anything but Python when using PyTorch is a painful process fraught with pain. And yes, I expect someone to drop in and say "but what about this C++ framework?" or "I use language X with PyTorch every day and it works fine for me!" But again, the point stands, anything but Python with PyTorch is painful. The support of other languages for TensorFlow is far richer and far better. And I will preface this with, "my knowledge may be out of date" but I've also noticed the type of models and project code available for TensorFlow and PyTorch diverge wildly once you get outside of the toy projects. If you are doing computer vision, especially with video and people, and you are not working on the most simplest of pose analysis, TensorFlow offers a lot more options of stuff straight out of the box. PyTorch has some good projects and models, but they are mostly of the Mickey Mouse hobby stuff, or an abstract research project that isn't very robust or immediately deployable. I use TensorFlow in my day-to-day job. All that said, I like PyTorch for its quirkiness, its rapid prototyping, its popularity, and the fact that so many people are trying out a lot of different things, even if they don't work particularly well. I use PyTorch in almost all of my personal research projects. I expect in the future for PyTorch to get more stable and more deployable and have better tools, if it can move slightly away from the "research tool" phase it is currently in. I expect Google to do the usual Google-Fuck-Up and completely change TF for the worse, break compatibility (TF1 to TF2) or just abandon the project entirely and move on to the next new shiny.
lvl100 5y ago
I just want an alternative to CUDA. I am sick of Nvidia’s monopoly in this space.
- nestorD 5y ago
  I believe all big frameworks have some work being done to make them compatible with AMD GPUs. Here is the relevant issue for JAX (support seem to be in alpha but viable): https://github.com/google/jax/issues/2012 https://github.com/google/jax/issues/2012
- YetAnotherNick 5y ago
  The best thing that worked for me is the Apple's tensorflow pluggable device for metal. It could utilize both AMD and M1 GPU to 100% capacity. It's a shame apple could do it as a small project in the side to use metal but AMD couldn't do the same with Vulkan.
g42gregory 5y ago
As a practitioner, I feel that oftentimes you are extending, fine tuning somebody else’s code or pre-trained models (DeepMind’s for example). This means that you should be able to work on whatever the platform this code came with. Basically, you should be able to work with JAX, TF or PyTorch with equal ease.
- SleekEagle 5y ago
  Given that the vast majority of publications are using PyTorch, it kind of makes it the go-to
bigbillheck 5y ago
I started with tensorflow back in 2017 or so, then in my current job I am touching a lot of pytorch, and as such I say a pox on both their houses. But keras is OK and I greatly appreciate that you can (sometimes) serialize everything to hdf5.
streamofdigits 5y ago
in the fullness of time JAX might prove to be more important than either. don't give people fish, teach them how to catch fish and all that...
bjourne 5y ago
Great article. While I only had time to skim the article, I'll still offer my uninformed opinions. :) None of the hw I own is particularly great. I don't even own a decent GPU. But I don't need to because you can train your models for FREE on Google Colab's TPUs and GPUs. PyTorch's TPU support is still not that great while TensorFlow's is maturing. It's obviously a priority for Google to make TensorFlow work well on their own hardware. So for me the choice is TF 2 because I can train models 5-10x faster using Google's TPUs than if I had used PyTorch. I know the PyTorch developers are working on TPU support but last I checked (this spring) it wasn't there yet and I wasn't able to make it work well on Google Colab.
synergy20 5y ago
As a layman in ML, I thought PyTorch is more(or only) geared towards researchers while Tensorflow has its problems it is the _only_ one that provides a commercial solutions that you can deploy, is this still true? JAX is totally new to me, is this Google's new Tensorflow in the future?
johnnyApplePRNG 5y ago
I always look to Google trends when trying to decide upon which framework to use. PyTorch just recently took the lead. [0] So if I were having to choose between learning the either of them, I would go with PyTorch. [0] https://trends.google.com/trends/explore?date=all&q=tensorflow,pytorch https://trends.google.com/trends/explore?date=all&q=tensorfl...
d4rkp4ttern 5y ago
Does TF have any advantages in terms of ease of acceleration (training and inference) with multiple GPUs? Our pipeline is all PyTorch Lightning — this made development easy but we have been having numerous issues trying to leverage multiple GPUs (this is for sequence models), keep getting strange errors.

350 points by SleekEagle 5y ago

At this point, there's so much conceptual and syntactical overlap between the two frameworks that there isn't a "winner." Like most DS/ML tools, use which one fits your use case.

SleekEagle 5y ago
I totally agree! I think their technical differences are superseded by other practical considerations at this point. It's interesting to see how the landscape for which is better given a use case is changing though - PyTorch has made a lot of effort to grab more of the industry sector with TorchServe and PyTorch Live. Do you think PyTorch can catch up here? I think Google's Coral devices give them a lock on embedded devices in the coming years
- dontreact 5y ago
  I tried and failed to use Google Coral about 8months ago. The dev experience was terrible. Our company just went with deploying using openvino on cpu, which was fast enough. I’m not sure coral has enough of an edge to make it worthwhile relative to simpler edge deployment options like cpu

tehf0x 5y ago

As a researcher in RL & ML in a big industry lab, I would say most of my colleagues are moving to JAX [https://github.com/google/jax https://github.com/google/jax], which this article kind of ignores. JAX is XLA-accelerated NumPy, it's cool beyond just machine learning, but only provides low-level linear algebra abstractions. However you can put something like Haiku [https://github.com/deepmind/dm-haiku https://github.com/deepmind/dm-haiku] or Flax [https://github.com/google/flax https://github.com/google/flax] on top of it and get what the cool kids are using :)

SleekEagle 5y ago
JAX is really exciting! JAX is mentioned in the Research subsection of the "Which should I pick" section. Do you think that the fundamental under-the-hood differences of JAX compared to TensorFlow and PyTorch will affect its adoption? Haiku is really cool - I haven't used Flax. It'll be really interested to see the development of JAX as time goes on. I also saw some benchmarks that show its neck-and-neck with PyTorch as the fastest of the three, but I think with more optimization its ceiling is higher than PyTorch's.
- sillysaurusx 5y ago
  > Do you think that the fundamental under-the-hood differences of JAX compared to TensorFlow and PyTorch will affect its adoption? Of course. It's the only library that can be explained from first principles: https://jax.readthedocs.io/en/latest/autodidax.html https://jax.readthedocs.io/en/latest/autodidax.html
  - SleekEagle 5y ago
    Wow that's a really cool resource. Thanks for linking! Even still, do you think researchers will want to take the time to learn all of that when PyTorch gives them no real reason to switch? Every day spent learning JAX is another day spent not reviewing literature, writing papers, or developing new models.
    - 6gvONxR4sf7o 5y ago
      > Every day spent learning JAX is another day spent not reviewing literature, writing papers, or developing new models. Every day spent learning JAX is also another day spent not trying to fit a round peg into a square hole of other libraries. I made the leap when I was doing things that were painful in pytorch. In terms of time, I think I came out ahead. Not everything is a nail, and pytorch is better for some things, an jax is better for others. "Every day spent learning the screwdriver is a day spent not using your hammer."

cfcf14 5y ago

For any kind of research or experimental work, I cannot imagine using anything other than PyTorch, with the caveat that I do think JAX is extremely impressive and I've been meaning to learn more about it for a while. Even though I've been working with Tensorflow for a few years now and I feel like I do understand the API pretty well, to some extent that just means I'm _really_ good at navigating the documentation, because there's no way to intuit the way things work. And I still run into bizarre performance issues when profiling graphs pretty much all the time. Some ops are just inefficient - oh but it was fixed in 2.x.yy! Oh but then it broke again in 2.x.yy+1! Sigh. However - and I know this is a bit of a tired trope, but any kind of industrial deployment is just vastly, vastly easier with Tensorflow. I'm currently working with ultra-low-latency model development targeting a Tensorflow-Lite inference engine (C-API, wrapped via Rust) and it's just incredibly easy. With some elbow grease and willingness to dive into low level TF-Lite optimisations, one can see end to end model inference times in the order of 10-100us for simple models (say, a fully connected dnn with a few million parameters), and between 100us-1ms for fairly complex models utilising contemporary architectures in computer vision or NLP. Memory overhead and control over inference computation semantics are easy. As a nice cherry on top, we can take the same Tensorflow SavedModels that get compiled to TF-Lite files and instead compile them to tensorflow-js for easy web deployment, which is a great portability upside. However, I know there's some incredible progress being made on what one might call 'environmental agnostic computational graph ILs' (on second thought, let's not keep that name) which should open up more options for inference engines and graph optimisations (operator fusion, rollups, hardware dependant stuff, etc). Overall I feel like things have been continuously getting better for the last 5 years or so. I'm pleased to see so many more options.

SleekEagle 5y ago
Agreed - JAX is really cool. It will be interesting to see how TF & JAX develop considering they're both made by Google. I also think JAX has the potential to be the fastest, although right now it's neck-and-neck with PyTorch. Yes - a lot of TF users don't realize that knowing the "tricks of the trade" for wrangling TF don't apply in PT because it just works more easily. I agree that industry-centric applications should probably use TF. TFX is just invaluable. Have you checked out Google's Coral devices? TFLite + Coral = revolution for a lot of industries. Thanks for all your comments - I'm also really excited to see what the coming years bring. While we might debate if PT or TF is better, they're both undoubtedly improving very rapidly! So excited to see how ML/DL applications start permeating other industries

Kalanos 5y ago

Post doesn't talk about the actual libraries, just the ecosystems surrounding them. TF has more layer types, parallelizes better, is easier to assemble w keras, and you don't have to recreate the optimizer when loading from disk. pytorch doesn't have metrics out of the box. TF all the way. github.com/aiqc/aiqc

SleekEagle 5y ago
Do you think it's a bit easier to build custom layers in PyTorch though? Also, I think Lightning handles the issue of loading optimizers, but I'm not sure about that. It's nice to see TF get some love, but I still think PyTorch has easier debugging and is more pythonic which lowers the barrier to entry for a lot of people.
dbish 5y ago
Curious since I’ve been looking at this recently. What out of the box metrics would matter most to you? There are lots of libraries in the ecosystem for metrics but I’ve seen the request for built in metrics a few times now so it must be a clear need.
- Kalanos 5y ago
  mostly just accuracy and r squared. i use lightning's torchmetrics.
Kalanos 5y ago
challenging to get multi-dimensional layer sizes right. torch is not compatible numpy as input, which is painful for pre and post processing. no metrics means no History object. no activation attribute for Linear layers and Softmax layer behaves weird.

303bookworm 5y ago

Mumble grumble Keras. ("Shakes old man's fist at sky, and at people who seems to enjoy boilerplate code too much")

yboris 5y ago
Keras is wonderful. I can't understand whether your comment is in favor or against it. Either way, could you share why?
- oofbey 5y ago
  Keras is great for easy problems. But as soon as you want to color outside the lines its opinionated simple model just gets in the way. Every time I've tried to use keras for real work, it was a total pain in the butt. Do your homework in keras. Use something else for real work.
  - yboris 5y ago
    As the originally-posted-link suggests, Keras is a great way to start in ML (this is how I started, with the book Deep Learning with Python by François Chollet, the creator of Keras). And after one is comfortable with the concepts and the general flow of things, and if one needs to do more advanced stuff (as you say), then move on to something like PyTorch (I love that they mention PyTorch Lightning in the article).
  - otacust 5y ago
    I used fastai quite a bit with PyTorch and ended up feeling the same way. Great for spinning up an effective model in a few lines, but really tough to make it do exactly what you want it to do.

dash2 5y ago

"PyTorch and TensorFlow are far and away the two most popular Deep Learning frameworks today. The debate over whether PyTorch or TensorFlow is superior is a longstanding point of contentious debate, with each camp having its share of fervent supporters. Both PyTorch and TensorFlow..." Can an article really be any good if it starts off with such obvious SEO spam?

breezeTrowel 5y ago
> Can an article really be any good if it starts off with such obvious SEO spam? That's an interesting take. I fail to see how such mentions, in an article that compares two things and, thus, mentions both things together, are in any way SEO spam? For example, in an article comparing apples and oranges I would expect to see a rather high number of mentions of "apples and oranges". After all, that is the topic.
- ojosilva 5y ago
  It's SEO for the OP site. The parent comment is saying the words TF and PyTorch are repeated exhaustively, I think sometimes senselessly, throughout the entire article. It doesn't mean the content is not of value.
- edgyquant 5y ago
  That may be the topic but that first sentence reads like they are aiming to have a high number of mentions not a quality of content. For instance you can replace the second and third mentions of “PyTorch and Tensorflow” with something like “the two” or “they both.”
  - deleted 5y ago
    [deleted]
- arrakis2021 5y ago
  This is keyword stuffing, a very old SEO technique which is obsolete. In fact such behavior works against you these days.
5y ago

axegon_ 5y ago

At one point I lost interest in both and ML/AI in general. I think eventually I got frustrated with the insane amounts of abuses for marketing purposes and never truly delivering what was promised(I know, I know, fake it till you make it). For better or worse far too few managed to make it, so most stuck with faking. I think I lost interest completely around 2019. But even back then they were starting to seem like twins - practically identical with some minor but sometimes subtle differences. Looking at random sections of the documentation, all you gotta do is ignore the semantics... > tf.nn.conv2d(input, filters, strides, padding, data_format='NHWC', dilations=None, name=None) > torch.nn.functional.conv2d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1)

hwers 5y ago
Personally I wish we'd get beyond these low level abstractions by now. Machine learning is so wonderfully mathematical that making abstractions from the details should be incredibly easy and powerful. I can't believe like 8 years after the big ML wave e.g. frontend javascript developers aren't enjoying the fruits of these labors (despite there being no good reason for them not to be able to).
- dbish 5y ago
  What do you envision that would help JavaScript devs take advantage of ML? There is tensorflow.js. Are you thinking completely different ‘building blocks’ that provide higher level apis like fast.ai geared towards frontend devs or something else?
  - SleekEagle 5y ago
    They actually made ml5.js for this reason, not sure if you've seen it https://ml5js.org/ https://ml5js.org/ Pretty cool imo
- SleekEagle 5y ago
  Have you checked out TensorFlow.js? May be what you're looking for...

tadeegan 5y ago

Tensorflow is just such a classic clusterfuck google project. V2 had huge breaking changes (reminiscent of angular) and tons of the apis are clunky and don’t work well together. There are like 3 different ways to save a model. It’s almost like a bunch of teams built features with no oversight. I’m pretty sure tf is considered in maintenance mode within google as Brain and the tf creators themselves have moved to Jax. I do think Google learned a lot from tensorflow and am excited to see Jax pan out. Pytorch is a pleasure to debug. I think pytorch jit could close the deployment gap.

dbish 5y ago
Any other deployment issues for PyTorch you’re aware of that would help ‘close the gap’?
- tadeegan 5y ago
  I think PyTorch c++ api is less mature and harder to compile into other projects. Tensorflow started with the c++ api exposed which is why the graph format is so stable and favorable to deployment in heterogeneous environments.
stathibus 5y ago
The article gives credit to TF server and TFLite and so forth as being better for deployment, but leaves out the fact that those systems don't fucking work most of the time, and support is pretty much over at this point. The same goes for model support; even the models in TF's own repository are sometimes broken or don't follow the API conventions set forth in the documentation. I honestly don't know how anyone uses TF in production at this point, unless they are frozen on a specific old version and have figured out an environment that works with their specific models already.
- SleekEagle 5y ago
  Yeah, TensorFlow's API has definitely gotten convoluted and confusing. I think the shift from TF1 to TF2 and then later wrapping Keras in TF just caused a lot of problems. TensorFlow seems to be spreading itself pretty thin. Maintaining so many language bindings, TensorFlow.js, TFlite, Server, etc. seem like they could all use some focus, BUT, and this is a big but, do you think if they can get each part of their ecosystem to an easily usable point that they'll have cornered the industry sector? PyTorch is taking a much more targeted approach as seen with PyTorch Live, but I truly think that TFLite + Coral will be a game-changer for a lot of industries (and Google will make a fortune in the process). To me it seems like this is where Google's focus has lain in the AI space for the past couple of years. What do you think?

lettergram 5y ago

The thing is, tensorflow has more ability to run cross platform. I help maintain https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run distributed and federated use cases with custom layers. All of which are improved in tensorflow. That said, I’d use pytorch if I could. Simply put, it has a better user experience.

SleekEagle 5y ago
Yeah, it's pretty funny how reluctantly some people use TF because they have too, lol. The fact that PyTorch is pythonic and easier to debug makes it better for a ton of users, but TensorFlow keeps the entire DL process in mind more, not just modeling.
ojosilva 5y ago
I'm on that boat. Tensorflow.js is the only decent ML library for JS. Google support for TF.js has been dwindling, but new versions are still coming out AFAIK and we've just got Apple Silicon (M1) support.

astroalex 5y ago

No mention of Google MediaPipe (https://mediapipe.dev/ https://mediapipe.dev/), which is a mobile/edge framework for deploying TFLite models. MediaPipe has the advantage of letting you stitch together multiple models, transformations, and heuristics into a higher level computational graph. I'm not aware of any equivalent for PyTorch, although PyTorch Live seems like baby steps in that direction.

justinlloyd 5y ago
I am using mediapipe extensively in my day-to-day job. I've been impressed with it so far, and the ability to patch together multiple models in to, well... a media pipe, has been impressively useful. But I am also waiting for the penny to drop and google to announce they've abandoned it, or that their API has completely changed.
- astroalex 5y ago
  Do you work at my company? Because that's our biggest fear too. I asked a friend of mine at Google to sleuth around internally and get a sense for the health of the project. He said that it's used on some internal projects and seems to have a pretty healthy internal website. So hopefully it won't be cancelled soon.
  - justinlloyd 5y ago
    > Do you work at my company? Maybe. But I only get paid in whatever snacks I can scrounge from the employee fridge and pickings have been slim of late since so few people come into the office these days. I am down to the "expired mystery mozzarella cheese sticks" and the leftover ketchup packages from Woodranch BBQ & Grill that are in the drawer with all the unused chopsticks and plastic forks. I too spoke to a friend at Google that is part of the team, and whilst he said there were no plans to cancel it, or make radical changes, when I asked about unplanned plans, he kinda just shrugged and said "You know Google..." I have a dual solution approach, Mediapipe for "in use now" and OpenPose for validation, slower processing and the "Google just **ed us" moment we're both anticipating. I need to build my own pose analysis system, but right now I don't have the bandwidth. On the last day of Christmas the CEO sent to me: Thirty-two Manfrotto Tripod extenders Sixteen Manfrotto tripods Sixteen high speed cables Sixteen Manfrotto C-clamps Sixteen Manfrotto 3/8 to 1/4-20 reducers Sixteen Quick release mounts Sixteen 4K cameras Fooouuuurrrrr high-speeeeeed PCIe capture cards Three days to hit deadline Two triggered circuit breakers One really huge headache And a new VR H.M.D.

nmca 5y ago

This article says that Google and DeepMind research use TF - but they don't. DeepMind use JAX almost exclusively, and many brain researchers use JAX too. ML eng is my area of expertise, and I would advise strongly against tensorflow.

artdigital 5y ago
What would you recommend for new comers that want to get into the space?

cinntaile 5y ago

In TF you don't have to manually calculate the number of parameters of a fully connected layer. That's kind of nice. I don't really understand why PyTorch requires me to do that. Surely this can be infered from the previous layer.

SleekEagle 5y ago
"same" padding for convolutional layers is super convenient too
- cinntaile 5y ago
  Yeah true, that's also a bit annoying in PyTorch compared to TF. I remember reading on the PyTorch forums that this was a bit hard to implement for some reason, but I can't recall any details if there were any.
- jbschlosser 5y ago
  FYI PyTorch now supports "same" padding for convolutions with stride=1; see the docs here: https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.ht... The stride>1 case has been a bit more controversial within TensorFlow, and there is ongoing discussion on the correct way to implement it within PyTorch on the issue: https://github.com/pytorch/pytorch/issues/3867 https://github.com/pytorch/pytorch/issues/3867
  - SleekEagle 5y ago
    Thanks for letting me know!
jbschlosser 5y ago
This is supported by PyTorch as well with nn.LazyLinear: https://pytorch.org/docs/stable/generated/torch.nn.LazyLinear.html https://pytorch.org/docs/stable/generated/torch.nn.LazyLinea....
- 5y ago

strzibny 5y ago

This looks like a good comparison. A lot of comparisons online are way superficial or plain wrong.

SleekEagle 5y ago
I think the checklist of technical differences is a bit outdated at this point!

justinlloyd 5y ago

I will preface with the statement that my knowledge may be slightly out of date as I don't keep up on every nuanced change. I use PyTorch and TensorFlow, and the article is spot-on in regard to "mystery operations that take a long time" with no real rhyme or reason behind them with regard to TensorFlow. That said, on the whole, I skew more towards TensorFlow because it is generally easier to reason about the graph and how it connects. I also find the models that are available to usually be more refined, robust and useful straight out of the box. With PyTorch I am usually fighting a slew of version incompatibilities in the API between even more point releases. The models often feel more slap-dash thrown together, research like projects or toy projects, and whilst the article points out the number of papers that use PyTorch far exceeds those that use TensorFlow, and the number of models for PyTorch dwarfs that of TensorFlow, there isn't a lot of quality in the quantity. "90% of everything is crap." Theodore Sturgeon. And that goes double for PyTorch models. A lot of the models, and even some datasets, just feel like throwaway projects that someone put up online. If you are on macOS or Linux and using Python, PyTorch works fine, but don't step outside of that boundary. PyTorch and TensorFlow work with other operating systems, and other languages besides Python, but working with anything but Python when using PyTorch is a painful process fraught with pain. And yes, I expect someone to drop in and say "but what about this C++ framework?" or "I use language X with PyTorch every day and it works fine for me!" But again, the point stands, anything but Python with PyTorch is painful. The support of other languages for TensorFlow is far richer and far better. And I will preface this with, "my knowledge may be out of date" but I've also noticed the type of models and project code available for TensorFlow and PyTorch diverge wildly once you get outside of the toy projects. If you are doing computer vision, especially with video and people, and you are not working on the most simplest of pose analysis, TensorFlow offers a lot more options of stuff straight out of the box. PyTorch has some good projects and models, but they are mostly of the Mickey Mouse hobby stuff, or an abstract research project that isn't very robust or immediately deployable. I use TensorFlow in my day-to-day job. All that said, I like PyTorch for its quirkiness, its rapid prototyping, its popularity, and the fact that so many people are trying out a lot of different things, even if they don't work particularly well. I use PyTorch in almost all of my personal research projects. I expect in the future for PyTorch to get more stable and more deployable and have better tools, if it can move slightly away from the "research tool" phase it is currently in. I expect Google to do the usual Google-Fuck-Up and completely change TF for the worse, break compatibility (TF1 to TF2) or just abandon the project entirely and move on to the next new shiny.

lvl100 5y ago

I just want an alternative to CUDA. I am sick of Nvidia’s monopoly in this space.

nestorD 5y ago
I believe all big frameworks have some work being done to make them compatible with AMD GPUs. Here is the relevant issue for JAX (support seem to be in alpha but viable): https://github.com/google/jax/issues/2012 https://github.com/google/jax/issues/2012
YetAnotherNick 5y ago
The best thing that worked for me is the Apple's tensorflow pluggable device for metal. It could utilize both AMD and M1 GPU to 100% capacity. It's a shame apple could do it as a small project in the side to use metal but AMD couldn't do the same with Vulkan.

g42gregory 5y ago

As a practitioner, I feel that oftentimes you are extending, fine tuning somebody else’s code or pre-trained models (DeepMind’s for example). This means that you should be able to work on whatever the platform this code came with. Basically, you should be able to work with JAX, TF or PyTorch with equal ease.

SleekEagle 5y ago
Given that the vast majority of publications are using PyTorch, it kind of makes it the go-to

bigbillheck 5y ago

I started with tensorflow back in 2017 or so, then in my current job I am touching a lot of pytorch, and as such I say a pox on both their houses. But keras is OK and I greatly appreciate that you can (sometimes) serialize everything to hdf5.

streamofdigits 5y ago

in the fullness of time JAX might prove to be more important than either. don't give people fish, teach them how to catch fish and all that...

bjourne 5y ago

Great article. While I only had time to skim the article, I'll still offer my uninformed opinions. :) None of the hw I own is particularly great. I don't even own a decent GPU. But I don't need to because you can train your models for FREE on Google Colab's TPUs and GPUs. PyTorch's TPU support is still not that great while TensorFlow's is maturing. It's obviously a priority for Google to make TensorFlow work well on their own hardware. So for me the choice is TF 2 because I can train models 5-10x faster using Google's TPUs than if I had used PyTorch. I know the PyTorch developers are working on TPU support but last I checked (this spring) it wasn't there yet and I wasn't able to make it work well on Google Colab.

synergy20 5y ago

As a layman in ML, I thought PyTorch is more(or only) geared towards researchers while Tensorflow has its problems it is the _only_ one that provides a commercial solutions that you can deploy, is this still true? JAX is totally new to me, is this Google's new Tensorflow in the future?

johnnyApplePRNG 5y ago

I always look to Google trends when trying to decide upon which framework to use. PyTorch just recently took the lead. [0] So if I were having to choose between learning the either of them, I would go with PyTorch. [0] https://trends.google.com/trends/explore?date=all&q=tensorflow,pytorch https://trends.google.com/trends/explore?date=all&q=tensorfl...

d4rkp4ttern 5y ago

Does TF have any advantages in terms of ease of acceleration (training and inference) with multiple GPUs? Our pipeline is all PyTorch Lightning — this made development easy but we have been having numerous issues trying to leverage multiple GPUs (this is for sequence models), keep getting strange errors.