28 ms·
Standardizing OpenAI’s deep learning framework on PyTorch
- syntaxing 7y agoHas anyone taken the course mentioned "Spinning Up in Deep RL"? I've been meaning to learning some Deep RL and I was wondering if this is the best first step.
- tdevito 7y agoHere's a great place to start https://developers.google.com/machine-learning/crash-course/ https://developers.google.com/machine-learning/crash-course/
- rishy 7y agoThe lecture series and reference material by David Silver: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html. You can then supplement that with "Spinning up in Deep RL" for more hands-on experiments.
- antome 7y agoAs someone who has used both PyTorch and TensorFlow for a couple years now, I can can attest to the faster research iteration times for PyTorch. TensorFlow has always felt like it was designed for some mythical researcher that could come up with a complete architecture ahead of time, based on off-the-shelf parts.
- cillaway 7y agoIndeed, no wonder PyTorch has beaten Tensorflow so thoroughly in the last 3 years, going up from 1% of the papers to ~50% of the papers (TensorFlow is now down to only 23% of the papers): https://paperswithcode.com/trends https://paperswithcode.com/trends
- nl 7y agoAccording to the methodology on that page that would classify the standalone version of Keras (using from keras.models imports as recommended by the Keras docs) as "Other". (I tried finding source code to verify this, but couldn't find it) And if that is correct, then I'd be astonished if the vast majority of the "Other" papers aren't Keras. I work in ML and I don't think I've seen a paper that didn't use PyTorch, TensorFlow or Keras in years. And is that's the case then almost certainly there are more that use TF than PyTorch: Pytorch is 42%, TF is 23% but Other is 36%. (In terms of biases, I hate working in Tensorflow, and much prefer PyTorch and Keras. But numbers are numbers).
- ma2rten 7y agoJax?
- nl 7y agoAre there any papers that use it for things other than demonstrating Jax? I can't think of one off the top of my head. Perhaps I should have specified "papers outside those introducing new frameworks, or around speed benchmarking". There are a bunch of interesting papers using custom libraries for distributed training, and ones targeted at showing off the performance of specific hardware (NVidia has a bunch of interesting work in this space, and Intel and other smaller vendors have done things too).
- alevskaya 7y agoIt's still early days for JAX, but there's neural tangents https://arxiv.org/abs/1912.02803 https://arxiv.org/abs/1912.02803 and reformer https://arxiv.org/abs/2001.04451 https://arxiv.org/abs/2001.04451 from iclr.
- nl 7y agoI agree about it being early days. Reformer is a good example that I'd missed. Neural Tangents is another paper demoing a framework.
- bitL 7y agoKeras is pretty good unless you hit some custom loss function that needs to do operations that aren't defined in Keras' backend, then you suddenly have to switch over to write them in TensorFlow with some ugly consequences (sometimes you don't know which operations will be GPU-accelerated; slicing vectors to compute and aggregate partial loss functions with some complicated math formulas might force computation onto a CPU).
- minimaxir 7y agoIt's somewhat disappointing that research is the primary motivator for the switch. PyTorch still has a ways to go in tooling for toy usage of models and deployment of models to production compared to TensorFlow (incidentally, GPT-2, the most public of OpenAI's released models, uses TensorFlow 1.X as a base). For AI newbies, I've seen people recommend PyTorch over TensorFlow just because "all the big players are using it," without listing the caveats. The future of AI research will likely be interoperability between multiple frameworks to support both needs (e.g. HuggingFace Transformers which started as PyTorch-only but now also supports TF 2.X with relative feature parity).
- Nimitz14 7y agoOpenAI doesn't have any products right? So research is a lot more important than production for them. Surprises me it took them this long.
- minimaxir 7y agoThe aforementioned GPT-2 and Spinning Up were released, open-sourced tools using TF; however, other recent models such as the OpenAI Five Dota 2 bot, the robot dexterity demo, and the hide-and-seek simulation were not open-sourced. If OpenAI is signaling that they are changing their open-source strategy, they should be more explicit about that.
- Voloskaya 7y ago> It's somewhat disappointing that research is the primary motivator for the switch. They are a research organization, how is that disappointing?
- minimaxir 7y agoMaking AI more open is synonymous with making AI more accessible, which (IMO) is much better facilitated with TensorFlow/Keras versus PyTorch. Many AI tutorials imply that the more complicated an AI approach is, the more effective it is, which isn't practical, especially for newbies without a deep background.
- sillysaurusx 7y agoThis is a surprisingly unintelligent move from OpenAI. It adds corporate inertia to something as mundane as choice of DL framework. Imagine you worked at OpenAI. Imagine you wanted to experiment with Jax, and that it turned out to be the best solution for the problem. Now you can't ship without a solid technical justification. Except, it's not really a technical justification that you need. You need corporate clout. You can't just be a junior engineer and make a decision that goes against corporate policy. That's the point of having a corporate policy. I can hear a thousand people about to type "C'mon, OpenAI isn't a normal corporation." But it is. Every corporation is a normal corporation. And having policies against specific tech should make productive programmers pause. People get jobs at companies based on whether they use React or Vue, for example. And in DL, a programming library is basically a programming language, so it's one step more powerful than that. Here's an example. Pytorch, as far as I can tell, doesn't support running code on a TPU's CPU. (I could be wrong about this!) When you enumerate the list of accelerators available after connecting to a TPU, you get a list of 8 entries. That means they only support executing code on the cores of a TPU, not the TPU's CPU. This is a huge difference. It means you're restricted to 8GB on TPUv2-8's (which you get on Colab) instead of 300GB. Does that count as a solid technical justification to use Tensorflow for a research project instead of Pytorch? Who knows. But who wants to be the odd one out on corporate politics? Especially if a project doesn't generate any tangible results, which is often the case for research.
- Erlich_Bachman 7y agoOr they see this problem and that's why the policy is sanely phrased as follows: Going forward we’ll primarily use PyTorch as our deep learning framework but sometimes use other ones when there’s a specific technical reason to do so.
- sillysaurusx 7y agoIt never works out this way in practice. You need corporate clout to go against corporate policy. That's the point of having a corporate policy. Of course they added that caveat. That's probably how this idea got through in the first place. Just point at the caveat and say "But we're not really throwing all the other frameworks under the bus. If everyone decides it's a good idea to use something else, we'll use something else." Except that likely won't happen, because now as a junior engineer you need to convince N other people that using Jax was a decent choice. And it's against your company's culture to use anything but Pytorch. This battle of Tensorflow vs Pytorch is bad for everybody involved. OpenAI released a lot of cool and important code related to Tensorflow. They did GPT-2 (tensorflow 1.x), blocksparse (also tensorflow), memory saving gradients (tensorflow 1.x), and now they're announcing they'll likely never be releasing such tooling again. Memory saving gradients have been hugely helpful to us for scaling our models beyond the normal limits.
- cs702 7y agoAt work, we switched over from TensorFlow to PyTorch when 1.0 was released, both for R&D and production... and our productivity and happiness with PyTorch noticeably, significantly improved. Back when we were using TensorFlow, whenever we wanted to try something new that wasn't already provided out-of-the-box by existing APIs, sooner or later we would find ourselves wrestling with its machinery, especially for models with more complex control flow. TensorFlow feels like it was built from the ground up to scale up to billions of users and all kinds of devices, with developer productivity and happiness a secondary priority. PyTorch feels like it was built the other way around, prioritizing developer productivity and happiness; other considerations were secondary. That said, we are keeping an eye on Swift + MLIR + TensorFlow. We think it could unseat PyTorch for R&D and eventually, production, due to (a) the promise of automatic creation of high-performance GPU/TPU kernels without hassle, (b) Swift's easy learning curve, and (c) Swift's fast performance and type safety. Jeremy Howard has a good post about this: https://www.fast.ai/2019/03/06/fastai-swift/ https://www.fast.ai/2019/03/06/fastai-swift/
- RcouF1uZ4gsC 7y agoIf you are looking for more performance and type safety versus python, PyTorch does have a C++ frontend. https://pytorch.org/cppdocs/frontend.html#end-to-end-example https://pytorch.org/cppdocs/frontend.html#end-to-end-example
- jorlow 7y ago> TensorFlow feels like it was built from the ground up to scale up to billions of users and all kinds of devices, with developer productivity and happiness a secondary priority. PyTorch feels like it was built the other way around, prioritizing developer productivity and happiness; other considerations were secondary. I recently moved from Google to Facebook and this is how I'd characterize most of the differences I see: Facebook optimizes for your ability to make progress above everything else. Google, not so much.
- kabes 7y agoSeems to hold up for most google stuff I tried. E.g. kubernetes and angular.
- sbrother 7y agoIs there any equivalent of TF Serving for PyTorch? We have been thrilled with how robust and easy it is to deploy our models to production on the TF stack, and it worries me that the inertia in the deep learning community seems to be toward PyTorch.
- calebkaiser 7y agoHave you checked out Cortex? It's an open source platform for deploying PyTorch models easily. I wrote an article for the PyTorch blog about it: https://medium.com/pytorch/how-to-build-production-software-with-pytorch-9a8725382f2a https://medium.com/pytorch/how-to-build-production-software-... GitHub: https://github.com/cortexlabs/cortex https://github.com/cortexlabs/cortex Full disclosure/shameless plug: I work on Cortex
- sbrother 7y agoThanks! I was not aware of this and it looks fantastic. Is it in the roadmap to target GCP, or even just a generic Kubernetes cluster?
- calebkaiser 7y agoGCP is on the immediate short-term roadmap, and we're investigating on-premise, but don't have a firm timeline on it quite (we're still a small team).
- smhx 7y agoThere's also fastapi, which is well-respected in Python. Someone wrapped fastapi / rabbitRPC to serve PyTorch models (with auto-batching to increase serving efficiency) in https://github.com/catalyst-team/reaction https://github.com/catalyst-team/reaction
- MiroF 7y agoYou can convert the checkpoints over for deployment. I agree the serving infra on TF is much better.
- theferalrobot 7y agoHappy to see PyTorch get some love. The company I am at made the same switch and everyone has loved PyTorch. It has more expressive power than Tensorflow 1.x (there are models that cannot be done with static graphs) and is simultaneously much easier to use.
- sandGorgon 7y agoThis is second large framework making the switch to Pytorch. https://medium.com/syncedreview/japanese-unicorn-preferred-networks-migrates-its-dl-platform-to-pytorch-a509ac8f4ba0 https://medium.com/syncedreview/japanese-unicorn-preferred-n...
- zackmorris 7y agoJust FYI I looked at PyTorch for the first time now, and unfortunately they require Mac OS users to build it from source in order to get CUDA support: https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/ Please if someone at PyTorch is reading this, put in a request to make CUDA support the default on Mac OS. Also, it looks like PyTorch doesn't currently support OpenCL: https://github.com/pytorch/pytorch/issues/488 https://github.com/pytorch/pytorch/issues/488 I can't tell by the issue comments if it's been added yet or if they plan to use Intel's oneAPI or similar. To me, these are prerequisites for switching to PyTorch. Hopefully someone can clarify the state of these thanks!
- MiroF 7y agoAre you really GPU training on your home laptop? I absolutely get why CUDA support for MacOS isn't a priority
- lrei 7y agoI would if I could - I have an external GPU at home. Unfortunately Apple is (not without reason) angry at nvidia so they dropped support for Nvidia in Mac OS. I’d have to use Windows which is a big no no for me. Obvious pytorch can’t support it.
- cepp 7y ago> Please if someone at PyTorch is reading this, put in a request to make CUDA support the default on Mac OS. It's unlikely this will ever happen. Apple doesn't officially support NVIDIA drivers anymore and even Tensorflow no longer lists MacOS as having official GPU support[0]. Don't hold your breath. [0]: https://www.tensorflow.org/install/gpu https://www.tensorflow.org/install/gpu
- smhx 7y agoHi I am a PyTorch maintainer. NVIDIA has dropped CUDA support for macOS: http://www.cgchannel.com/2019/11/nvidia-drops-macos-support-for-cuda/ http://www.cgchannel.com/2019/11/nvidia-drops-macos-support-... This was pretty evident for a few years, and it's one of the top reasons for us to not provide official binaries with CUDA support -- the maintainer overhead was way too much. We did work to make sure it still builds with CUDA support from source (with a contbuild) but once CUDA 10.3 or 11 releases, we have to drop that too.
- m0zg 7y agoIf PyTorch had a viable way to convert models to run on a mobile GPU or DSP, that's all I'd ever use. Currently I have to do my research in PyTorch and then laboriously port to TF to convert to TFLite, which kinda sucks because TF is full of bugs, and there are gotchas due to differences in how ops are implemented.
- klowrey 7y agoFor folks interested in Julia and RL, I've been involved in https://www.lyceum.ml/ https://www.lyceum.ml/ a set of tools for continuous control problems like robotics. It's pretty quick.
- julialover 7y agoYeah!! Let's switch to Lyceum!
- tastyminerals 7y agoI think it was just a matter of time till TF would get superseded by PyTorch. The only reason we kept TF on prod is the java api which allowed us to quickly load and serve TF models. I spent so many sleepless nights trying to port Torch model to TF back in the days and make it work the same as Lua based prototype. Whole TF "experience" made us switch to plain Python services model throwing away all the boilerplate Scala/Java code for TF. It doesn't happen often in tech that a better engineered product gets more traction and recognition eventually and I am glad that PyTorch did.
- lasagnaphil 7y agoPytorch actually got an experimental Java API in version 1.4 (about two weeks ago), if you're interested.
- stabbles 7y agoI've started working with Flux [1] in Julia, and it's so elegant and such a great experience :). Just look at this definition of a U-net model for image segmentation: https://gist.github.com/haampie/bceb1d59fd9a44f092f913062e58d482 https://gist.github.com/haampie/bceb1d59fd9a44f092f913062e58.... Apart from that, you can write your own custom loss functions in pure Julia that run efficiently on the GPU, language level automatic differentiation, proper integration with other packages. If people are moving away from Tensorflow, then Flux could be a solid alternative as well. [1] https://github.com/FluxML/Flux.jl https://github.com/FluxML/Flux.jl
- chillee 7y agoIMO, these kinds of functional abstractions look nice on paper, but are a pain in the ass to actually use. In practice, you'll want to print out things in between each layer, you'll want to log each layer's activations, you might want to redirect a layer into another network, etc. Both PyTorch and Tensorflow have purely functional abstractions, but they're relegated to super basic functionalities.
- eigenspace 7y agoSorry, can you clarify on what you think is a purely functional abstraction? Flux is incredibly flexible and can do all sorts of things that are not limited to purely functional code and Flux is capable of many things that are straight up impossible or infeasible in PyTorch or TensorFlow (with or without their 'purely functional' abstractions).
- chillee 7y agoSuper late reply, so it's likely you won't see this... (Too bad HN doesn't notify on replies). I'm not complaining about Flux in general, I'm talking about the specific example (the UNet) he brought up that he uses to claim that Julia is so elegant. Can you elaborate on what Flux can do that Pytorch can't?
- ChrisRackauckas 7y ago
- bitL 7y agoI believe these days one has to know both, TensorFlow (Keras) and PyTorch; most new research is in PyTorch and most deployments are in TensorFlow. Academia can afford to run on PyTorch only, stable businesses on TensorFlow only, but for individual developers they need to know both.