9 ms·
PyTorch 1.10
- 6gvONxR4sf7o 5y agoI’ve largely moved to jax, but it looks like pytorch is maybe moving that direction with torch.fx? The docs on it aren’t really clear though. Has anyone used it?
- chillee 5y agoIMO, FX is more of a toolkit for writing transforms over your FX modules than "moving in Jax's direction" (although there are certainly some similarities!) It's not totally clear what "Jax's direction" means to you, but I'd consider its defining characteristics as 1. composable transformations, 2. a functional way of programming (related to its function transformations) I'd say that Pytorch is moving towards the first (see https://github.com/pytorch/functorch https://github.com/pytorch/functorch) but not the second. Disclaimer: I work on PyTorch, and Functorch more specifically, although my opinions here aren't on behalf of PyTorch.
- 6gvONxR4sf7o 5y agoThat's a great answer, thank you :) Stuff like vmap and grad and pmap and all the rest have been a huge boon in simplifying some of my work, so I'm glad to see it's expanding into pytorch!
- ersiees 5y agoThere also are new application libraries (torch vision, audio, X) being released together with 1.10: https://pytorch.org/blog/pytorch-1.10-new-library-releases/ https://pytorch.org/blog/pytorch-1.10-new-library-releases/
- singularity2001 5y agoany update on mac m1 support?
- olavgg 5y agoIt should work fine as long you're using the CPU, but it is not accelerated using the Apple’s ML Compute framework. Tensorflow supports it though.
- DonHopkins 5y agoIs M1 support something people are working on? What are the issues / difficulties / magnitude of that project?
- RavlaAlvar 5y agoIt seems everyone wants deep learning to work on M1 chips. At this point, I am quite sure there is enough interest for major frameworks to consider supporting it.
- Salgat 5y agoYou'd think Apple would be doing their best to help with this, considering they market their "neural engine" pretty heavily.
- microtonal 5y agoSmall addtion: matrix multiplications (and other operations implemented through BLAS) do use the M1's AMX matrix co-processor (through the Apple Accelerate framework).
- melling 5y agoAny recommendations on learning PyTorch. I see many people on Kaggle who use it.
- matsemann 5y agoFast.ai is basically pytorch + loads of utility functions. By following the course/book, one to some extend learns what fastai does, and how it uses pytorch for practical stuff.
- carbocation 5y agoI have also had good success with FastAI as my initial hook into PyTorch. A brief post that shows just how thin the FastAI layer can be (if you want!) is here: https://muellerzr.github.io/fastblog/2021/02/14/Pytorchtofastai.html https://muellerzr.github.io/fastblog/2021/02/14/Pytorchtofas...
- kriro 5y agoI second this but once you are comfortable with fast.ai it's a good idea to reproduce some things in pure PyTorch eventually because it'll give you a deep understanding and appreciation for fast.ai as well.
- willj 5y agoOne thing with fastai that annoyed me when I built a project with it was that the v1 and v2 APIs are totally different, and if you google or search stackoverflow for help with something, I found it more likely to stumble on answers for the v1 API than the v2 API. I also didn’t find their documentation super helpful for more than the most basic things (though not all documentation can be as amazing as scikit-learn).
- apohn 5y agoAm I allowed to have a contrarian opinion :) I went through the first version of FastAI (when it was Keras, torch?/tensorflow?) and forgot most of it because never did anything with Deep Learning. Then I did the FastAI V1 course again where they use FastAI library V2. I really liked the first version of the course because I felt like Jeremy did an awesome job of balancing understanding the guts of using a DL library with getting stuff done. It was a tough course, but I felt like I really understood things. I felt like the version with FastAI library V2 went too far into "Here are some commands you can use in the FastAI V2 library to to do this sexy thing with Deep Learning." I completed that course and really felt it should have been titled "A Course on the Fast AI V2 Library" I recently purchased "Deep Learning with PyTorch" by Eli Stevens. I've been working through this book and feel like it explains things a lot more. I'm haven't finished the book with it, but I do like it so far.
- quietbritishjim 5y agoAny recommendations about how to use a PyTorch trained model for inference? Is it best to load it up with PyTorch directly, or convert to ONNX and use ONNX-runtime [1] instead? This seems to be the required way at least if you want to TensorRT the model. I appreciate this is a very general question. [1] https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime
- vpj 5y agoHave you looked at TorchServe https://pytorch.org/serve/ https://pytorch.org/serve/
- V__ 5y agoAs far as I know, the ONNX format won't give you a performance boost on its own. However, there are ONNX optimizers for the ONNX runtime which will speed up your inference. But if you are using Nvidia Hardware, then TensorRT should give you the best performance possible, especially if you change the precision level. Don't forget to simplify your ONNX model before you converting it to TensorRT though: https://github.com/daquexian/onnx-simplifier https://github.com/daquexian/onnx-simplifier
- sailingparrot 5y ago> However, there are ONNX optimizers for the ONNX runtime I think you meant that there are optimizers for the ONNX format. ONNX Runtime being one of them.
- V__ 5y agoYou are right, thanks. Mixed those up.
- sideshowb 5y agoRelated to this question, can someone explain the design goal of torch.jit to me? Is it supposed to boost performance or just give a means to export models? I found my jitted code ran slower than interpreted pytorch, and the latter despite its asynchronous nature spent most of its time waiting for the next gpu kernel to start. Having got a working torch model on cpu, what's the best path to actually making it run as fast as I feel it has potential to?
- sandGorgon 5y ago>Distributed Training: Gloo is now supported for distributed training jobs. This is very interesting. Can someone talk about the roadmap of pytorch here ? It seems everyone is kinda rolling their own - Pytorch has a very confusing distribution story - OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH - https://pytorch.org/tutorials/beginner/dist_overview.html https://pytorch.org/tutorials/beginner/dist_overview.html - https://pytorch.org/docs/stable/distributed.elastic.html https://pytorch.org/docs/stable/distributed.elastic.html - https://pytorch.org/torchx/latest/ https://pytorch.org/torchx/latest/ - https://www.kubeflow.org/docs/components/training/pytorch/ https://www.kubeflow.org/docs/components/training/pytorch/ - Pytorch-Biggraph is specifically using torch.distributed with gloo (with an MPI backend). So here's the question - if ur a 2 person startup that wants to do Pytorch distributed training using one of the cloud-managed EKS/AKS/GKE services... what should you use ?
- orbifold 5y agoThe pytorch lightning people have come up with grid.ai, I personally have obtained good results by using pytorch lightning plus slurm on HPC machines. If I were a startup, I would probably try to build my own small HPC cluster, since that is far more cost effective than renting.
- sandGorgon 5y agoso most early stage startups get tens of thousands of dollars of free AWS credits. https://aws.amazon.com/activate/ https://aws.amazon.com/activate/ 100K if ur part of a university accelerator. it is far far more efficient (as a proportion of time-to-market) to rent and build on top of services. Kubernetes is where the wider ecosystem is. I dont like it ...but it is what it is. So Grid.ai is something like AWS Sagemaker. I wanted to figure out what someone can use on a readymade kubernetes cluster.
- another_ 5y agoCheck out Determined (https://github.com/determined-ai/determined https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc. Disclaimer: I work for Determined.
- Kalanos 5y agoWhat does FX enable? What are the use cases?
- davidatbu 5y agoCan someone please answer this? I'm so curious. The only real life use case that I've seen mentioned is "programmatically generating models, for example from a config file". But due to Python's dynamic nature, this is already possible. AllenNLP is a great example of that.
- m_ke 5y agoIt makes it possible to automate optimizing python models, adding things like conv and batch norm fusion for inference. It also allows you to plug in other ops to for example make quantization or profiling easier (see https://pytorch.org/tutorials/intermediate/fx_profiling_tutorial.html https://pytorch.org/tutorials/intermediate/fx_profiling_tuto...). Another example is a feature that was just added to torchvision which can take a classification model and extract the backbone for generating embeddings.
- davidatbu 5y agoThanks for this reply! Got some follow up questions if you don't mind being bothered ... > It makes it possible to automate optimizing python models, adding things like conv and batch norm fusion for inference. By "optimize", do you mean "reduce computational load", or "use Adam/SGD/whatever to minimize a loss function"? What is "conv and batch norm fusion"? How does FX help with any of this? > It also allows you to plug in other ops to for example make quantization or profiling easier. I can indeed see how it could make profiling easier. I'd love to get pointers/links as to quantization methods that would necessitate adding new ops. > Another example is a feature that was just added to torchvision which can take a classification model and extract the backbone for generating embeddings. Hasn't it always been possible to extract a certain set of weights from some `nn.Module`?
- 5y ago
- mkaic 5y agoThis looks great! Excited to see more pretrained models made easily accessible through Torchvision, and nn.Module parameterization seems like a really intuitive and neat way of tackling the 'parameterize the parameters' problem effectively. Kudos to the team at PyTorch, I can't wait to start playing with the new features!