10 ms·
PyTorch 2.0
- ngcc_hk 4y agoMeta is sort of strange. They bet a lot of unusual language and invest a lot on others. And unlike google which just try this that (which also have a good point as we need innovation, just do not assume it will exist a few month/year down the road). Sad the strange taste of that thing. At least it is not as disruptive like Twitter and only hurt itself.
- marban 4y agoFrom what I see, still no 3.11 support — Same for Tensorflow which won't ship before Q1 23.
- joelfried 4y agoWho is still supporting Windows 3.11?
- sairahul82 4y agoIt’s python 3.11 :)
- robertlagrant 4y agoGuido works for Microsoft, so Python 95 will be out soon.
- fatneckbeardz 4y agoi hear that it has sockets built right in. no more Trumpet.
- sgt 4y agoSpeaking of Guido, be sure to check out the podcast with Lex Fridman. Guido is such a down to Earth guy.
- itgoon 4y agoI'm going to skip Python ME.
- minimaxir 4y agoIt's not a huge deal, as the speed improvements in 3.11 likely wouldn't trickle down at the core PyTorch level.
- black3r 4y agobut if you do some data pre-processing or post-processing in python, that would be affected by 3.11 speed improvements... or if you have a pytorch based model integrated into a bigger application as just one of many features, there are still some devs who prefer monoliths over microservices....
- quietbritishjim 4y ago> Today, we announce torch.compile, a feature that pushes PyTorch performance to new heights and starts the move for parts of PyTorch from C++ back into Python. I'll admit I don't know enough about PyTorch to know what torch.compile is exactly. But does this means some features of PyTorch will no longer be available in the core C++ library? One of the nice things about PyTorch had been that you could do your training in Python then deploy with a pure C++ application.
- microtonal 4y agoOne of the nice things about PyTorch had been that you could do your training in Python then deploy with a pure C++ application. Or even train in C++ or Rust without much loss in functionality.
- synergy20 4y agoRust really has not had any presence in AI training engine yet, it's probably 100% c++.
- microtonal 4y agoI was referring to the libtorch library, which you can use through the tch crate. It is possible to make such rich bindings because so much of Torch is exposed through the C++ API. When more new functionality is moved to Python, it makes it harder to use functionality from the C++ interface and downstream bindings.
- synergy20 4y agoFacebook did similar thing to its original code PHP, it uses HHVM to 'compile' PHP(now called Hacklang) to gain performance, it seems doing similar thing with python here.
- pjmlp 4y agoIf I remember correctly, the JIT version of the PHP compiler proved itself against the AOT compilation to native code via translation to C++, so they went with the more productive variant.
- belval 4y ago> We believe that this is a substantial new direction for PyTorch – hence we call it 2.0. torch.compile is a fully additive (and optional) feature and hence 2.0 is 100% backward compatible by definition. How about just calling it PyTorch 1.14 if it's backward compatible? Version numbering shouldn't be used as a marketing gimmick.
- posharma 4y agoIs this really the biggest problem that needs to be solved in AI?
- robertlagrant 4y agoQuite the non sequitur you have there.
- belval 4y agoNot sure I understand that question, is versioning the biggest problem no, but it costs nothing to keep semver and prevent production headaches later. If you meant inference speed then yeah it's a very big problem so it's good that they are addressing it.
- mi_lk 4y agowhat exact production headaches you are expecting by bump the number from 1.13 -> 2.0, while all existing codes keep working as before? And how is it different from bumping 1.13 to 1.14, even if they named it 1.14?
- belval 4y agoThe soft kind. Major versions are deeply ingrained as "possible backward-compatibility issues" in most engineers' brain. If you handle model development, evaluation and deployment yourself than sure you won't have any issues, but in a bigger organization you have to get people to switch and that version number will mean that everyone will ask the same "hang on this is a major version change?!" question every step of the way.
- poorman 4y agoSill no support for/from Apple Silicon?
- cube2222 4y agoIt's supported since 1.12[0], no? [0]: https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/ https://pytorch.org/blog/introducing-accelerated-pytorch-tra...
- chadykamar 4y agoIt's also officially in beta as of 1.13 https://pytorch.org/blog/PyTorch-1.13-release/#beta-support-for-m1-devices https://pytorch.org/blog/PyTorch-1.13-release/#beta-support-...
- deleted 4y ago[deleted]
- chazeon 4y agoHow is PyTorch compares to JAX and its stack?
- staunch 4y agoPyTorch and JAX are both open-source libraries for developing machine learning models, but they have some important differences. PyTorch is a more general-purpose library that provides a wide range of functionalities for developing and training machine learning models. It also has strong support for deep learning and is used by many researchers and companies in production environments. JAX, on the other hand, is designed specifically for high-performance machine learning research. It is built on top of the popular NumPy library and provides a set of tools for creating, optimizing, and executing machine learning algorithms with high performance. JAX also integrates with the popular Autograd library, which allows users to automatically differentiate functions for training machine learning models. Overall, the choice between PyTorch and JAX will depend on the specific requirements and goals of the project. PyTorch is a good choice for general-purpose machine learning development and is widely used in industry, while JAX is a better choice for high-performance research and experimentation. https://chat.openai.com/chat https://chat.openai.com/chat
- whimsicalism 4y agoI was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has the utilities and idioms to support it.
- dekhn 4y agojax is not numpy on steroids. jax is "use python idiomatically to generate optimized XLA code for evaluating functions both forward and backward."
- hintymad 4y agoA big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Another lesson is probably old but worth repeating: investment and professional polishing matters to open source projects. Meta reportedly had more than 300 (?) people working on PyTorch, helping the community resolve issues, producing tons of high-quality documentation and libraries, and marketing themselves nicely in all kinds of conferences and media.
- t-vi 4y ago> Another lesson is probably old but worth repeating: investment and professional polishing matters to open source projects While I'm not going to disagree that it matters, to my mind the main thing about early (0.x, 1.x) PyTorch's success was a clear vision (most visible to me from Soumith and Adam) that put the targeted users first (as in your productivity comment), awesome execution on it, and a creating a community where I, as an outside contributor with modes skill and no AI track record, and likely many others, felt welcome. Now, the many great people employed by Meta, but also other players like MS, NVidia, AMD, Intel, ... to work on PyTorch enabled all the things that make PyTorch 2.0 nicer and more broadly applicable than 1.0. (And to my mind it's quite a non-trivial accomplishment to enable other corporate players to join the party.)
- SleekEagle 4y agoExactly, especially in the age of ridiculously rapid development that we have found ourselves in over the past few years. This is exactly why TensorFlow is dying
- amitport 4y agoI do not believe TensorFlow is anywhere close to dying.
- la_fayette 4y ago
- ansk 4y agoSo this looks like a further convergence of the tensorflow and pytorch APIs (the lower-level APIs at least). Tensorflow was designed with compilable graphs as the primary execution model and as part of their 2.0 release, they redesigned the APIs to encompass eager execution as well. Pytorch is coming from the other end, with eager execution being the default and now emphasizing improved tools for graph compilation in their 2.0 release. The key differentiator going forward seems to be that tensorflow is using XLA as their compiler and pytorch is developing their own toolset for compilation. As someone who cares far more about performance than API ergonomics, the quality of the compiler is the main selling point for me and I'll gladly switch over to whatever framework is winning in the compiler race. Does anyone know of any benchmarks comparing the performance of pytorch's compilers with XLA?
- amelius 4y agoHow much performance can be squeezed from going from the plain python API to the graph-based solution, typically?
- ansk 4y agoThis varies quite a bit based on the type of model. The graph-based approach has two benefits: (1) removing overhead from executing python between operations and (2) enabling compilers to make optimizations based on the graph structure. The benefit from (1) is relatively modest for models which run a few large ops in series (e.g. image classifiers and most feedforward models) but can be significant for models with many ops that are smaller and not necessarily wired up sequentially (e.g. RNNs). In my experience, I've had RNN models run several times faster in tensorflow's graph mode than in its eager mode. The benefit from (2) is significant in almost any model since the typical "layer" building block (matmul/conv/einsum->bias->activation) can be fused together which improves throughput on GPUs. In my experience compilation can offer performance increases from 1.5x to 3x, but I don't know if this holds generally. Also note that the distinction between graph and eager execution can be somewhat blurry, as even an "eager" API could be calling a fused layer under the hood.
- PartiallyTyped 4y ago
- amelius 4y agoOne thing I'm noticing lately is that these DL libraries and their supporting libraries are getting unwieldy large, and difficult to version-manage. In my mind, DL is doing little more than performing some inner-products between tensors, so I'm curious why we should have libraries such as libcudnn, libcudart, libcublas, torch, etc. containing gigabytes of executable code. I just checked and I have 2.4GB (!!) of cuda-related libraries on my system, and this doesn't even include torch. Also, going to a newer minor version of e.g. libcudnn might cause your torch installation to break. Why isn't this backward compatible?
- modeless 4y agoThe complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other kernels, and again times several different GPU architectures. If you don't care about getting peak utilization of your hardware you can run state of the art neural nets with a truly tiny amount of code. The algorithms are so simple you don't even need any libraries, it's easy enough to write everything from scratch even in low level languages. It's a fun exercise. But it will be many orders of magnitude less efficient so you'll have to wait a really long time for it to run.
- claytonjy 4y agoWhile I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms. Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.
- modeless 4y agoI disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.
- singularity2001 4y agoGetting Started … and zero words on how to get started. pip3 install torch2? pip3 install torch==2.0? nope
- SekstiNi 4y agohttps://pytorch.org/get-started/pytorch-2.0/#requirements https://pytorch.org/get-started/pytorch-2.0/#requirements
- algon33 4y agoThe FAQ contains re-states the content for point 14 in point 13. Point 14 is about why your code might be slower when using 2.0. 13 should be about how to keep up with PT 2.0 developments. Someone should change that.
- singularity2001 4y agoNot available for Mac M1 yet (?)
- deleted 4y ago[deleted]
- anigbrowl 4y agoI get that Nvidia is the favorite GPU (because CUDA) and that library maintainers want to chase the latest and best to do the most. But I don't get why support for older hardware (including CUDA stuff) is just deprecated or abandoned, nor why support for other GPU architectures is lacking across many popular ML libraries. A lot of this is down to driver availability and software stacks...but is all of it? Game engines/engineers seem to be able to be productive on a wide variety of GPU hardware, why do so many ML libraries just not provide any support at all? Sure 75% of potential performance is less satisfying than 100%, but it's also infinitely better than 0%. How come every ML library doesn't at least have an OpenCL fallback or the like?
- qayxc 4y agoNVIDIA invested billions in their software infrastructure over the past 15 years (CUDA was first released in 2007) and basically brute forced their way into academia back in 2008, sponsoring labs around the world to get GPGPU going. Both mindshare and market share (outside of super computers) are just overwhelming at this point. Their market share in the consumer market is ~86% as of this quarter. The data centre market is quite fragmented when it comes to accelerators, but in AI training, NVIDIA is still the market leader. > Game engines/engineers seem to be able to be productive on a wide variety of GPU hardware, why do so many ML libraries just not provide any support at all? That's a different kettle of fish. Game engines rely on the graphics driver's implementations of low- and mid level APIs like Vulkan or Direct3D. The brunt of the work is also often performed by the middle-ware (mostly Unreal Engine and Unity or in-house engines like Frostbite) that had been in development for decades; with most games focusing on high level optimisation wrt. the middle-ware used. ML-frameworks on the other hand need to optimise compute kernels as well as data flow between host CPU and accelerator (e.g. GPU). This involves hand-tuning algorithms to best match specific GPU architectures, while most shaders used in games are basically the same across all GPU vendors and it's the vendors themselves who do the fine tuning and per-game optimisations in their graphics drivers (hence the obscene sizes of GPU drivers these days). While that's a good enough approach for games, it's simply not possible to do the same for ML-models. There's just too much flexibility (no API that dictates which calls do what, when, and how) to make general ML-optimisations at the driver level. > How come every ML library doesn't at least have an OpenCL fallback or the like? OpenCL is horrible to work with and stopped being properly supported by vendors (e.g. newer versions are rarely being implemented and optimised). The difference between CUDA and OpenCL from an implementor's perspective is that CUDA works seamlessly with surrounding C++ code and compute kernels can be embedded in the host CPU code base. OpenCL on the other hand is modelled after the ancient OpenGL 2.x paradigm and requires tedious setup and careful integration (checking capabilities and all that jazz). OpenCL is basically dead at this point. There are alternatives to CUDA, but most frameworks rely heavily on the highly optimised libraries NVIDIA ships (e.g. CuDNN) and don't have the resources to implement the functionality themselves. Some hardware vendors offer proprietary backends, like Apple or Intel and you just have to wait for them to catch up. AMD has ROCm, but that's more of a drop-in replacement that aims at running CUDA code on AMD cards.