18 ms·
PyTorch – Tensors and Dynamic neural networks in Python
- programnature 10y agoActually not clear if there is an official affiliation with Facebook, other than some of the primary devs.
- throwawayish 10y agoCopyright (c) 2016- Facebook, Inc (Adam Paszke) Copyright (c) 2014- Facebook, Inc (Soumith Chintala) Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) Notably absent is the otherwise Facebook-typical PATENTS license thing. Which I see as a good sign. Also, it doesn't look like this has happened just now? PRs in the repo go back a couple months and the repo has 100+ contributors.
- smhx 10y agoit's the same license file as https://github.com/torch/torch7 https://github.com/torch/torch7 and http://torch.ch http://torch.ch The C libraries are shared among the Lua and Python variants
- spyspy 10y agoThis project aside, I'm in love with that setup UI on the homepage telling you exactly how to get started given your current setup.
- demonshalo 10y agoyes indeed!
- artursapek 10y agoAgreed. Reminds me of this scary page I found the other day when googling "certbot setup": https://certbot.eff.org/all-instructions/ https://certbot.eff.org/all-instructions/
- jmportilla 10y agoyeah, that's a great way to quickly show the set up and OS requirements
- aaron-lebo 10y agoIs this related to lua's Torch at all? http://torch.ch/ http://torch.ch/
- superdaniel 10y agoI was wondering the same thing. There's even another repo that seems mildly popular called pytorch on Github: https://github.com/hughperkins/pytorch https://github.com/hughperkins/pytorch
- zo7 10y agoThey don't seem to explicitly say it, but it might be using the same core code given the structure of the framework and their mentioning that it's a mature codebase several years old. The license file also goes back to NYU before being taken over by Facebook, similar to Torch.
- pavanky 10y agoThey share the same underlying C and CUDA libraries. The Python and Lua modules are different. You can see both projects have pretty much the same contributors because they are sharing the code base using git-subtree.
- ankitml 10y agoI am confused with the license file. What does it mean? Some rights reserved and copyright... Doesnt look like a real open source project.
- s_ngularity 10y agoLooks like standard BSD-3 to me
- deleted 10y ago[deleted]
- wyldfire 10y agoCopyrights are appropriate, they make it explicit who produced the work and make it easier to enforce the license. This license follows the general form of the BSD-style license [1]. [1] https://opensource.org/licenses/BSD-3-Clause https://opensource.org/licenses/BSD-3-Clause
- yincrash 10y agoIt is a standard 3-clause BSD license. The "All rights reserved" portion definitely adds ambiguity (and only exists in the BSD license out of all major OSS licenses). There is StackExchange answer that goes into the history of it[1]. [1] http://opensource.stackexchange.com/questions/2121/mit-license-and-all-rights-reserved http://opensource.stackexchange.com/questions/2121/mit-licen...
- ankitml 10y agoGot it. It makes sense now.
- smhx 10y agoIt's a community-driven project, a Python take of Torch http://torch.ch/ http://torch.ch/. Several folks involved in development and use so far (a non-exhaustive list): * Facebook * Twitter * NVIDIA * SalesForce * ParisTech * CMU * Digital Reasoning * INRIA * ENS The maintainers work at Facebook AI Research
- tsomctl 10y agoNot only that, but it appears to use the same core c libray (TH) as Lua torch.
- smhx 10y agowe actually share the same git-subtree between Lua and Python variants. TH, THNN, THC, THCUNN are shared.
- divbit 10y agoI have been running in the back of my mind the idea of attempting a julialang interface to torch for a few weeks now, using the ccall interface: http://docs.julialang.org/en/release-0.5/manual/calling-c-and-fortran-code/?highlight=ccall http://docs.julialang.org/en/release-0.5/manual/calling-c-an.... Do you have any thoughts / recommendations w'r't' that? (This would be more of a fun / weekend(s) project for me than anything else) My goal would be to have the tensors override the .* and * operators as used here: https://gist.github.com/divbit/ec57ad2f1989bf13aecdf9e1e10563f0 https://gist.github.com/divbit/ec57ad2f1989bf13aecdf9e1e1056...
- rtcoms 10y agoI've never fiddled with machine learning thing so don't know anything about it. I am wondering if CUDA is mandatory for torch installation ? I use a Macbook air which doesn't have graphics card, so not sure if torch can be installed and used on my machine.
- RangerScience 10y agoI believe that's a "no". I was able to set up a docker'ized Deep Style on my Macbook Pro, although it takes for bloody ever to do a single image. CUDA is, AFAIK, a substantial speed boost, but not a requirement.
- itg 10y agoIt's not mandatary, but for some problems such as using image data, it provides as substantial speedup when training a classifier.
- zitterbewegung 10y agoYou could probably train MNIST on your Macbook Air but anything much more complicated for that you would want to use A GPU.
- tdees40 10y agoAt this point I've used PyTorch, Tensorflow and Theano. Which one do people prefer? I haven't done a ton of benchmarking, but I'm not seeing huge differences in speed (mostly executing on the GPU).
- sandGorgon 10y agoKeras is going to be the interface to Tensorflow - https://news.ycombinator.com/item?id=13413487 https://news.ycombinator.com/item?id=13413487
- tdees40 10y agoYes, but Keras works just fine using Theano as a backend as well...
- shmatt 10y agoi've been running their dcgan.torch code in the past few days and results have been pretty amazing for plug and play
- taterbase 10y agoIs there any reason this might not work in windows? I see no installation docs for it.
- smhx 10y agothe C libraries are compatible with Windows, they are used in Torch windows ports. We just dont have any Windows devs on the project to help and maintain it :( .
- randomx89 10y agoAre you guys looking for Windows devs to contribute or help maintaining it? I'd be interested in helping out if I can. I currently use Chainer, but I'd like to try pytorch
- apaszke 10y agoYes! There's an issue on that, where we'll be coordinating the work: https://github.com/pytorch/pytorch/issues/494 https://github.com/pytorch/pytorch/issues/494
- vegabook 10y agoGuess there's no escaping Python. I had hoped Lua(jit) might emerge as a scientific programming alternative but with Torch now throwing its hat into the Python ring I sense a monoculture in the making. Bit of a shame really because Lua is a nice language and was an interesting alternative.
- jjawssd 10y agoLua is extremely flexible to the point where there is basically no standard library. This causes problems with code reuse and moving between codebases because everyone does things drastically differently. Compare this to Numpy in the Python world, a single fundamental package for scientific computing in Python. Lua is less used than Python in the scientific community, and a lot of the most innovative machine learning researchers already work with C++ and Python. Using yet another language with only marginal benefit increases cognitive load and drains from the researcher's mental innovation budget, forcing the researcher to learn the ins and outs of Lua rather than working on innovative machine learning solutions. Lua is a nice language. Python 3 is a nice language and there are many new exciting features and development styles (hello async programming?) in the making which will prevent a monoculture from forming in the near term.
- vegabook 10y agoThanks for the interesting and informative comment. Do I sense just a tiny bit of regret though? Yet another Python interface. YAPI. You heard it here first. And no, Py3 is not that nice. Too much cruft by far. And lua is miles faster than Python when you're outside the tensor domain, ie while you're sourcing and wrangling your data. Arguably luajit obviates the need for C , something you can't say about Python. Disclosure: I am a massive, but increasingly disenchanted, user of Python. I had actually started looking at Torch7, foregoing tensorflow, precisely because of Lua. But the walls are closing in....
- jjawssd 10y agoA very large portion of performance problems can be mitigated with the use of cython and the new asyncio stuff. asyncio success story: https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/ https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-pytho... cython: http://scikit-learn.org/stable/developers/performance.html http://scikit-learn.org/stable/developers/performance.html
- Smerity 10y agoOnly a few months ago people saying that the deep learning library ecosystem was starting to stabilize. I never saw that as the case. The latest frontier for deep learning libraries is ensuring efficient support for dynamic computation graphs. Dynamic computation graphs arise whenever the amount of work that needs to be done is variable. This may be when we're processing text, one example being a few words while another being paragraphs of text, or when we are performing operations against a tree structure of variable size. This problem is particularly prominent in particular subfields, such as natural language processing, where I spend most of my time. PyTorch tackles this very well, as do Chainer[1] and DyNet[2]. Indeed, PyTorch construction was directly informed from Chainer[3], though re-architected and designed to be even faster still. I have seen all of these receive renewed interest in recent months, particularly amongst many researchers performing cutting edge research in the domain. When you're working with new architectures, you want the most flexibility possible, and these frameworks allow for that. As a counterpoint, TensorFlow does not handle these dynamic graph cases well at all. There are some primitive dynamic constructs but they're not flexible and usually quite limiting. In the near future there are plans to allow TensorFlow to become more dynamic, but adding it in after the fact is going to be a challenge, especially to do efficiently. Disclosure: My team at Salesforce Research use Chainer extensively and my colleague James Bradbury was a contributor to PyTorch whilst it was in stealth mode. We're planning to transition from Chainer to PyTorch for future work. [1]: http://chainer.org/ http://chainer.org/ [2]: https://github.com/clab/dynet https://github.com/clab/dynet [3]: https://twitter.com/jekbradbury/status/821786330459836416 https://twitter.com/jekbradbury/status/821786330459836416
- PieSquared 10y agoCould you elaborate on what you find lacking in TensorFlow? I regularly use TensorFlow for exactly these sorts of dynamic graphs, and it seems to work fairly well; I haven't used Chainer or DyNet extensively, so I'm curious to see what I'm missing!
- Smerity 10y agoWhen you say "exactly these sorts of dynamic graphs", what do you mean? TensorFlow has support for dynamic length RNN unrolling but that really doesn't extend well to any dynamic graph structure such as recursive tree structure creation. Since the computation graph has a different shape and size for every input they are difficult to batch and any pre-defined static graph is likely excessive, wasting computation, or inexpressive. The primary issue is that the computation graph is not imperative - you define it explicitly. Chainer describes this as the difference between "Define-and-Run" frameworks and "Define-by-Run" frameworks[1]. TensorFlow is "Define-and-Run". For loops and conditionals end up needing to be defined and injected into the graph structure before it's run. This means there are "tf.while_loop" operations for example - you can't use a "while" loop as it exists in Python or C++. This makes debugging difficult as the process of defining the computation graph is separate to the usage of it and also restricts the flexibility of the model. In comparison, both Chainer, PyTorch, and DyNet are "Define-by-Run", meaning the graph structure is defined on-the-fly via the actual forward computation. This is a far more natural style of programming. If you perform a for loop in Python, you're actually performing a for loop in the graph structure as well. This has been a large enough issue that, very recently, a team at Google created "TensorFlow Fold"[2], still unreleased and unpublished, that handles dynamic computation graphs. In it they tackle specifically dynamic batching within the tree structured LSTM architecture. If you compare the best example of recursive neural networks in TensorFlow[3] (quite complex and finicky in the details) to the example that comes with Chainer[4], which is perfectly Pythonic and standard code, it's pretty clear why one might prefer "Define-by-Run" ;) [1]: http://docs.chainer.org/en/stable/tutorial/basic.html http://docs.chainer.org/en/stable/tutorial/basic.html [2]: https://openreview.net/pdf?id=ryrGawqex https://openreview.net/pdf?id=ryrGawqex [3]: https://github.com/bogatyy/cs224d/tree/master/assignment3 https://github.com/bogatyy/cs224d/tree/master/assignment3 [4]: https://github.com/pfnet/chainer/blob/master/examples/sentiment/train_sentiment.py#L125 https://github.com/pfnet/chainer/blob/master/examples/sentim...
- theoracle101 10y agoMost important question. Is this still 1 indexed (Lua was 1 indexed, which means porting code you need to be aware of this)?
- plg 10y agoEvery time I decide I'm going to get into Python frameworks again, and I start looking at code, and I see people making everything object-oriented, I bail Just a personal (anti-)preference I guess
- closed 10y agoSame. I know there can be nice, composable OO approaches, but every time I bump into a super crazy stacktrace, or need one of those police-detective-style boards with yarn to connect everything, I start to wonder.
- apaszke 10y agoBut it is possible to write your model in purely functional style. Check out the PR to examples repo with functional ResNets https://github.com/pytorch/examples/pull/22 https://github.com/pytorch/examples/pull/22.
- gallerdude 10y agoWhat's the highest level neural network lib I can use? I'm a total programming idiot but I find neural nets fascinating.
- nickdavidhaynes 10y agohttp://playground.tensorflow.org/ http://playground.tensorflow.org/ Pretty much no way to use neural networks (except for playing, like above) without writing code.
- eudoxus 10y agoThis[1] was posted earlier today to HN. Seems pretty simple to play with NNs without coding. [1] - http://kur.deepgram.com/ http://kur.deepgram.com/
- visarga 10y agoKeras requires just a few lines of code, it's designed for easy use and practicality.
- apaszke 10y agotorch.nn offers a very similar interface to Keras (e.g. see Alexnet definition at https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py#L13 https://github.com/pytorch/vision/blob/master/torchvision/mo...).
- baq 10y agoVery nice to see Python 3.5 there.
- EternalData 10y agoBeen using PyTorch for a few things. Love how it integrates with Numpy.
- 0mp 10y agoIt is worth adding that there is a wip branch focused on making PyTorch tensors distributable across machines in a master-workers model: https://github.com/apaszke/pytorch-dist/ https://github.com/apaszke/pytorch-dist/
- jbsimpson 10y agoThis is really interesting, I've been wanting to learn more about Torch for a while but have been reluctant to commit to learning Lua.
- veli_joza 10y agoLua is a pleasure to learn and use. The language core is so simple and elegant, you can learn it in a day. Standard library is also very light, which is both strength and weakness. I use it more and more for hobby projects. Combine it with LuaJIT (which torch uses) and you have the fastest interpreted language around. Give it a try.
- etiene 10y agoI want to reiterate this. I started learning it for guilt because it was created in the university I studied. Then I realised it was really a pleasure to use it. I still use it in many hobby projects nowadays whenever I can.