10 ms·

Leaving Meta and PyTorch

721 points by saikatsg 1y ago

msmd74 1y ago
Sounds like you had a momentous run. If you take advice from reformed Internet trolls, consider turning off all your devices and trying to give yourself at least a week, but ideally a month offline staring at your new baby. You'll never get that time back and there's nothing your brain will appreciate more than loading up those memories as they grow. Good luck.
qmatch 1y ago
As a loyal JAX user, I hope they can play catchup. PyTorch has dominated the AI scene since TF1 fumbled the ball at 10th yard line. What Matt Johnson has done turning Autograd into JAX is hopefully going to be worthy of as much praise as what Soumith has received.
- n_u 1y ago
  > PyTorch has dominated the AI scene since TF1 fumbled the ball at 10th yard line can you explain why you think TensorFlow fumbled?
  - zapnuk 1y ago
    For me it was about 8 years ago. Back then TF was already bloated but had two weaknesses. Their bet on static compute graphs made writing code verbose and debugging difficult. The few people I know back then used keras instead. I switched to PyTorch for my next project which was more "batteries included".
  - michaelt 1y ago
    Imagine a total newbie trying to fine-tune an image classifier, reusing some open source example code, about a decade ago. If their folder of 10,000 labelled images contains one image that's a different size to the others, the training job will fail with an error about unexpected dimensions while concatenating. But it won't be able to say the file's name, or that the problem is an input image of the wrong size. It'll just say it can't concatenate tensors of different sizes. An experienced user will recognise the error immediately, and will have run a data cleansing script beforehand anyway. But it's not experienced users who bounce from frameworks, it's newbies.
    - mschuster91 1y ago
      > An experienced user will recognise the error immediately, and will have run a data cleansing script beforehand anyway. But it's not experienced users who bounce from frameworks, it's newbies. Even seasoned developers will bounce away from frameworks or libraries - no matter if old dogs or the next hot thing - if the documentation isn't up to speed or simple, common tasks require wading through dozens of pages of documentation. Writing good documentation is hard enough, writing relevant "common usage examples" is even harder... but keeping them up to date and working is a rarely seen art. And the greatest art of all of it is logging. Soooo many libraries refuse to implement detailed structured logging in internal classes (despite particularly Java and PHP offering very powerful mechanisms), making it much more difficult to troubleshoot problems in the field.
  - qmatch 1y ago
    I personally believe TF1 was serving the need of its core users. It provided a compileable compute graph with autodiff, and you got very efficient training and inference from it. There was a steep learning curve, but if you got past it, things worked very very well. The distributed TF never really took off—it was buggy, and I think they made some wrong earlier bets in the design for performance reasons that they should have been sacrificed in favor of simplicity. I believe some years after the TF1 release, they realized the learning curve was too steep, they were losing users to PyTorch. I think also the Cloud team was attempting to sell customers on their amazing DL tech, which was falling flat. So they tried to keep the TF brand while totally changing the product under the hood by introducing imperative programming and gradient tapes. They killed TF1, upsetting those users, while not having a fully functioning TF2, all the while having plenty of documentation pointing to TF1 references that didn’t work. Any new grad student made the simple choice of using a tool that was user-friendly and worked, which was PyTorch. And most old TF1 users hopped on the band wagon.
  - tdullien 1y ago
    I only remember 2015 TF and I was wondering: why would I use Python to assemble a computational graph when what I really want is to write code and then differentiate through it?
  - Gazoche 1y ago
    I'm no machine learning engineer but I've dabbled professionally with both frameworks a few years ago and the developer experience didn't even compare. The main issue with TF was that you could only chose between a powerful but incomprehensible, poorly documented [1], ultra-verbose and ever changing low-level API, and an abstraction layer (Keras) that was too high level to be really useful. Maybe TF has gotten better since but at the time it really felt like an internal tool that Google decided to just throw into the wild. By contrast PyTorch offered a more reasonable level of abstraction along with excellent API documentation and tutorials, so it's no wonder that machine learning engineers (who are generally more interested in the science of the model than the technical implementation) ended up favoring it. [1] The worst part was that Google only hosted the docs for the latest version of TF, so if you were stuck on an older version (because, oh I don't know, you wanted a stable environment to serve models in production), well tough luck. That certainly didn't gain TF any favors.
  - HarHarVeryFunny 1y ago
    The original TensorFlow had an API similar to the original Lua-based Torch (the predecessor to PyTorch) that required you to first build the network, node by node, then run it. PyTorch used a completely different, and much more convenient approach, where the network is built automatically for you just by running the forward pass code (and will then be used for the backward pass), using both provided node types and arbitrary NumPy compatible code. You're basically just writing differentiable code. This new PyTorch approach was eventually supported by TensorFlow as well ("immediate mode"), but the PyTorch approach was such a huge improvement that there had been an immediate shift by many developers from TF to PyTorch, and TF never seemed able to regain the momentum. TF also suffered from having a confusing array of alternate user libraries built on top of the core framework, none of which had great documentation, while PyTorch had a more focused approach and fantastic online support from the developer team.
    - liuliu 1y ago
      LuaTorch is eager-execution. The problem with LuaTorch is the GC. You cannot rely on traditional GC for good work, since each tensor is megabytes (at the time), now gigabytes large, you need to collect them aggressively rather than at intervals (Python's reference-counting system solves this issue, and of course, by "collecting", I don't mean free the memory (PyTorch has a simple slab allocator to manage CUDA memory)).
      - HarHarVeryFunny 1y ago
        With Lua Torch the model execution was eager, but you still had to construct the model graph beforehand - it wasn't "define by run" like PyTorch. Back in the day, having completed Andrew Ng's ML coursew, I then built my own C++ NN framework copying this graph-mode Lua Torch API. One of the nice things about explicitly building a graph was that my framework supported having the model generate a GraphViz DOT representation of itself so I could visualize it.
        liuliu 1y ago
        Ah, I get what you mean now. I am mixing up the nn module and the tensor execution bits. (to be fair, the PyTorch nn module carries over many these quirks!).
  - stared 1y ago
    In 2018, I co-wrote a blog post with the inflammatory title “Don’t use TensorFlow, try PyTorch instead” (https://news.ycombinator.com/item?id=17415321 https://news.ycombinator.com/item?id=17415321). As it gained traction here, it was changed to “Keras vs PyTorch” (some edgy things that work for a private blog are not good for a corporate one). Yet the initial title stuck, and you can see it resonated well with the crowd. TensorFlow (while a huge step on top of Theano) had issues with a strange API, mixing needlessly complex parts (even for the simplest layers) with magic-box-like optimization. There was Keras, which I liked and used before it was cool (when it still supported the Theano backend), and it was the right decision for TF to incorporate it as the default API. But it was 1–2 years too late. At the same time, I initially looked at PyTorch as some intern’s summer project porting from Lua to Python. I expected an imitation of the original Torch. Yet the more it developed, the better it was, with (at least to my mind) the perfect level of abstraction. On the one hand, you can easily add two tensors, as if it were NumPy (and print its values in Python, which was impossible with TF at that time). On the other hand, you can wrap anything (from just a simple operation to a huge network) in an nn.Module. So it offered this natural hierarchical approach to deep learning. It offered building blocks that can be easily created, composed, debugged, and reused. It offered a natural way of picking the abstraction level you want to work with, so it worked well for industry and experimentation with novel architectures. So, while in 2016–2017 I was using Keras as the go-to for deep learning (https://p.migdal.pl/blog/2017/04/teaching-deep-learning/ https://p.migdal.pl/blog/2017/04/teaching-deep-learning/), in 2018 I saw the light of PyTorch and didn’t feel a need to look back. In 2019, even for the intro, I used PyTorch (https://github.com/stared/thinking-in-tensors-writing-in-pytorch https://github.com/stared/thinking-in-tensors-writing-in-pyt...).
    - stared 1y ago
      Actually, I opened “Teaching deep learning” and smiled as I saw how it evolved: > There is a handful of popular deep learning libraries, including TensorFlow, Theano, Torch and Caffe. Each of them has Python interface (now also for Torch: PyTorch) > [...] > EDIT (July 2017): If you want a low-level framework, PyTorch may be the best way to start. It combines relatively brief and readable code (almost like Keras) but at the same time gives low-level access to all features (actually, more than TensorFlow). > EDIT (June 2018): In Keras or PyTorch as your first deep learning framework I discuss pros and cons of starting learning deep learning with each of them.
  - probably_wrong 1y ago
    I see good answers already, but here's a concrete example: In my University we had to decide between both libraries so, as a test, we decided to write a language model from scratch. The first minor problem with TF was that (if memory serves me right) you were supposed to declare your network "backwards" - instead of saying "A -> B -> C" you had to declare "C(B(A))". The major problem, however, was that there was no way to add debug messages - either your network worked or it didn't. To make matters worse, the "official" TF tutorial on how to write a Seq2Seq model didn't compile because the library had changed but the bug reports for that were met for years with "we are changing the API so we'll fix the example once we're done". PyTorch, by comparison, had the advantage of a Python-based interface - you simply defined classes like you always did (including debug statements!), connected them as variables, and that was that. So when I and my beginner colleagues had to decide which library to pick, "the one that's not a nightmare to debug" sounded much better than "the one that's more efficient if you have several billions training datapoints and a cluster". Me and my colleagues then went on to become professionals, and we all brought PyTorch with us.
    - jszymborski 1y ago
      The inability to use print debug to tell me the dimensions of my hidden states was 100% why TF was hard for me to use as a greenhorn MSc student. Another consequence of this was that PyTorch let you use regular old Python for logic flow.
    - n_u 1y ago
      This was also my experience. TensorFlow's model of constructing then evaluating a computation graph felt at odds with Python's principles. It made it extremely difficult to debug because you couldn't print tensors easily! It didn't feel like Python at all. Also the API changed constantly so examples from docs or open source repos wouldn't work. They also had that weird thing about all tensors having a unique global name. I remember I tried to evaluate a DQN network twice in the same script and it errored because of that. It's somewhat vindicating to see many people in this thread shared my frustrations. Considering the impact of these technologies I think a documentary about why TensorFlow failed and PyTorch took off would be a great watch.
  - morshu9001 1y ago
    I just remember TF1 being super hard to use as a beginner and Google repeatedly insisting it had to be that way. People talk about the layering API, but it's more than that, everything about it was covered with sharp edges.
  - htrp 1y ago
    Greenfielding TF2.X and not maintaining 1.X compatibility
  - rockinghigh 1y ago
    First, the migration to 2.0 in 20219 to add eager mode support was horribly painful. Then, starting around 2.7, backward compatibility kept being broken. Not being able to load previously trained models with a new version of the library is wildly painful.
- intermerda 1y ago
  Do you have experience in both JAX and PyTorch? Why do you prefer JAX?
  - cl3misch 1y ago
    Not OP. I prefer JAX for non-AI tasks in scientific computing because of the different mental model than PyTorch. In JAX, you think about functions and gradients of functions. In PyTorch you think about tensors which accumulate a gradient while being manipulated through functions. JAX just suits my way of thinking much better. I also like that jax.jit forces you to write "functional" functions free of side effects or inplace array updates. It might feel weird at first (and not every algorithm is suited for this style) but ultimately it leads to clearer and faster code. I am surprised that JIT in PyTorch gets so little attention. Maybe it's less impactful for PyTorch's usual usecase of large networks, as opposed to general scientific computing?
    - imtringued 1y ago
      >I also like that jax.jit forces you to write "functional" functions free of side effects or inplace array updates. It might feel weird at first (and not every algorithm is suited for this style) but ultimately it leads to clearer and faster code. It's not weird. It's actually the most natural way of doing things for me. You just write down your math equations as JAX and you're done.
      - Majromax 1y ago
        > You just write down your math equations as JAX and you're done. It's natural when your basic unit is a whole vector (tensor), manipulated by some linear algebra expression. It's less natural if your basic unit is an element of a vector. If you're solving sudoku, for example, the obvious 'update' is in-place. In-place updates are also often the right answer for performance reasons, such as writing the output of a .map() operation directly to the destination tensor. Jax leans heavily on compile-time optimizations to turn the mathematically-nice code into computer-nice code, so the delta between eager-Jax and compiled-Jax is much larger than the delta between eager-Pytorch and compiled-Pytorch.
- bjourne 1y ago
  JAX seems great but the Google ghost is a lot to stomach. The risk of JAX getting axed or replaced with a JAX 2.0 - completely incompatible with existing JAX code - is not insignificant.
chopete3 1y ago
>>Every major AI company and hardware vendor are on a speed dial. This kind of power is really hard to give up. But curiosity ultimately won out in my head. A simple feeling has such a power. May he gets an opportunity to create one more powerful tool before retiring.
- Lord-Jobo 1y ago
  If the curiosity dies, the entire thing crumbles. The second I stop being curious I stop finding new and exciting things to do, and I stop feeling fulfillment. It’s one of the biggest signs saying “it’s time to move on”. I feel so strongly for the people who can’t afford the luxury. Ive been there, unfulfilling jobs for years because bills or resumè building.
  - rubicon33 1y ago
    Gosh, given you’ve been there I have to ask what allowed you to get out of that and pursue only things that interest and excite you?
    - Lord-Jobo 11mo ago
      80% lucky, 10% low standards, 10% hard work. In short I went from a small company position with very little growth and constant paycheck delays to a job with very clear monthly metric goals and a manager that lets me pursue projects for our office and client that actually help and improve everyone’s experience. My field is really small and most hiring managers/HR people frankly have no clue how to hire for it, so I have a genuinely great resume and still got ghosted for many positions I was easily qualified for. So hopping from a low paying position I was very tired with, to a job with daily activities I actually enjoy, that pays way more appropriately, was a several year process when it would be a 2 month process in a bigger field. Realistically if I had to attribute anything in particular it’s these two things: 1: knowing what kind of work would actually fulfill me, which turned out to be fairly repetitive grunt work with 10% special projects. 2: learning our primary software VERY well. I have not met any other analyst who knows the quirks and tricks as well as I do, but that mostly just comes from 8 years of struggling with it and their nonexistent customer support. The low standards are important for this as well because I am making 15% over the median personal income, and many people on here have ambitions WAY higher than that. But my partner makes double what I do, we have no kids, and we have cheap rent for our area so I really am thankful for what we have. Pursuing what actually interests me is easy as well because I’m happy with a job that’s 90% assembly line work and 10% actually novel project work.
perfmode 1y ago
Respect.
mxkopy 1y ago
PyTorch is one of those tools that’s so simple and easy to take apart that you feel like you might’ve been able to make it yourself. I can’t imagine how much engineering effort was behind all those moments where I thought to myself, “of course it should work like that, how can it be any other way?”
- TechnicolorByte 1y ago
  Can anyone recommend a technical overview describing the design decisions PyTorch made that led it to win out?
  - huevosabio 1y ago
    I don't know the full list, but back when it came out, TF felt like a crude set of bindings to the underlying c++/CUDA workhorse. PyTorch felt, in contrast, pythonic. It was much closer in feeling to numpy.
  - puttycat 1y ago
    I think it was mostly the eager evaluation that made it possible to debug every step in the network forward/backward passes. Tensorflow didn't have that at the time which made debugging practically impossible.
  - GistNoesis 1y ago
    The choice of the dynamic computation graph [1] of PyTorch made it easier to debug and implement, leading to higher adoption, even though running speed was initially slower (and therefore training cost higher). Other decisions follow from this one. Tensorflow started with static and had to move to dynamic at version 2.0, which broke everything. Fragmentation between tensorflow 1, tensorflow 2, keras, jax. Pytorch's compilation of this computation graph erased the remaining edge of Tensorflow. Is the battle over ? From a purely computational point, Pytorch solution is very far from optimal and billions of dollars of electricity and GPUs are burned every year, but major players are happy with circular deals to entrench their positions. So at the pace of current AI code development, probably one or two years before Pytorch is old history. [1] https://www.geeksforgeeks.org/deep-learning/dynamic-vs-static-computational-graphs-pytorch-and-tensorflow/ https://www.geeksforgeeks.org/deep-learning/dynamic-vs-stati...
BoredPositron 1y ago
The last few years must have been incredibly exhausting. Thanks for your work good luck and 73.
vintermann 1y ago
That man has an infective enthusiasm. I remember the DCGAN paper inspired me to try getting the (Lua) Torch code to work, and I tried it on the Oxford flowers dataset early on. It worked surprisingly well, and Soumith Chintala even shared it around in social media, surprised at how well it worked on such a small dataset. Of course back then we didn't really appreciate the problem of mode collapse. Pytorch and old Lua Torch were a pleasure to work with compared to the contemporary Tensorflow. Lots of S.C's code was copied around liberally, it had its quirks (I remember the DCGAN code had a pretty odd way of doing parameter passing) but it was also really easy to understand and made random people like me feel like we had suddenly stumbled onto something crazy powerful (which we had!). It was wonderfully hackable.
- whitten 1y ago
  What is the Oxford flowers dataset ? Where is it available ?
  - rockinghigh 1y ago
    https://www.robots.ox.ac.uk/~vgg/data/flowers/ https://www.robots.ox.ac.uk/~vgg/data/flowers/
zkmon 1y ago
[flagged]
theusus 1y ago
[flagged]
utopiah 1y ago
What I find most interesting with this is that it shows they believe there is nothing unique at Meta related to AI. There is no resource, people and computing power, that they can't get elsewhere for whatever they believe would be more interesting for them. I mention this because it feels analogous to military research, where people "dream" of how advanced the military is, how forward they are compared to public research... and yet, it seems to be a recurring myth they love to sustain. So the signal I get here is AI "labs" in BigTech have nothing worth waiting for around the corner, it's just more of the same and boring for people who stick there.
- rtpg 1y ago
  I don't think that's the read? Guy says he wants to work on something small. If you want to work on something big you probably want to be in a big corp to have the resources to do the big thing. Also absolutely unknown if the "new thing" is AI-related at all!
  - utopiah 1y ago
    Well he left so whatever is coming next, AI related or not, "small" or not (small for them might be reaching just a million people, he wrote that he "lead the software layer that powers the entire AI industry." so his notion of scale is probably unlike mine, maybe yours too) is more exciting to him that whatever he could do next with all of Meta's resources. Edit: to be clear, I didn't mean to imply their next thing is AI related, solely that they obviously know more about AI at Meta than e.g. XR at Meta, just because that's their expertise.
    - hombre_fatal 1y ago
      Your assumption is a bad read because it only works if his set of life priorities contains nothing else but maximizing his impact in the world of AI. If he has just one other priority in that set (which could still include a robotic min/max of AI impact), then your assumption fails.
  - radicalbyte 1y ago
aabhay 1y ago
For anyone that’s curious, the underlying Torch library is also a joy to work with, as are the many other torch bindings. For example, Rust has tch and Burn which both work with libtorch. PyTorch of course has the benefit of being dynamically debuggable. Can’t forget the first time I break pointed my pytorch model and wrote pytorch calls inside the terminal to inspect the behavior. That’s still something I miss a lot now that I’m working with only “fast” compiled code.
- lysecret 1y ago
  I wrote som truly awful code back in the day because of that but god it was glorious.
- deleted 1y ago
  [deleted]
irthomasthomas 1y ago
Counterfactual Regret Minimization irl
numice 1y ago
I read one post on his blog and found that Adam Paszke reached out to the author and got an internship. I wonder if it was that easy to get an internship at FAIR. I thought that they hire only PhDs.
- vintermann 1y ago
  I didn't know that. Soumith Chintala certainly paid it forward. He was very helpful and supportive of random strangers (like me!) in the early pytorch days. I count him with Andrej Karpathy and Chris Olah as one of the people who really made machine learning accessible to regular software engineers.
  - SpaceManNabs 1y ago
    Chris Olah is the goat. I reached out to him myself years ago and was surprised at getting a response. And the response was incredibly generous. I probably wouldn't have had the confidence to do my switch if it wasn't for Olah. And as I got further into this path, I learned that Olah had done the same for some of my mentors and colleagues. Every time Olah speaks, I listen.
- chrneu 1y ago
  You can't do anything if you never try.
- nswizzle31 1y ago
  I was pretty involved in the PyTorch ecosystem in the early days around 2016 and Adam was nothing short of a genius and prolific developer whose contributions to the codebase and community were immense. I think he was like an undergrad in Poland at the time. My understanding is that his contributions came before the internship, but I don’t know. My memory is that Souminth was really open to other people’s contributions and questions, no matter their credentials. He was a great leader who felt approachable to the open-source community.
gdiamos 1y ago
This is the end of an era. Amazing work soumith.
hshdhdhehd 1y ago
Nice, that is the dream career!
Glamklo 1y ago
[dead]
isusmelj 1y ago
Very proud as a Swiss that Soumith has a .ch domain!
- roflmaostc 1y ago
  Probably because his first name is Chintala
  - spprashant 1y ago
    That d be his last name
    - roflmaostc 1y ago
      true haha
sumedh 1y ago
His homepage says he wants to build a robot. So he is probably going to work with robots for his next role. He is an investor in Anthropic, didnt know you could do that working for Meta.
- geodel 1y ago
  Could be Meta is quite liberal in this area. Or it could be one of those "For my friend, anything, for everyone else its corporate policy."
deleted 1y ago
[deleted]
ergocoder 1y ago
I wonder how much this guy has earned from Meta in total. Would it reach $100M?
- stephenlf 1y ago
  Considering Meta was trying to Poach AI talent for $250M, I wouldn’t be surprised if this guy has his own 8-figure income
  - assemblyman 1y ago
    If someone made $2 million/year over 10 years, after taxes, it would be $1 million (NYC has local taxes too). Let's say all of it was saved and invested in SP500 or Meta. SP500: tripled over 10 years i.e. ~12% a year. Reinvesting dividends gives ~14% a year Meta: 8x over 10 years i.e. ~23% a year. If growth was uniform over 10 years and compensation/savings was uniform over 10 years, total portfolio would be: ((1+r)^11-1)/r (geometric series since each year's contributions grow for different amount of times) 1 (this year) + (1+r) (previous year) + (1+r)^2 (previous-to-previous year) and so on SP500: 14% -> $23M Meta: 23% -> $38M Now, it's entirely possible, the compensation for a position like this runs into $10s of millions and one can easily account for non-uniform compensation. Even in NYC, actually even in Manhattan, $10M is more than comfortable for retirement. It lets you draw $300-$400K (3-4% per year adjusted for inflation annually). If one is taking a short sabbatical, then it's a no-brainer.
    - lovecg 1y ago
      This seems to assume unusual optimism or foresight, most people don’t invest their life savings 100% into stocks and don’t hold on to 100% of their company vests through ups and downs. You might as well say “assuming he put all his money in NVDA…”
      - assemblyman 1y ago
        It's a back-of-the-envelope calculation not a precise one. It doesn't take foresight to invest in SP500. DCAing (dollar-cost averaging) into an index fund is actually the recommended savings strategy with a short-term cash balance of 6 months-2 years of cash savings depending on your plans (sabbatical etc.), especially when one is decades away from retirement age. I only included meta because he works/worked at meta and it's not unusual for people to just leave their rsus in their accounts after they vested. I agree though that one shouldn't pick stocks that happened to explode (e.g. nvda). There are several unrealistic assumptions I did make: * Presumably when someone starts, they earn less than in recent years. He probably wasn't making huge amounts his first few years. Amounts invested in earlier years are smaller but have more time to compound and amounts invested in recent years are larger but have had less time to compound. * Returns aren't constant. * I pulled the $2 million/yr out of thin air. It could be $1 million/yr or even $10 million/yr. I have no idea what the overall head of a project like PyTorch would make. * Everyone's expenses are different. In and around NYC, one can live on $80k/year, $120-150k/year as well as as on $1 million/yr. I assumed zero since I wanted a nice even $1 million/yr savings. Maybe it was $500k/yr of savings in which case all the numbers should be halved. In any case, I can't see how one wouldn't end up with at least $10 million in a position like this with 10 years at meta. Unless one buys a $5 million unit in Manhattan and is burdened by a high mortgage.
kleiba 1y ago
You forgot to thank Jürgen. /scnr
jsrozner 1y ago
Is this also partially AI generated? What's with the repeated short phrases? Is this just everyone's style now?
- Cthulhu_ 1y ago
  You're asking a lot of questions but are you willing to think about it? For one, no it's not "everyone's style" because you wouldn't have asked whether it was, you'd know.
- kelvinjps10 1y ago
  I felt it very human the writing on this post
sherinjosephroy 1y ago
[dead]
ishouldbework 1y ago
Look, I get that some pages require javascript, but <style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style> which is then unset by JS, with no <noscript> anywhere, is just... I just get white page. Changing it to <style class="fallback">body{white-space:pre-wrap;font-family:monospace}</style> gives perfectly readable web, so it seem bit... pointless.
philipwhiuk 1y ago
There's no context around 'FAIR' - is it https://www.go-fair.org/ https://www.go-fair.org/?
- abracos 1y ago
  it's https://ai.meta.com/research/ https://ai.meta.com/research/
- aprotyas 1y ago
  Facebook Artificial Intelligence Research, c.f. https://engineering.fb.com/category/ai-research/#:~:text=Artificial%20Intelligence%20Research%20(-,FAIR,-)%20seeks%20to%20understand https://engineering.fb.com/category/ai-research/#:~:text=Art...
  - overfeed 1y ago
    Now retconned as "Fundamental AI Research" since the Meta rebrand.
    - stuxnet79 1y ago
      Interesting, is Yann Lecun still heading FAIR? From the outside looking in, it seems like he is getting sidelined
shevy-java 1y ago
To me it sounds as if he is trying to open a new chapter in his life. Good for him, but I wonder if everything was really as smooth as described. People often write how everything is always perfect on their blog. Well - could be. But it could also be that not everything was perfect but no description followed on the blog.
xpe 1y ago
It is notable (but perhaps not surprising) that this is mostly about the people and the work itself. The writing is silent on the downstream impacts on the world. In contrast, there are fields (global development, medicine, etc.) where people tend to focus on the impact on humanity (especially when reaching a milestone in their career).
ninjagoo 1y ago
Soumith's 2nd release? https://github.com/pytorch/pytorch/releases/tag/v0.1.1 https://github.com/pytorch/pytorch/releases/tag/v0.1.1 Also, looking at the contribution history for a long career is very interesting; reflects the changing roles over time https://github.com/soumith https://github.com/soumith
mugivarra69 1y ago
[dead]
deleted 1y ago
[deleted]
CommenterPerson 1y ago
Firstly, Good work. Ironical but one HN front page item today is this: "Meta projected 10% of 2024 revenue came from scams and banned goods, Reuters reports" Glad you're leaving, hopefully you're in a good place financially. Take a page from Bill Gates and work on something that attempts to improve society. Stay away from surveillance capitalism and enshittification.
deaux 1y ago
[flagged]
odyssey7 1y ago
> It's taught in classrooms from MIT to rural India. The tools I dreamed about making accessible? They are. The barrier to entry I wanted to lower? It's almost gone. I have an ironic sense that there are classrooms in rural India with better pedagogy and lower barriers to entry than some of our elite engineering programs.
- john01dav 1y ago
  Many elite engineering programs in the United States (I don't know if this is what you mean by "our") are elite solely due to social status (rankings need to publish rankings that feel right, or they're ignored, and they accept bribes to rank specific programs) and research output, with little to do with quality of pedagogy. Instead, pedagogy is generally poor because the elite researchers usually view teaching as a chore and many don't have any real skill in it either.
- crazygringo 1y ago
  I thought traditional Indian pedagogy was heavily criticized for being heavily based on rote memorization over conceptual understanding or problem solving. And also being heavily hierarchical and exam-oriented. This isn't to say engineering programs in the US can't be improved, but there seems to be widespread consensus that they don't suffer from the kinds of serious problems that ones in India commonly do.
  - deleted 1y ago
    [deleted]
  - goku12 1y ago
    India isn't as uniform as you may imagine. Indian states are about as diverse as the EU. So don't assume that the Indian educational system adheres to any one system or paradigm. I for one, wouldn't have graduated in engineering if it was like what you described. I'm an utter failure at rote learning anything.
    - crazygringo 1y ago
mmaunder 1y ago
“ What's next for me? Something small. Something new. Something I don't fully understand yet. Something uncomfortable. I could have moved to something else inside Meta. But I needed to know what's out there. I needed to do something small again. I couldn't live with the counterfactual regret of never trying something outside Meta.” Shades of Siddhartha. Back to the forest.
- galoisscobi 1y ago
  Ah yes, shades of Siddhartha. I almost forgot about the part where he worked for a megacorp that was ripping society’s social fabric apart and wanted to do something else for a while.
  - abustamam 1y ago
    I don't think he was involved in that though.
    - dbgrman 1y ago
      He was. he didn't just parachute in Meta to start working on PyTorch. he worked in many areas of the product and a member of the senior technical staff, was knowledgable about many aspects of the company.
cs702 1y ago
Many of the comments here are judging PyTorch in hindsight, which is unfair. When Soumith Chintala co-launched PyTorch, and for many years after, the alternatives for fast, interactive, convenient development were much worse. There was no Jax. Every single AI researcher I know, including me, who tried PyTorch back then immediately wanted to switch to it, because it was so much better. Andrej Karpathy described what PyTorch felt like back then when he tweeted, on May 2017, "I've been using PyTorch a few months now and I've never felt better. I have more energy. My skin is clearer. My eyesight has improved."[a] THANK YOU SOUMITH for your hard work over all these years! Your hard work has made a difference for a huge number of people, including many of us here on HN. We wish you success in your future endeavors, whatever they turn out to be! Please ignore all the petty criticism. --- [a] https://x.com/karpathy/status/868178954032513024 https://x.com/karpathy/status/868178954032513024
- golly_ned 1y ago
  There was Chainer, which originated the define-by-run model that characterized PyTorch’s effectiveness. It was developed by a much smaller, much less influential company in Japan. Early PyTorch is transparent about the debt owed to Chainer.
  - maxc01 1y ago
    Yes, exactly—not many people know about Chainer nowadays. Back in 2016, PyTorch's interface was actually inferior to Chainer's, and I think Chainer's design was really ahead of its time.
  - NalNezumi 1y ago
    The company is called preferred networks, and they're still around, and have some branched off subsidiaries too.
  - cs702 1y ago
    Thanks. Yes, I remember Chainer, but only vaguely. I kinda remember looking at it, but not actually using it. My recollection is that when I looked at Chainer back then, it didn't offer a comprehensive library of preexisting components for deep learning. When I tried PyTorch, on the other hand, I vividly remember it as already having lots of prebuilt components (common layers, activation functions, etc.) in `torch.nn`, so it was easier and faster to get going. These memories are vague, so I could be wrong.
jmward01 1y ago
All I can say is 'thanks!'. It does take a team, and a community, but individuals matter. I use pytorch daily, it has made it possible for me to play with ideas that I would have only dreamed of. It is a big part of my life so, thanks for your contributions and best of luck on the next thing!
RobRivera 1y ago
Good luck, good job, and may more fantastical journeys lay ahead.
jurschreuder 1y ago
It's just a python wrapper around Torch, an open source C program by a Swiss non-profit, so META could compete with TensorFlow of Google. I don't know why this is celebrated so much, a big company rebranding a non-profit for profit. But I guess that's the norm for AI now.
- whymauri 1y ago
  Wait, PyTorch and the ecosystem are much more than just that. You can't be serious?
  - jurschreuder 1y ago
    It's just Torch for idiots
w10-1 1y ago
Given the technical achievements and industry impact, I'm struck that his emphasis is on the people, and in particular growing people to take on challenges. His influence might lie as much in what they do. The connection linking them seems to be building curiosity and wonder into delight for users. If there's a soul to silicon valley, that's it. However many jerks and political/power players I encounter, I remain inspired by the few like this.
mupuff1234 1y ago
https://news.ycombinator.com/item?id=45847465 https://news.ycombinator.com/item?id=45847465 Seems relevant.
jeffreysmith 1y ago
I'm one of the many people who Soumith hired to Meta and PyTorch. I had the privilege of working on PyTorch with him and lots of the folks on this post. As his longtime colleague, the one thing I would want people to know about him and this decision is that Soumith has always viewed PyTorch as a community project. He consistently celebrated the contributions of his co-creators Adam and Sam, and he extended the same view towards the Yangqing and the Caffe2 crew that we merged into PyTorch. At the very beginning, by Soumith's highly intentional design, PyTorch was aimed at being truly developed by and for the AI research community and for many years that was the key way in which we grew the framework, FB PT team, and the wider community. At every single stage of PT's lifecycle, he always ensured that our conception of PT and its community grew to include and celebrate the new people and organizations growing what was possible with PT. He's an incredible talent magnet, and thus more and more smart people kept dedicating their blood, sweat, and tears to making PT bigger and better for more people. I've worked with some very well known and highly compensated leaders in tech, but *no one* has done the job he has done with ameliorating a bus factor problem with his baby. PT has a unique level of broad support that few other open source technology can reach. In a world of unbounded AI salaries, people who want to move AI research methods forward still freely give their time and attention to PyTorch and its ecosystem. It's the great lever of this era of AI that is moving the world, *due in large part* to the strength of the community he fostered and can now let continue without his direct involvement. His departure is the end of an era, but it's also operationally a true non-event. PyTorch is going strong and can afford to let one of its creators retire from stewardship. This is precisely what success looks like in open source software. He deserves our congratulations and our thanks. Enjoy your PT retirement, man.
- casualscience 1y ago
  Also worked with Soumith. The man is a legend, moves mountains and completely changed the course of my career because he liked something I wrote. No arrogance, no politics, just an extremely down to earth and chill guy who elevates everyone around him. Hope him the best!
  - sumedh 1y ago
    What did you write?
warbaker 1y ago
I just want to say: thank you. It is hard to overstate how much Pytorch has accelerated ML/AI development, across the board.
livelaughlove69 1y ago
What an amazing impact this man has had. He seems like a great guy too. I don't know him at all personally but his replies online have always been so nice. I really wish him all the best with whatever he does with his time next
sota_pop 1y ago
I started my deep learning journey circa 2018 hand crafting autoencoders with Matlab cells. Pretty quickly moving on to Python, I remember reading about PyTorch and TensorFlow. Back then, TF lacked dynamic graphs which made PT way more favorable. Even still, many compared PT to Keras. However I always thought a closer comparison to Keras was something like Fast.ai from Jeremy Howard et. al., which I really enjoyed and don’t see many people use. In any case, I ended up sticking with PT and am extremely grateful for all the work put into it. Thank you.

721 points by saikatsg 1y ago

Sounds like you had a momentous run. If you take advice from reformed Internet trolls, consider turning off all your devices and trying to give yourself at least a week, but ideally a month offline staring at your new baby. You'll never get that time back and there's nothing your brain will appreciate more than loading up those memories as they grow. Good luck.

qmatch 1y ago

As a loyal JAX user, I hope they can play catchup. PyTorch has dominated the AI scene since TF1 fumbled the ball at 10th yard line. What Matt Johnson has done turning Autograd into JAX is hopefully going to be worthy of as much praise as what Soumith has received.

n_u 1y ago
> PyTorch has dominated the AI scene since TF1 fumbled the ball at 10th yard line can you explain why you think TensorFlow fumbled?
- zapnuk 1y ago
  For me it was about 8 years ago. Back then TF was already bloated but had two weaknesses. Their bet on static compute graphs made writing code verbose and debugging difficult. The few people I know back then used keras instead. I switched to PyTorch for my next project which was more "batteries included".
- michaelt 1y ago
  Imagine a total newbie trying to fine-tune an image classifier, reusing some open source example code, about a decade ago. If their folder of 10,000 labelled images contains one image that's a different size to the others, the training job will fail with an error about unexpected dimensions while concatenating. But it won't be able to say the file's name, or that the problem is an input image of the wrong size. It'll just say it can't concatenate tensors of different sizes. An experienced user will recognise the error immediately, and will have run a data cleansing script beforehand anyway. But it's not experienced users who bounce from frameworks, it's newbies.
  - mschuster91 1y ago
    > An experienced user will recognise the error immediately, and will have run a data cleansing script beforehand anyway. But it's not experienced users who bounce from frameworks, it's newbies. Even seasoned developers will bounce away from frameworks or libraries - no matter if old dogs or the next hot thing - if the documentation isn't up to speed or simple, common tasks require wading through dozens of pages of documentation. Writing good documentation is hard enough, writing relevant "common usage examples" is even harder... but keeping them up to date and working is a rarely seen art. And the greatest art of all of it is logging. Soooo many libraries refuse to implement detailed structured logging in internal classes (despite particularly Java and PHP offering very powerful mechanisms), making it much more difficult to troubleshoot problems in the field.
- qmatch 1y ago
  I personally believe TF1 was serving the need of its core users. It provided a compileable compute graph with autodiff, and you got very efficient training and inference from it. There was a steep learning curve, but if you got past it, things worked very very well. The distributed TF never really took off—it was buggy, and I think they made some wrong earlier bets in the design for performance reasons that they should have been sacrificed in favor of simplicity. I believe some years after the TF1 release, they realized the learning curve was too steep, they were losing users to PyTorch. I think also the Cloud team was attempting to sell customers on their amazing DL tech, which was falling flat. So they tried to keep the TF brand while totally changing the product under the hood by introducing imperative programming and gradient tapes. They killed TF1, upsetting those users, while not having a fully functioning TF2, all the while having plenty of documentation pointing to TF1 references that didn’t work. Any new grad student made the simple choice of using a tool that was user-friendly and worked, which was PyTorch. And most old TF1 users hopped on the band wagon.
- tdullien 1y ago
  I only remember 2015 TF and I was wondering: why would I use Python to assemble a computational graph when what I really want is to write code and then differentiate through it?
- Gazoche 1y ago
  I'm no machine learning engineer but I've dabbled professionally with both frameworks a few years ago and the developer experience didn't even compare. The main issue with TF was that you could only chose between a powerful but incomprehensible, poorly documented [1], ultra-verbose and ever changing low-level API, and an abstraction layer (Keras) that was too high level to be really useful. Maybe TF has gotten better since but at the time it really felt like an internal tool that Google decided to just throw into the wild. By contrast PyTorch offered a more reasonable level of abstraction along with excellent API documentation and tutorials, so it's no wonder that machine learning engineers (who are generally more interested in the science of the model than the technical implementation) ended up favoring it. [1] The worst part was that Google only hosted the docs for the latest version of TF, so if you were stuck on an older version (because, oh I don't know, you wanted a stable environment to serve models in production), well tough luck. That certainly didn't gain TF any favors.
- HarHarVeryFunny 1y ago
  The original TensorFlow had an API similar to the original Lua-based Torch (the predecessor to PyTorch) that required you to first build the network, node by node, then run it. PyTorch used a completely different, and much more convenient approach, where the network is built automatically for you just by running the forward pass code (and will then be used for the backward pass), using both provided node types and arbitrary NumPy compatible code. You're basically just writing differentiable code. This new PyTorch approach was eventually supported by TensorFlow as well ("immediate mode"), but the PyTorch approach was such a huge improvement that there had been an immediate shift by many developers from TF to PyTorch, and TF never seemed able to regain the momentum. TF also suffered from having a confusing array of alternate user libraries built on top of the core framework, none of which had great documentation, while PyTorch had a more focused approach and fantastic online support from the developer team.
  - liuliu 1y ago
    LuaTorch is eager-execution. The problem with LuaTorch is the GC. You cannot rely on traditional GC for good work, since each tensor is megabytes (at the time), now gigabytes large, you need to collect them aggressively rather than at intervals (Python's reference-counting system solves this issue, and of course, by "collecting", I don't mean free the memory (PyTorch has a simple slab allocator to manage CUDA memory)).
    - HarHarVeryFunny 1y ago
      With Lua Torch the model execution was eager, but you still had to construct the model graph beforehand - it wasn't "define by run" like PyTorch. Back in the day, having completed Andrew Ng's ML coursew, I then built my own C++ NN framework copying this graph-mode Lua Torch API. One of the nice things about explicitly building a graph was that my framework supported having the model generate a GraphViz DOT representation of itself so I could visualize it.
      - liuliu 1y ago
        Ah, I get what you mean now. I am mixing up the nn module and the tensor execution bits. (to be fair, the PyTorch nn module carries over many these quirks!).
- stared 1y ago
  In 2018, I co-wrote a blog post with the inflammatory title “Don’t use TensorFlow, try PyTorch instead” (https://news.ycombinator.com/item?id=17415321 https://news.ycombinator.com/item?id=17415321). As it gained traction here, it was changed to “Keras vs PyTorch” (some edgy things that work for a private blog are not good for a corporate one). Yet the initial title stuck, and you can see it resonated well with the crowd. TensorFlow (while a huge step on top of Theano) had issues with a strange API, mixing needlessly complex parts (even for the simplest layers) with magic-box-like optimization. There was Keras, which I liked and used before it was cool (when it still supported the Theano backend), and it was the right decision for TF to incorporate it as the default API. But it was 1–2 years too late. At the same time, I initially looked at PyTorch as some intern’s summer project porting from Lua to Python. I expected an imitation of the original Torch. Yet the more it developed, the better it was, with (at least to my mind) the perfect level of abstraction. On the one hand, you can easily add two tensors, as if it were NumPy (and print its values in Python, which was impossible with TF at that time). On the other hand, you can wrap anything (from just a simple operation to a huge network) in an nn.Module. So it offered this natural hierarchical approach to deep learning. It offered building blocks that can be easily created, composed, debugged, and reused. It offered a natural way of picking the abstraction level you want to work with, so it worked well for industry and experimentation with novel architectures. So, while in 2016–2017 I was using Keras as the go-to for deep learning (https://p.migdal.pl/blog/2017/04/teaching-deep-learning/ https://p.migdal.pl/blog/2017/04/teaching-deep-learning/), in 2018 I saw the light of PyTorch and didn’t feel a need to look back. In 2019, even for the intro, I used PyTorch (https://github.com/stared/thinking-in-tensors-writing-in-pytorch https://github.com/stared/thinking-in-tensors-writing-in-pyt...).
  - stared 1y ago
    Actually, I opened “Teaching deep learning” and smiled as I saw how it evolved: > There is a handful of popular deep learning libraries, including TensorFlow, Theano, Torch and Caffe. Each of them has Python interface (now also for Torch: PyTorch) > [...] > EDIT (July 2017): If you want a low-level framework, PyTorch may be the best way to start. It combines relatively brief and readable code (almost like Keras) but at the same time gives low-level access to all features (actually, more than TensorFlow). > EDIT (June 2018): In Keras or PyTorch as your first deep learning framework I discuss pros and cons of starting learning deep learning with each of them.
- probably_wrong 1y ago
  I see good answers already, but here's a concrete example: In my University we had to decide between both libraries so, as a test, we decided to write a language model from scratch. The first minor problem with TF was that (if memory serves me right) you were supposed to declare your network "backwards" - instead of saying "A -> B -> C" you had to declare "C(B(A))". The major problem, however, was that there was no way to add debug messages - either your network worked or it didn't. To make matters worse, the "official" TF tutorial on how to write a Seq2Seq model didn't compile because the library had changed but the bug reports for that were met for years with "we are changing the API so we'll fix the example once we're done". PyTorch, by comparison, had the advantage of a Python-based interface - you simply defined classes like you always did (including debug statements!), connected them as variables, and that was that. So when I and my beginner colleagues had to decide which library to pick, "the one that's not a nightmare to debug" sounded much better than "the one that's more efficient if you have several billions training datapoints and a cluster". Me and my colleagues then went on to become professionals, and we all brought PyTorch with us.
  - jszymborski 1y ago
    The inability to use print debug to tell me the dimensions of my hidden states was 100% why TF was hard for me to use as a greenhorn MSc student. Another consequence of this was that PyTorch let you use regular old Python for logic flow.
  - n_u 1y ago
    This was also my experience. TensorFlow's model of constructing then evaluating a computation graph felt at odds with Python's principles. It made it extremely difficult to debug because you couldn't print tensors easily! It didn't feel like Python at all. Also the API changed constantly so examples from docs or open source repos wouldn't work. They also had that weird thing about all tensors having a unique global name. I remember I tried to evaluate a DQN network twice in the same script and it errored because of that. It's somewhat vindicating to see many people in this thread shared my frustrations. Considering the impact of these technologies I think a documentary about why TensorFlow failed and PyTorch took off would be a great watch.
- morshu9001 1y ago
  I just remember TF1 being super hard to use as a beginner and Google repeatedly insisting it had to be that way. People talk about the layering API, but it's more than that, everything about it was covered with sharp edges.
- htrp 1y ago
  Greenfielding TF2.X and not maintaining 1.X compatibility
- rockinghigh 1y ago
  First, the migration to 2.0 in 20219 to add eager mode support was horribly painful. Then, starting around 2.7, backward compatibility kept being broken. Not being able to load previously trained models with a new version of the library is wildly painful.
intermerda 1y ago
Do you have experience in both JAX and PyTorch? Why do you prefer JAX?
- cl3misch 1y ago
  Not OP. I prefer JAX for non-AI tasks in scientific computing because of the different mental model than PyTorch. In JAX, you think about functions and gradients of functions. In PyTorch you think about tensors which accumulate a gradient while being manipulated through functions. JAX just suits my way of thinking much better. I also like that jax.jit forces you to write "functional" functions free of side effects or inplace array updates. It might feel weird at first (and not every algorithm is suited for this style) but ultimately it leads to clearer and faster code. I am surprised that JIT in PyTorch gets so little attention. Maybe it's less impactful for PyTorch's usual usecase of large networks, as opposed to general scientific computing?
  - imtringued 1y ago
    >I also like that jax.jit forces you to write "functional" functions free of side effects or inplace array updates. It might feel weird at first (and not every algorithm is suited for this style) but ultimately it leads to clearer and faster code. It's not weird. It's actually the most natural way of doing things for me. You just write down your math equations as JAX and you're done.
    - Majromax 1y ago
      > You just write down your math equations as JAX and you're done. It's natural when your basic unit is a whole vector (tensor), manipulated by some linear algebra expression. It's less natural if your basic unit is an element of a vector. If you're solving sudoku, for example, the obvious 'update' is in-place. In-place updates are also often the right answer for performance reasons, such as writing the output of a .map() operation directly to the destination tensor. Jax leans heavily on compile-time optimizations to turn the mathematically-nice code into computer-nice code, so the delta between eager-Jax and compiled-Jax is much larger than the delta between eager-Pytorch and compiled-Pytorch.
bjourne 1y ago
JAX seems great but the Google ghost is a lot to stomach. The risk of JAX getting axed or replaced with a JAX 2.0 - completely incompatible with existing JAX code - is not insignificant.

chopete3 1y ago

>>Every major AI company and hardware vendor are on a speed dial. This kind of power is really hard to give up. But curiosity ultimately won out in my head. A simple feeling has such a power. May he gets an opportunity to create one more powerful tool before retiring.

Lord-Jobo 1y ago
If the curiosity dies, the entire thing crumbles. The second I stop being curious I stop finding new and exciting things to do, and I stop feeling fulfillment. It’s one of the biggest signs saying “it’s time to move on”. I feel so strongly for the people who can’t afford the luxury. Ive been there, unfulfilling jobs for years because bills or resumè building.
- rubicon33 1y ago
  Gosh, given you’ve been there I have to ask what allowed you to get out of that and pursue only things that interest and excite you?
  - Lord-Jobo 11mo ago
    80% lucky, 10% low standards, 10% hard work. In short I went from a small company position with very little growth and constant paycheck delays to a job with very clear monthly metric goals and a manager that lets me pursue projects for our office and client that actually help and improve everyone’s experience. My field is really small and most hiring managers/HR people frankly have no clue how to hire for it, so I have a genuinely great resume and still got ghosted for many positions I was easily qualified for. So hopping from a low paying position I was very tired with, to a job with daily activities I actually enjoy, that pays way more appropriately, was a several year process when it would be a 2 month process in a bigger field. Realistically if I had to attribute anything in particular it’s these two things: 1: knowing what kind of work would actually fulfill me, which turned out to be fairly repetitive grunt work with 10% special projects. 2: learning our primary software VERY well. I have not met any other analyst who knows the quirks and tricks as well as I do, but that mostly just comes from 8 years of struggling with it and their nonexistent customer support. The low standards are important for this as well because I am making 15% over the median personal income, and many people on here have ambitions WAY higher than that. But my partner makes double what I do, we have no kids, and we have cheap rent for our area so I really am thankful for what we have. Pursuing what actually interests me is easy as well because I’m happy with a job that’s 90% assembly line work and 10% actually novel project work.

perfmode 1y ago

Respect.

mxkopy 1y ago

PyTorch is one of those tools that’s so simple and easy to take apart that you feel like you might’ve been able to make it yourself. I can’t imagine how much engineering effort was behind all those moments where I thought to myself, “of course it should work like that, how can it be any other way?”

TechnicolorByte 1y ago
Can anyone recommend a technical overview describing the design decisions PyTorch made that led it to win out?
- huevosabio 1y ago
  I don't know the full list, but back when it came out, TF felt like a crude set of bindings to the underlying c++/CUDA workhorse. PyTorch felt, in contrast, pythonic. It was much closer in feeling to numpy.
- puttycat 1y ago
  I think it was mostly the eager evaluation that made it possible to debug every step in the network forward/backward passes. Tensorflow didn't have that at the time which made debugging practically impossible.
- GistNoesis 1y ago
  The choice of the dynamic computation graph [1] of PyTorch made it easier to debug and implement, leading to higher adoption, even though running speed was initially slower (and therefore training cost higher). Other decisions follow from this one. Tensorflow started with static and had to move to dynamic at version 2.0, which broke everything. Fragmentation between tensorflow 1, tensorflow 2, keras, jax. Pytorch's compilation of this computation graph erased the remaining edge of Tensorflow. Is the battle over ? From a purely computational point, Pytorch solution is very far from optimal and billions of dollars of electricity and GPUs are burned every year, but major players are happy with circular deals to entrench their positions. So at the pace of current AI code development, probably one or two years before Pytorch is old history. [1] https://www.geeksforgeeks.org/deep-learning/dynamic-vs-static-computational-graphs-pytorch-and-tensorflow/ https://www.geeksforgeeks.org/deep-learning/dynamic-vs-stati...

BoredPositron 1y ago

The last few years must have been incredibly exhausting. Thanks for your work good luck and 73.

vintermann 1y ago

That man has an infective enthusiasm. I remember the DCGAN paper inspired me to try getting the (Lua) Torch code to work, and I tried it on the Oxford flowers dataset early on. It worked surprisingly well, and Soumith Chintala even shared it around in social media, surprised at how well it worked on such a small dataset. Of course back then we didn't really appreciate the problem of mode collapse. Pytorch and old Lua Torch were a pleasure to work with compared to the contemporary Tensorflow. Lots of S.C's code was copied around liberally, it had its quirks (I remember the DCGAN code had a pretty odd way of doing parameter passing) but it was also really easy to understand and made random people like me feel like we had suddenly stumbled onto something crazy powerful (which we had!). It was wonderfully hackable.

whitten 1y ago
What is the Oxford flowers dataset ? Where is it available ?
- rockinghigh 1y ago
  https://www.robots.ox.ac.uk/~vgg/data/flowers/ https://www.robots.ox.ac.uk/~vgg/data/flowers/

zkmon 1y ago

[flagged]

theusus 1y ago

[flagged]

utopiah 1y ago

What I find most interesting with this is that it shows they believe there is nothing unique at Meta related to AI. There is no resource, people and computing power, that they can't get elsewhere for whatever they believe would be more interesting for them. I mention this because it feels analogous to military research, where people "dream" of how advanced the military is, how forward they are compared to public research... and yet, it seems to be a recurring myth they love to sustain. So the signal I get here is AI "labs" in BigTech have nothing worth waiting for around the corner, it's just more of the same and boring for people who stick there.

rtpg 1y ago
I don't think that's the read? Guy says he wants to work on something small. If you want to work on something big you probably want to be in a big corp to have the resources to do the big thing. Also absolutely unknown if the "new thing" is AI-related at all!
- utopiah 1y ago
  Well he left so whatever is coming next, AI related or not, "small" or not (small for them might be reaching just a million people, he wrote that he "lead the software layer that powers the entire AI industry." so his notion of scale is probably unlike mine, maybe yours too) is more exciting to him that whatever he could do next with all of Meta's resources. Edit: to be clear, I didn't mean to imply their next thing is AI related, solely that they obviously know more about AI at Meta than e.g. XR at Meta, just because that's their expertise.
  - hombre_fatal 1y ago
    Your assumption is a bad read because it only works if his set of life priorities contains nothing else but maximizing his impact in the world of AI. If he has just one other priority in that set (which could still include a robotic min/max of AI impact), then your assumption fails.
- radicalbyte 1y ago

aabhay 1y ago

For anyone that’s curious, the underlying Torch library is also a joy to work with, as are the many other torch bindings. For example, Rust has tch and Burn which both work with libtorch. PyTorch of course has the benefit of being dynamically debuggable. Can’t forget the first time I break pointed my pytorch model and wrote pytorch calls inside the terminal to inspect the behavior. That’s still something I miss a lot now that I’m working with only “fast” compiled code.

lysecret 1y ago
I wrote som truly awful code back in the day because of that but god it was glorious.
deleted 1y ago
[deleted]

irthomasthomas 1y ago

Counterfactual Regret Minimization irl

numice 1y ago

I read one post on his blog and found that Adam Paszke reached out to the author and got an internship. I wonder if it was that easy to get an internship at FAIR. I thought that they hire only PhDs.

vintermann 1y ago
I didn't know that. Soumith Chintala certainly paid it forward. He was very helpful and supportive of random strangers (like me!) in the early pytorch days. I count him with Andrej Karpathy and Chris Olah as one of the people who really made machine learning accessible to regular software engineers.
- SpaceManNabs 1y ago
  Chris Olah is the goat. I reached out to him myself years ago and was surprised at getting a response. And the response was incredibly generous. I probably wouldn't have had the confidence to do my switch if it wasn't for Olah. And as I got further into this path, I learned that Olah had done the same for some of my mentors and colleagues. Every time Olah speaks, I listen.
chrneu 1y ago
You can't do anything if you never try.
nswizzle31 1y ago
I was pretty involved in the PyTorch ecosystem in the early days around 2016 and Adam was nothing short of a genius and prolific developer whose contributions to the codebase and community were immense. I think he was like an undergrad in Poland at the time. My understanding is that his contributions came before the internship, but I don’t know. My memory is that Souminth was really open to other people’s contributions and questions, no matter their credentials. He was a great leader who felt approachable to the open-source community.

gdiamos 1y ago

This is the end of an era. Amazing work soumith.

hshdhdhehd 1y ago

Nice, that is the dream career!

Glamklo 1y ago

[dead]

isusmelj 1y ago

Very proud as a Swiss that Soumith has a .ch domain!

roflmaostc 1y ago
Probably because his first name is Chintala
- spprashant 1y ago
  That d be his last name
  - roflmaostc 1y ago
    true haha

sumedh 1y ago

His homepage says he wants to build a robot. So he is probably going to work with robots for his next role. He is an investor in Anthropic, didnt know you could do that working for Meta.

geodel 1y ago
Could be Meta is quite liberal in this area. Or it could be one of those "For my friend, anything, for everyone else its corporate policy."

deleted 1y ago

[deleted]

ergocoder 1y ago

I wonder how much this guy has earned from Meta in total. Would it reach $100M?

stephenlf 1y ago
Considering Meta was trying to Poach AI talent for $250M, I wouldn’t be surprised if this guy has his own 8-figure income
- assemblyman 1y ago
  If someone made $2 million/year over 10 years, after taxes, it would be $1 million (NYC has local taxes too). Let's say all of it was saved and invested in SP500 or Meta. SP500: tripled over 10 years i.e. ~12% a year. Reinvesting dividends gives ~14% a year Meta: 8x over 10 years i.e. ~23% a year. If growth was uniform over 10 years and compensation/savings was uniform over 10 years, total portfolio would be: ((1+r)^11-1)/r (geometric series since each year's contributions grow for different amount of times) 1 (this year) + (1+r) (previous year) + (1+r)^2 (previous-to-previous year) and so on SP500: 14% -> $23M Meta: 23% -> $38M Now, it's entirely possible, the compensation for a position like this runs into $10s of millions and one can easily account for non-uniform compensation. Even in NYC, actually even in Manhattan, $10M is more than comfortable for retirement. It lets you draw $300-$400K (3-4% per year adjusted for inflation annually). If one is taking a short sabbatical, then it's a no-brainer.
  - lovecg 1y ago
    This seems to assume unusual optimism or foresight, most people don’t invest their life savings 100% into stocks and don’t hold on to 100% of their company vests through ups and downs. You might as well say “assuming he put all his money in NVDA…”
    - assemblyman 1y ago
      It's a back-of-the-envelope calculation not a precise one. It doesn't take foresight to invest in SP500. DCAing (dollar-cost averaging) into an index fund is actually the recommended savings strategy with a short-term cash balance of 6 months-2 years of cash savings depending on your plans (sabbatical etc.), especially when one is decades away from retirement age. I only included meta because he works/worked at meta and it's not unusual for people to just leave their rsus in their accounts after they vested. I agree though that one shouldn't pick stocks that happened to explode (e.g. nvda). There are several unrealistic assumptions I did make: * Presumably when someone starts, they earn less than in recent years. He probably wasn't making huge amounts his first few years. Amounts invested in earlier years are smaller but have more time to compound and amounts invested in recent years are larger but have had less time to compound. * Returns aren't constant. * I pulled the $2 million/yr out of thin air. It could be $1 million/yr or even $10 million/yr. I have no idea what the overall head of a project like PyTorch would make. * Everyone's expenses are different. In and around NYC, one can live on $80k/year, $120-150k/year as well as as on $1 million/yr. I assumed zero since I wanted a nice even $1 million/yr savings. Maybe it was $500k/yr of savings in which case all the numbers should be halved. In any case, I can't see how one wouldn't end up with at least $10 million in a position like this with 10 years at meta. Unless one buys a $5 million unit in Manhattan and is burdened by a high mortgage.

kleiba 1y ago

You forgot to thank Jürgen. /scnr

jsrozner 1y ago

Is this also partially AI generated? What's with the repeated short phrases? Is this just everyone's style now?

Cthulhu_ 1y ago
You're asking a lot of questions but are you willing to think about it? For one, no it's not "everyone's style" because you wouldn't have asked whether it was, you'd know.
kelvinjps10 1y ago
I felt it very human the writing on this post

sherinjosephroy 1y ago

[dead]

ishouldbework 1y ago

Look, I get that some pages require javascript, but <style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style> which is then unset by JS, with no <noscript> anywhere, is just... I just get white page. Changing it to <style class="fallback">body{white-space:pre-wrap;font-family:monospace}</style> gives perfectly readable web, so it seem bit... pointless.

philipwhiuk 1y ago

There's no context around 'FAIR' - is it https://www.go-fair.org/ https://www.go-fair.org/?

abracos 1y ago
it's https://ai.meta.com/research/ https://ai.meta.com/research/
aprotyas 1y ago
Facebook Artificial Intelligence Research, c.f. https://engineering.fb.com/category/ai-research/#:~:text=Artificial%20Intelligence%20Research%20(-,FAIR,-)%20seeks%20to%20understand https://engineering.fb.com/category/ai-research/#:~:text=Art...
- overfeed 1y ago
  Now retconned as "Fundamental AI Research" since the Meta rebrand.
  - stuxnet79 1y ago
    Interesting, is Yann Lecun still heading FAIR? From the outside looking in, it seems like he is getting sidelined

shevy-java 1y ago

To me it sounds as if he is trying to open a new chapter in his life. Good for him, but I wonder if everything was really as smooth as described. People often write how everything is always perfect on their blog. Well - could be. But it could also be that not everything was perfect but no description followed on the blog.

xpe 1y ago

It is notable (but perhaps not surprising) that this is mostly about the people and the work itself. The writing is silent on the downstream impacts on the world. In contrast, there are fields (global development, medicine, etc.) where people tend to focus on the impact on humanity (especially when reaching a milestone in their career).

ninjagoo 1y ago

Soumith's 2nd release? https://github.com/pytorch/pytorch/releases/tag/v0.1.1 https://github.com/pytorch/pytorch/releases/tag/v0.1.1 Also, looking at the contribution history for a long career is very interesting; reflects the changing roles over time https://github.com/soumith https://github.com/soumith

mugivarra69 1y ago

[dead]

deleted 1y ago

[deleted]

CommenterPerson 1y ago

Firstly, Good work. Ironical but one HN front page item today is this: "Meta projected 10% of 2024 revenue came from scams and banned goods, Reuters reports" Glad you're leaving, hopefully you're in a good place financially. Take a page from Bill Gates and work on something that attempts to improve society. Stay away from surveillance capitalism and enshittification.

deaux 1y ago

[flagged]

odyssey7 1y ago

> It's taught in classrooms from MIT to rural India. The tools I dreamed about making accessible? They are. The barrier to entry I wanted to lower? It's almost gone. I have an ironic sense that there are classrooms in rural India with better pedagogy and lower barriers to entry than some of our elite engineering programs.

john01dav 1y ago
Many elite engineering programs in the United States (I don't know if this is what you mean by "our") are elite solely due to social status (rankings need to publish rankings that feel right, or they're ignored, and they accept bribes to rank specific programs) and research output, with little to do with quality of pedagogy. Instead, pedagogy is generally poor because the elite researchers usually view teaching as a chore and many don't have any real skill in it either.
crazygringo 1y ago
I thought traditional Indian pedagogy was heavily criticized for being heavily based on rote memorization over conceptual understanding or problem solving. And also being heavily hierarchical and exam-oriented. This isn't to say engineering programs in the US can't be improved, but there seems to be widespread consensus that they don't suffer from the kinds of serious problems that ones in India commonly do.
- deleted 1y ago
  [deleted]
- goku12 1y ago
  India isn't as uniform as you may imagine. Indian states are about as diverse as the EU. So don't assume that the Indian educational system adheres to any one system or paradigm. I for one, wouldn't have graduated in engineering if it was like what you described. I'm an utter failure at rote learning anything.
  - crazygringo 1y ago

mmaunder 1y ago

“ What's next for me? Something small. Something new. Something I don't fully understand yet. Something uncomfortable. I could have moved to something else inside Meta. But I needed to know what's out there. I needed to do something small again. I couldn't live with the counterfactual regret of never trying something outside Meta.” Shades of Siddhartha. Back to the forest.

galoisscobi 1y ago
Ah yes, shades of Siddhartha. I almost forgot about the part where he worked for a megacorp that was ripping society’s social fabric apart and wanted to do something else for a while.
- abustamam 1y ago
  I don't think he was involved in that though.
  - dbgrman 1y ago
    He was. he didn't just parachute in Meta to start working on PyTorch. he worked in many areas of the product and a member of the senior technical staff, was knowledgable about many aspects of the company.

cs702 1y ago

Many of the comments here are judging PyTorch in hindsight, which is unfair. When Soumith Chintala co-launched PyTorch, and for many years after, the alternatives for fast, interactive, convenient development were much worse. There was no Jax. Every single AI researcher I know, including me, who tried PyTorch back then immediately wanted to switch to it, because it was so much better. Andrej Karpathy described what PyTorch felt like back then when he tweeted, on May 2017, "I've been using PyTorch a few months now and I've never felt better. I have more energy. My skin is clearer. My eyesight has improved."[a] THANK YOU SOUMITH for your hard work over all these years! Your hard work has made a difference for a huge number of people, including many of us here on HN. We wish you success in your future endeavors, whatever they turn out to be! Please ignore all the petty criticism. --- [a] https://x.com/karpathy/status/868178954032513024 https://x.com/karpathy/status/868178954032513024

golly_ned 1y ago
There was Chainer, which originated the define-by-run model that characterized PyTorch’s effectiveness. It was developed by a much smaller, much less influential company in Japan. Early PyTorch is transparent about the debt owed to Chainer.
- maxc01 1y ago
  Yes, exactly—not many people know about Chainer nowadays. Back in 2016, PyTorch's interface was actually inferior to Chainer's, and I think Chainer's design was really ahead of its time.
- NalNezumi 1y ago
  The company is called preferred networks, and they're still around, and have some branched off subsidiaries too.
- cs702 1y ago
  Thanks. Yes, I remember Chainer, but only vaguely. I kinda remember looking at it, but not actually using it. My recollection is that when I looked at Chainer back then, it didn't offer a comprehensive library of preexisting components for deep learning. When I tried PyTorch, on the other hand, I vividly remember it as already having lots of prebuilt components (common layers, activation functions, etc.) in `torch.nn`, so it was easier and faster to get going. These memories are vague, so I could be wrong.

jmward01 1y ago

All I can say is 'thanks!'. It does take a team, and a community, but individuals matter. I use pytorch daily, it has made it possible for me to play with ideas that I would have only dreamed of. It is a big part of my life so, thanks for your contributions and best of luck on the next thing!

RobRivera 1y ago

Good luck, good job, and may more fantastical journeys lay ahead.

jurschreuder 1y ago

It's just a python wrapper around Torch, an open source C program by a Swiss non-profit, so META could compete with TensorFlow of Google. I don't know why this is celebrated so much, a big company rebranding a non-profit for profit. But I guess that's the norm for AI now.

whymauri 1y ago
Wait, PyTorch and the ecosystem are much more than just that. You can't be serious?
- jurschreuder 1y ago
  It's just Torch for idiots

w10-1 1y ago

Given the technical achievements and industry impact, I'm struck that his emphasis is on the people, and in particular growing people to take on challenges. His influence might lie as much in what they do. The connection linking them seems to be building curiosity and wonder into delight for users. If there's a soul to silicon valley, that's it. However many jerks and political/power players I encounter, I remain inspired by the few like this.

mupuff1234 1y ago

https://news.ycombinator.com/item?id=45847465 https://news.ycombinator.com/item?id=45847465 Seems relevant.

jeffreysmith 1y ago

I'm one of the many people who Soumith hired to Meta and PyTorch. I had the privilege of working on PyTorch with him and lots of the folks on this post. As his longtime colleague, the one thing I would want people to know about him and this decision is that Soumith has always viewed PyTorch as a community project. He consistently celebrated the contributions of his co-creators Adam and Sam, and he extended the same view towards the Yangqing and the Caffe2 crew that we merged into PyTorch. At the very beginning, by Soumith's highly intentional design, PyTorch was aimed at being truly developed by and for the AI research community and for many years that was the key way in which we grew the framework, FB PT team, and the wider community. At every single stage of PT's lifecycle, he always ensured that our conception of PT and its community grew to include and celebrate the new people and organizations growing what was possible with PT. He's an incredible talent magnet, and thus more and more smart people kept dedicating their blood, sweat, and tears to making PT bigger and better for more people. I've worked with some very well known and highly compensated leaders in tech, but *no one* has done the job he has done with ameliorating a bus factor problem with his baby. PT has a unique level of broad support that few other open source technology can reach. In a world of unbounded AI salaries, people who want to move AI research methods forward still freely give their time and attention to PyTorch and its ecosystem. It's the great lever of this era of AI that is moving the world, *due in large part* to the strength of the community he fostered and can now let continue without his direct involvement. His departure is the end of an era, but it's also operationally a true non-event. PyTorch is going strong and can afford to let one of its creators retire from stewardship. This is precisely what success looks like in open source software. He deserves our congratulations and our thanks. Enjoy your PT retirement, man.

casualscience 1y ago
Also worked with Soumith. The man is a legend, moves mountains and completely changed the course of my career because he liked something I wrote. No arrogance, no politics, just an extremely down to earth and chill guy who elevates everyone around him. Hope him the best!
- sumedh 1y ago
  What did you write?

warbaker 1y ago

I just want to say: thank you. It is hard to overstate how much Pytorch has accelerated ML/AI development, across the board.

livelaughlove69 1y ago

What an amazing impact this man has had. He seems like a great guy too. I don't know him at all personally but his replies online have always been so nice. I really wish him all the best with whatever he does with his time next

sota_pop 1y ago

I started my deep learning journey circa 2018 hand crafting autoencoders with Matlab cells. Pretty quickly moving on to Python, I remember reading about PyTorch and TensorFlow. Back then, TF lacked dynamic graphs which made PT way more favorable. Even still, many compared PT to Keras. However I always thought a closer comparison to Keras was something like Fast.ai from Jeremy Howard et. al., which I really enjoyed and don’t see many people use. In any case, I ended up sticking with PT and am extremely grateful for all the work put into it. Thank you.