10 ms·

While PyTorch is obviously the future in the short term, it will be interesting to see how this space evolves. Before Tensorflow, people (myself included) were

While PyTorch is obviously the future in the short term, it will be interesting to see how this space evolves. Before Tensorflow, people (myself included) were largely coding all of this stuff pretty manually, or with the zoo of incredibly clucky homemade libs. Tensorflow and PyTorch made the whole situation far more accessible and sane. You can get a basic neural network working in a few lines of code. Magical. But it's still early days. George Hotz, author of tinygrad[0], a PyTorch "competitor", made a really insightful comment -- we will look back on PyTorch & friends like we look back on FORTRAN and COBOL. Yes, they were far better than what came before. But they are really clunky compared to what we have today. What will we have in 20 years? [0] https://github.com/geohot/tinygrad https://github.com/geohot/tinygrad, https://tinygrad.org https://tinygrad.org

KKKKkkkk1 4y ago
PyTorch and Tensorflow were good enough to implement all the major advanves in recent years: the AlexNet successors, GPT, Stable Diffusion. It's not early days, and the market will not evolve unless there is a new application that cannot be served by PyT and TF.
letitgo12345 4y ago
Theano preceded Tensorflow by 8 years. So I don't think it was all manual coding before TF. Torch (in Lua) was also around much before TF
- gamegoblin 4y ago
  That's definitely true, though worth noting that those libs didn't have anything close to the dominance the modern ones do. I was doing ML academia stuff in the pre-TF era, and most of my department was using custom C++/CUDA, or a pile of random stuff like https://en.wikipedia.org/wiki/Weka_(machine_learning) https://en.wikipedia.org/wiki/Weka_(machine_learning) and https://en.wikipedia.org/wiki/Waffles_(machine_learning) https://en.wikipedia.org/wiki/Waffles_(machine_learning). Our department also didn't even have GPUs to use for the first part of my time there, whereas now it would seem ridiculous to have an ML research wing without access to GPUs.
  - quietbritishjim 4y ago
    I think the TensorFlow release roughly coincided with the widespread adoption of proper deep learning library, but it didn't cause it. In fact the first TensorFlow seemed to be just after (by 6 months to a year) the inflection point of deep learning library usage. At the time, there seemed to be a lively ecosystem but no clear winner. Theano, Caffe and Torch were the main hitters but there were smaller players too. Torch was particularly good, maybe functionally ahead (but I don't really remember for sure) but hampered because, at the time, it was Lua only. In fact Keras's first release said "Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python / Theano so as not to have to deal with the dearth of ecosystem in Lua." When TensorFlow was released, it looked like that would be the death of all the other libraries. The fact that any others survived at all is really down to how hard to use TensorFlow was (/is?). It didn't live up to the expectation of being the killer of other libraries.
vgatherps 4y ago
My work might make me biased but I think there’s a lot of room to make more general purpose code differentiable. I work on a ton of stuff that would very much benefit from being differentiable, but also very much can’t fit at all into the “stack a ton of linalg-like layers”. It’s a huge engineering effort to even try to think about how I might start taking out derivatives. It’s possible but there’s so much overhead to do anything with it.
- wenc 4y ago
  > make more general purpose code differentiable Non-smooth functions (e.g. abs(x)) can be handled with bundle methods, but how would one make inherently discontinuous (non-convex) functions differentiable? (e.g. if x then 1 else 5) Discrete problems are inherently non-differentiable. There are approaches like complementarity methods and switching functions (tanh) usually end up with numerical issues.
  - wnkrshm 4y ago
    And then people go to empirical data and apply the Great Smoothing: By dropping ML/DL methods on the data (also including results from discontinuous behavior), continuity is often implicitly assumed.
  - taylorius 4y ago
    This already happens to an extent in existing ML pipelines. The ReLU activation function, is discontinuous in its derivative, and is one of the most widely used functions in neural networks. Its derivative looks like this. if(i<0) return 0; else return 1; Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms. I think this is where the problem lies - not with differentiability itself, but with gradient descent not working due to the highly non-convex search space that such general programming constructs will produce.
    - wenc 4y ago
      ReLU is not discontinuous; it is nonsmooth but continuous, hence derivatives exist except at the hinge points. Inherently discontinuous functions OTOH are disconnected and nonconvex. Gradient descent works, but you have to add a step to first partition the discrete space like branch and bound. This involves solving the continuous relaxation to find a bound. This does not require differentiability (it is not differentiable), but the price to pay is that it is combinatorial (NP hard) The OP was talking about general differentiability but inherently discontinuous functions form a large and important class of functions (from software programming) that are not differentiable.
singularity2001 4y ago
one step in the right directions are labeled dimensions, where you can write input = width * hight * color * batch and never have to worry about tensor permutations again. experimental support in pytorch
- amelius 4y ago
  Is that part of the type system?
  - singularity2001 4y ago
    maybe I misremembered; https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html
- albertzeyer 4y ago
  It's called named dimensions, named tensors, dimension tags: https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html It's already there in PyTorch since a while (2019). However, as far as I know, it has not found too much adoption yet. In our framework RETURNN, we also make heavy use of dimension tags: https://returnn.readthedocs.io/en/latest/getting_started/data.html https://returnn.readthedocs.io/en/latest/getting_started/dat...
  - CoastalCoder 4y ago
    I forget the details, but I think Nervana also had a library(?) that used this concept before they were bought by Intel.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
mattlondon 4y ago
> What will we have in 20 years? Whatever it is, I sincerely hope that it is not driven by Python, and instead we have a more accessible language. Say what you will about Python, but it is a total pain in the arse to work with. The 2.x legacy still casts a long shadow (in terms of online guides, unmaintained libs etc), anaconda, pip, easy install, strings/unicode, type hints (or not), unreadable & unmaintainable list comprehensions etc etc etc. What a dumpster fire, and a truly hateful language. Here is hoping that some other language takes over. I have my personal take, but I won't let that ruin the rant :)
- deleted 4y ago
  [deleted]
- ergo14 4y ago
  You are holding it wrong ;)
- visarga 4y ago
  Hey, I like list comprehensions, thank you very much.
  - jjgreen 4y ago
    Bummer, list comprehension is deprecated, you should instead use a "virtual list" maintained by some kid who will shortly be losing interest ...
    - geysersam 4y ago
      That's a strange comment. Python encourages using its basic container types. Very unusual to use a third party library for something like that. I've never encountered a "virtual" anything in Python. That's more common in c++ or Java.
- CoastalCoder 4y ago
  I hate python for big software systems as much as the next guy, but for whatever reason some people are quite productive with it. I think programmers fall into different camps on this issue.
imiric 4y ago
> What will we have in 20 years? Hopefully something that doesn't require the user to be a math wizard. I'm not into data science by any means, and I first learned about tensors with Tensorflow. While I would like to experiment with neural nets, the current frameworks require me to learn the foundations of how they work. I'm sure I'll get arguments that this is a good thing, but I just want to do some cool stuff with computers, not take math courses to learn how the cake is made. High-level programming languages are abstractions to machine code that, for the most part, don't require the user to know how they work internally. I think we're missing similar abstractions for ML, or it's possible they already exist and I haven't looked hard enough.
- cm2187 4y ago
  The right analogy is cryptography. You don't need to understand the math behind cryptography to calculate a hash, or encrypt something. You just need to understand at a very high level what is asymmetric encryption, IV, etc.
  - netheril96 4y ago
    That’s actually a bad analogy in my opinion. People who don’t learn cryptography make many mistakes calling cryptography libraries and those mistakes can’t be uncovered by testing.
    - bborud 4y ago
      Oh, it'll all be great when ChatGPT writes all that code for us and we can be off somewhere having a beer and a cry instead.
      - liotier 4y ago
        You'll cry but, to listen to your sorrow and offer friendly support, ChatGPT will be there for you !
        4y ago
thom 4y ago
So you mean we’ll still be compiling PyTorch as a dependency for various libraries for the next 40 years?
jackblemming 4y ago
>George Hotz, author of tinygrad[0], a PyTorch "competitor", made a really insightful comment -- we will look back on PyTorch & friends like we look back on FORTRAN and COBOL. Excuse me but, what? You could say this about pretty much every "hot" framework. Technology moves fast.
catwell 4y ago
> While PyTorch is obviously the future in the short term Well, that's not obvious for me. The metrics do not support the death of Keras / TF (https://twitter.com/fchollet/status/1614322127460782087 https://twitter.com/fchollet/status/1614322127460782087) François Chollet's Deep Learning with Python is arguably the best book on the topic and teaches people using Keras / TF2. The Keras documentation is excellent as well, and in the long run it will matter. Personally I prefer the way the Keras / TF2 APIs work to PyTorch as well. Also, I am pretty sure a JAX backend is coming to Keras, and there is TF Lite... I cannot make a prediction as to which stack will dominate in five years.
- adw 4y ago
  The relevant metric is the literature - adoption is driven by new models, and most new models in academia are being implemented in PyTorch or JAX. (In particular, TensorFlow 2 is nowhere to be seen.) The new grads joining the workforce are going to continue to use the tools they are comfortable with and new projects are built around new models.
  - grepLeigh 4y ago
    If you're familiar with the plumbing/porcelain API paradigm, JAX depends on TensorFlow plumbing (XLA) with a more ergonomic porcelain API. You might not see TensorFlow's plumbing much anymore if you're a new grad running experiments in a notebook, but the "porcelain API" is just the tip of the ice berg of modern machine learning. If you do any work on the JAX framework, you're frequently working with both the JAX and TensorFlow code repositories: https://github.com/google/jax/blob/main/WORKSPACE#L17 https://github.com/google/jax/blob/main/WORKSPACE#L17
  - zone411 4y ago
    Yes, I have been reading a lot of papers in the last couple of years and the corresponding code, if available, is using PyTorch much more often than TensorFlow.
fxj 4y ago
Hey! I am still using Fortran and love it. For me it is as fast as C, but much better to write and debug. But YMMV.
mmmmmbop 4y ago
George Hotz is a great hacker and problem solver, but I greatly doubt he has the patience to actually build a product. This is a fantastic skill set to solo-hack a complicated piece of software and produce a proof-of-concept, but not for building a tool that people actually use. See his track record of never building any successful product after his PS3 & iOS jailbreaks, his latest resignation memo from Comma.ai, and his resignation from Twitter after just 4 weeks. So I think it's kind of hilarious that you're citing him and his newest tinygrad project in one sentence with the assertion that PyTorch & friends will soon be considered ancient and clunky tools. While I don't necessarily dispute the latter, I'm certain that tinygrad won't be the one replacing them.
- goodside 4y ago
  That’s a lot of meanness just to refute a detail that was never asserted.
  - lkrubner 4y ago
    It's not mean, it's just a statement of a fairly obvious pattern in his career, which should influence how much weight we give to one of his predictions.
- kelipso 4y ago
  Why do we have so many of these overhyped personalities in the programming community? Guessing it's because half of the community have less than 5 years experience and younger people are more online but these personality cults are very offputting.
  - nathan11 4y ago
    Largely, because we're human. Nothing I've seen from other fields makes me think they're immune to it, or that devs are more susceptible to it. Some fields are literally defined/created by personality cults.
  - birdyrooster 4y ago
    I think because so many of us are invisible that when you have someone like George Hotz who received a ton of media exposure, you sort of are impressed that they did it or something?

KKKKkkkk1 4y ago

PyTorch and Tensorflow were good enough to implement all the major advanves in recent years: the AlexNet successors, GPT, Stable Diffusion. It's not early days, and the market will not evolve unless there is a new application that cannot be served by PyT and TF.

letitgo12345 4y ago

Theano preceded Tensorflow by 8 years. So I don't think it was all manual coding before TF. Torch (in Lua) was also around much before TF

gamegoblin 4y ago
That's definitely true, though worth noting that those libs didn't have anything close to the dominance the modern ones do. I was doing ML academia stuff in the pre-TF era, and most of my department was using custom C++/CUDA, or a pile of random stuff like https://en.wikipedia.org/wiki/Weka_(machine_learning) https://en.wikipedia.org/wiki/Weka_(machine_learning) and https://en.wikipedia.org/wiki/Waffles_(machine_learning) https://en.wikipedia.org/wiki/Waffles_(machine_learning). Our department also didn't even have GPUs to use for the first part of my time there, whereas now it would seem ridiculous to have an ML research wing without access to GPUs.
- quietbritishjim 4y ago
  I think the TensorFlow release roughly coincided with the widespread adoption of proper deep learning library, but it didn't cause it. In fact the first TensorFlow seemed to be just after (by 6 months to a year) the inflection point of deep learning library usage. At the time, there seemed to be a lively ecosystem but no clear winner. Theano, Caffe and Torch were the main hitters but there were smaller players too. Torch was particularly good, maybe functionally ahead (but I don't really remember for sure) but hampered because, at the time, it was Lua only. In fact Keras's first release said "Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python / Theano so as not to have to deal with the dearth of ecosystem in Lua." When TensorFlow was released, it looked like that would be the death of all the other libraries. The fact that any others survived at all is really down to how hard to use TensorFlow was (/is?). It didn't live up to the expectation of being the killer of other libraries.

vgatherps 4y ago

My work might make me biased but I think there’s a lot of room to make more general purpose code differentiable. I work on a ton of stuff that would very much benefit from being differentiable, but also very much can’t fit at all into the “stack a ton of linalg-like layers”. It’s a huge engineering effort to even try to think about how I might start taking out derivatives. It’s possible but there’s so much overhead to do anything with it.

wenc 4y ago
> make more general purpose code differentiable Non-smooth functions (e.g. abs(x)) can be handled with bundle methods, but how would one make inherently discontinuous (non-convex) functions differentiable? (e.g. if x then 1 else 5) Discrete problems are inherently non-differentiable. There are approaches like complementarity methods and switching functions (tanh) usually end up with numerical issues.
- wnkrshm 4y ago
  And then people go to empirical data and apply the Great Smoothing: By dropping ML/DL methods on the data (also including results from discontinuous behavior), continuity is often implicitly assumed.
- taylorius 4y ago
  This already happens to an extent in existing ML pipelines. The ReLU activation function, is discontinuous in its derivative, and is one of the most widely used functions in neural networks. Its derivative looks like this. if(i<0) return 0; else return 1; Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms. I think this is where the problem lies - not with differentiability itself, but with gradient descent not working due to the highly non-convex search space that such general programming constructs will produce.
  - wenc 4y ago
    ReLU is not discontinuous; it is nonsmooth but continuous, hence derivatives exist except at the hinge points. Inherently discontinuous functions OTOH are disconnected and nonconvex. Gradient descent works, but you have to add a step to first partition the discrete space like branch and bound. This involves solving the continuous relaxation to find a bound. This does not require differentiability (it is not differentiable), but the price to pay is that it is combinatorial (NP hard) The OP was talking about general differentiability but inherently discontinuous functions form a large and important class of functions (from software programming) that are not differentiable.

singularity2001 4y ago

one step in the right directions are labeled dimensions, where you can write input = width * hight * color * batch and never have to worry about tensor permutations again. experimental support in pytorch

amelius 4y ago
Is that part of the type system?
- singularity2001 4y ago
  maybe I misremembered; https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html
albertzeyer 4y ago
It's called named dimensions, named tensors, dimension tags: https://pytorch.org/docs/stable/named_tensor.html https://pytorch.org/docs/stable/named_tensor.html It's already there in PyTorch since a while (2019). However, as far as I know, it has not found too much adoption yet. In our framework RETURNN, we also make heavy use of dimension tags: https://returnn.readthedocs.io/en/latest/getting_started/data.html https://returnn.readthedocs.io/en/latest/getting_started/dat...
- CoastalCoder 4y ago
  I forget the details, but I think Nervana also had a library(?) that used this concept before they were bought by Intel.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

mattlondon 4y ago

> What will we have in 20 years? Whatever it is, I sincerely hope that it is not driven by Python, and instead we have a more accessible language. Say what you will about Python, but it is a total pain in the arse to work with. The 2.x legacy still casts a long shadow (in terms of online guides, unmaintained libs etc), anaconda, pip, easy install, strings/unicode, type hints (or not), unreadable & unmaintainable list comprehensions etc etc etc. What a dumpster fire, and a truly hateful language. Here is hoping that some other language takes over. I have my personal take, but I won't let that ruin the rant :)

deleted 4y ago
[deleted]
ergo14 4y ago
You are holding it wrong ;)
visarga 4y ago
Hey, I like list comprehensions, thank you very much.
- jjgreen 4y ago
  Bummer, list comprehension is deprecated, you should instead use a "virtual list" maintained by some kid who will shortly be losing interest ...
  - geysersam 4y ago
    That's a strange comment. Python encourages using its basic container types. Very unusual to use a third party library for something like that. I've never encountered a "virtual" anything in Python. That's more common in c++ or Java.
CoastalCoder 4y ago
I hate python for big software systems as much as the next guy, but for whatever reason some people are quite productive with it. I think programmers fall into different camps on this issue.

imiric 4y ago

> What will we have in 20 years? Hopefully something that doesn't require the user to be a math wizard. I'm not into data science by any means, and I first learned about tensors with Tensorflow. While I would like to experiment with neural nets, the current frameworks require me to learn the foundations of how they work. I'm sure I'll get arguments that this is a good thing, but I just want to do some cool stuff with computers, not take math courses to learn how the cake is made. High-level programming languages are abstractions to machine code that, for the most part, don't require the user to know how they work internally. I think we're missing similar abstractions for ML, or it's possible they already exist and I haven't looked hard enough.

cm2187 4y ago
The right analogy is cryptography. You don't need to understand the math behind cryptography to calculate a hash, or encrypt something. You just need to understand at a very high level what is asymmetric encryption, IV, etc.
- netheril96 4y ago
  That’s actually a bad analogy in my opinion. People who don’t learn cryptography make many mistakes calling cryptography libraries and those mistakes can’t be uncovered by testing.
  - bborud 4y ago
    Oh, it'll all be great when ChatGPT writes all that code for us and we can be off somewhere having a beer and a cry instead.
    - liotier 4y ago
      You'll cry but, to listen to your sorrow and offer friendly support, ChatGPT will be there for you !
      - 4y ago

thom 4y ago

So you mean we’ll still be compiling PyTorch as a dependency for various libraries for the next 40 years?

jackblemming 4y ago

>George Hotz, author of tinygrad[0], a PyTorch "competitor", made a really insightful comment -- we will look back on PyTorch & friends like we look back on FORTRAN and COBOL. Excuse me but, what? You could say this about pretty much every "hot" framework. Technology moves fast.

catwell 4y ago

> While PyTorch is obviously the future in the short term Well, that's not obvious for me. The metrics do not support the death of Keras / TF (https://twitter.com/fchollet/status/1614322127460782087 https://twitter.com/fchollet/status/1614322127460782087) François Chollet's Deep Learning with Python is arguably the best book on the topic and teaches people using Keras / TF2. The Keras documentation is excellent as well, and in the long run it will matter. Personally I prefer the way the Keras / TF2 APIs work to PyTorch as well. Also, I am pretty sure a JAX backend is coming to Keras, and there is TF Lite... I cannot make a prediction as to which stack will dominate in five years.

adw 4y ago
The relevant metric is the literature - adoption is driven by new models, and most new models in academia are being implemented in PyTorch or JAX. (In particular, TensorFlow 2 is nowhere to be seen.) The new grads joining the workforce are going to continue to use the tools they are comfortable with and new projects are built around new models.
- grepLeigh 4y ago
  If you're familiar with the plumbing/porcelain API paradigm, JAX depends on TensorFlow plumbing (XLA) with a more ergonomic porcelain API. You might not see TensorFlow's plumbing much anymore if you're a new grad running experiments in a notebook, but the "porcelain API" is just the tip of the ice berg of modern machine learning. If you do any work on the JAX framework, you're frequently working with both the JAX and TensorFlow code repositories: https://github.com/google/jax/blob/main/WORKSPACE#L17 https://github.com/google/jax/blob/main/WORKSPACE#L17
- zone411 4y ago
  Yes, I have been reading a lot of papers in the last couple of years and the corresponding code, if available, is using PyTorch much more often than TensorFlow.

fxj 4y ago

Hey! I am still using Fortran and love it. For me it is as fast as C, but much better to write and debug. But YMMV.

mmmmmbop 4y ago

George Hotz is a great hacker and problem solver, but I greatly doubt he has the patience to actually build a product. This is a fantastic skill set to solo-hack a complicated piece of software and produce a proof-of-concept, but not for building a tool that people actually use. See his track record of never building any successful product after his PS3 & iOS jailbreaks, his latest resignation memo from Comma.ai, and his resignation from Twitter after just 4 weeks. So I think it's kind of hilarious that you're citing him and his newest tinygrad project in one sentence with the assertion that PyTorch & friends will soon be considered ancient and clunky tools. While I don't necessarily dispute the latter, I'm certain that tinygrad won't be the one replacing them.

goodside 4y ago
That’s a lot of meanness just to refute a detail that was never asserted.
- lkrubner 4y ago
  It's not mean, it's just a statement of a fairly obvious pattern in his career, which should influence how much weight we give to one of his predictions.
kelipso 4y ago
Why do we have so many of these overhyped personalities in the programming community? Guessing it's because half of the community have less than 5 years experience and younger people are more online but these personality cults are very offputting.
- nathan11 4y ago
  Largely, because we're human. Nothing I've seen from other fields makes me think they're immune to it, or that devs are more susceptible to it. Some fields are literally defined/created by personality cults.
- birdyrooster 4y ago
  I think because so many of us are invisible that when you have someone like George Hotz who received a ton of media exposure, you sort of are impressed that they did it or something?