3 ms·

Thank goodness I am retired and don't have to publish stuff in the current environment. Thirty years ago when machine learning was a sleepy backwater, life was

by 2sk21 5y ago

Thank goodness I am retired and don't have to publish stuff in the current environment. Thirty years ago when machine learning was a sleepy backwater, life was a lot easier for us.

sillysaurusx 5y ago
As a present-day researcher (or at least someone who aspires to be), I'm eagerly waiting for the hype to die down, and for ML to become a sleepy backwater again. You were lucky to play around with ML during those days. The neat thing is, the next AI winter, we'll still have massive hardware rigs. That's very different from 30 years ago. It's a lot easier to discover new things when you can just test every possibility overnight, rather than carefully planning. So if "winter is coming," it will be a lot less harsh this time.
- busterarm 5y ago
  That kind of compute still costs money. You still have to find the funding for it.
  - sillysaurusx 5y ago
    Nah, a motivated hacker can use a Colab TPU or GPU for free. It's how I started. There's an ungodly amount of resources available now compared to even 10 years ago, let alone 30. The bottleneck is usually motivation.
    - YeGoblynQueenne 5y ago
      Note that all the advances that eventually led to today's successes and applications of deep learning happened 20 or more years ago (e.g. LSTMs was in the '90s, the neocognitron in the '80s and so on). Then the hardware and the data came online and the field boomed, but no progress has been made in terms of new approaches and the overall state of research has stagnated with the inevitable results discussed in the article above (I can quote at least one Turing Award winner on that if that sounds just like me bloviating). In a sense, because it's now easy enough to get good results by throwing a bunch of data at a large computer, everyone's at it and nobody is looking for a way to obtain good results without a lot of data and a big computer. Which of course is not sustainable, not least because it means only large corporations can obtain state of the art results (and to be honest, I don't think any such have been obtained by "motivated hackers" using colab). No, my hope is that things will continue as they are now, until academics realise they've been kicked out of the game by the Big Players, and then go look for a way to compete that doesn't require gigantic amounts of compute. After all, academia can still motivate people to do actual research and find actually new things. The same motivation is much harder in industry, that only cares about one thing and only knows how to hand out one kind of reward.
- kordlessagain 5y ago
  “The hope is that the progress in hardware will cure all software ills. However, a critical observer may observe that software manages to outgrow hardware in size and sluggishness." https://en.m.wikipedia.org/wiki/Wirth%27s_law https://en.m.wikipedia.org/wiki/Wirth%27s_law
- deleted 5y ago
  [deleted]
- YeGoblynQueenne 5y ago
  >> The neat thing is, the next AI winter, we'll still have massive hardware rigs. So the next big thing in AI will still come from brute-forcing solutions by sheer power of compute. To clarify, you say this is a good thing?

by 2sk21 5y ago

Thank goodness I am retired and don't have to publish stuff in the current environment. Thirty years ago when machine learning was a sleepy backwater, life was a lot easier for us.

sillysaurusx 5y ago

As a present-day researcher (or at least someone who aspires to be), I'm eagerly waiting for the hype to die down, and for ML to become a sleepy backwater again. You were lucky to play around with ML during those days. The neat thing is, the next AI winter, we'll still have massive hardware rigs. That's very different from 30 years ago. It's a lot easier to discover new things when you can just test every possibility overnight, rather than carefully planning. So if "winter is coming," it will be a lot less harsh this time.

busterarm 5y ago
That kind of compute still costs money. You still have to find the funding for it.
- sillysaurusx 5y ago
  Nah, a motivated hacker can use a Colab TPU or GPU for free. It's how I started. There's an ungodly amount of resources available now compared to even 10 years ago, let alone 30. The bottleneck is usually motivation.
  - YeGoblynQueenne 5y ago
    Note that all the advances that eventually led to today's successes and applications of deep learning happened 20 or more years ago (e.g. LSTMs was in the '90s, the neocognitron in the '80s and so on). Then the hardware and the data came online and the field boomed, but no progress has been made in terms of new approaches and the overall state of research has stagnated with the inevitable results discussed in the article above (I can quote at least one Turing Award winner on that if that sounds just like me bloviating). In a sense, because it's now easy enough to get good results by throwing a bunch of data at a large computer, everyone's at it and nobody is looking for a way to obtain good results without a lot of data and a big computer. Which of course is not sustainable, not least because it means only large corporations can obtain state of the art results (and to be honest, I don't think any such have been obtained by "motivated hackers" using colab). No, my hope is that things will continue as they are now, until academics realise they've been kicked out of the game by the Big Players, and then go look for a way to compete that doesn't require gigantic amounts of compute. After all, academia can still motivate people to do actual research and find actually new things. The same motivation is much harder in industry, that only cares about one thing and only knows how to hand out one kind of reward.
kordlessagain 5y ago
“The hope is that the progress in hardware will cure all software ills. However, a critical observer may observe that software manages to outgrow hardware in size and sluggishness." https://en.m.wikipedia.org/wiki/Wirth%27s_law https://en.m.wikipedia.org/wiki/Wirth%27s_law
deleted 5y ago
[deleted]
YeGoblynQueenne 5y ago
>> The neat thing is, the next AI winter, we'll still have massive hardware rigs. So the next big thing in AI will still come from brute-forcing solutions by sheer power of compute. To clarify, you say this is a good thing?