3 ms·
Note that all the advances that eventually led to today's successes and applications of deep learning happened 20 or more years ago (e.g. LSTMs was in the '90s,
by YeGoblynQueenne 5y ago
Note that all the advances that eventually led to today's successes and applications of deep learning happened 20 or more years ago (e.g. LSTMs was in the '90s, the neocognitron in the '80s and so on).
Then the hardware and the data came online and the field boomed, but no progress has been made in terms of new approaches and the overall state of research has stagnated with the inevitable results discussed in the article above (I can quote at least one Turing Award winner on that if that sounds just like me bloviating). In a sense, because it's now easy enough to get good results by throwing a bunch of data at a large computer, everyone's at it and nobody is looking for a way to obtain good results without a lot of data and a big computer. Which of course is not sustainable, not least because it means only large corporations can obtain state of the art results (and to be honest, I don't think any such have been obtained by "motivated hackers" using colab).
No, my hope is that things will continue as they are now, until academics realise they've been kicked out of the game by the Big Players, and then go look for a way to compete that doesn't require gigantic amounts of compute. After all, academia can still motivate people to do actual research and find actually new things. The same motivation is much harder in industry, that only cares about one thing and only knows how to hand out one kind of reward.