3 ms·

Actually most of the scientific progress in these tasks had already been made and all that was missing was the compute power. Just like the know-how and basic p

by jsinai 6y ago

Actually most of the scientific progress in these tasks had already been made and all that was missing was the compute power. Just like the know-how and basic principles of how electric cars and have been around for more than a 100 years and what’s most been missing are cheap and powerful enough batteries. So it depends on how you measure progress? Scientifically and advancement of knowledge, or success in applying such knowledge to business problems and use cases. That the deep learning community insists on publishing most of these breakthroughs as research is the basis of the critique being made in the OP.

nl 6y ago
This isn't correct. No amount of compute power will let a RNN/LSTM outperform a Transformer. No amount of compute power will let an AlexNet or Inception style network train as well as a ResNet.
- anukriti 6y ago
  > I think the fact that we're seeing so much progress in deep learning without any theoretical basis I don't understand what you mean by this? Any neural network is first build in theory and then tried out practically by coding and testing. I think practitioner and scientist play equal role.
martindbp 6y ago
I think it was Nassim Taleb[1] that made a case for that engineering and tinkering drives science rather than the other way around, and that most of the knowledge gained comes from the practitioners rather than the scientists, who swoop in to claim credit after the fact (he uses the jet-engine as an example of this). That's not to say fundamental science is unimportant, just that it's getting too much of the credit for all the stuff that's been built. There's a tremendous amount of know-how required to successfully build an airplane, a jet-engine, a processor, or to successfully train a large neural network. And while theoretical physics no doubt can explain it all, it's not sufficient to build something that works. I think the fact that we're seeing so much progress in deep learning without any theoretical basis is another example of this. [1] Antifragile, Chapter: History Written By the Losers

by jsinai 6y ago

nl 6y ago

This isn't correct. No amount of compute power will let a RNN/LSTM outperform a Transformer. No amount of compute power will let an AlexNet or Inception style network train as well as a ResNet.

anukriti 6y ago
> I think the fact that we're seeing so much progress in deep learning without any theoretical basis I don't understand what you mean by this? Any neural network is first build in theory and then tried out practically by coding and testing. I think practitioner and scientist play equal role.

martindbp 6y ago

I think it was Nassim Taleb[1] that made a case for that engineering and tinkering drives science rather than the other way around, and that most of the knowledge gained comes from the practitioners rather than the scientists, who swoop in to claim credit after the fact (he uses the jet-engine as an example of this). That's not to say fundamental science is unimportant, just that it's getting too much of the credit for all the stuff that's been built. There's a tremendous amount of know-how required to successfully build an airplane, a jet-engine, a processor, or to successfully train a large neural network. And while theoretical physics no doubt can explain it all, it's not sufficient to build something that works. I think the fact that we're seeing so much progress in deep learning without any theoretical basis is another example of this. [1] Antifragile, Chapter: History Written By the Losers