3 ms·

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. There's no way for you or for an

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. There's no way for you or for anyone else to know that it's a dead end. Also, it's not "simply throwing more compute" - the algorithms have evolved too and they'll continue to. Also also, it's not "hopefully attain AI results" - we have AI results now and they keep getting better. People are spending resources on a trajectory which has worked well - we've been cranking up the compute and better results keep coming out, why stop now?

dpflan 2y ago
One must consider the concept of "diminishing returns" eventually. Has Generative AI Already Peaked? - Computerphile (https://www.youtube.com/watch?v=dDUC-LqVrPU https://www.youtube.com/watch?v=dDUC-LqVrPU)
- soared 2y ago
  Moores law is the inverse
  - Retric 2y ago
    Moores law doesn’t help when you’re spending money today rather than 10 years from now.
max_ 2y ago
All I am saying is that there are other strategies to explore. It's ok to have deep learning. But the herding towards deep learning is just too much. Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions" Scientific progress is made by researchers exploring other alternatives. Herding can only slow down progress.
- jerpint 2y ago
  The imagenet results’ “main” contribution was showing that scaling the compute on large datasets worked surprisingly well. They didn’t contribute any new models as much as they showed that scaling is what helped these models learn and obtain SOTA. It’s no surprise that industry has been pushing for more compute ever since.
- andsoitis 2y ago
  > All I am saying is that there are other strategies to explore. Are people not exploring other strategies?
  - max_ 2y ago
    Do you know anyone in AI doing research on something that isn't Neural Networks? They are very few. Too few.
- dxbydt 2y ago
  > All I am saying is that there are other strategies to explore The “other strategies” have been completely sidelined. Even back when we had no transformers, only CNNs and RNNs, the other strategies were tossed aside because they didn’t show much promise in cifar or mnist or whatever benchmarks the NN community deemed the most suitable sota. Even before that, when we only had decision trees and catboost, people had already dismissed loess/splines etc. You can get a flavor of the dismissals here - https://news.ycombinator.com/item?id=19145706 https://news.ycombinator.com/item?id=19145706 At this point that horse has left the barn.
- lostmsu 2y ago
  > Herding can only slow down progress. You don't know that either. What if there are actually no alternatives.
  - max_ 2y ago
    There are solutions. The human brain is one example. Isn't that what we are trying to simulate?
axus 2y ago
So much is being learned, and some stock prices have gone up. If nothing else, the people receiving coding suggestions and AI-generated artworks appreciate the billions of dollars and million programmer-hours spent.

dpflan 2y ago

One must consider the concept of "diminishing returns" eventually. Has Generative AI Already Peaked? - Computerphile (https://www.youtube.com/watch?v=dDUC-LqVrPU https://www.youtube.com/watch?v=dDUC-LqVrPU)

soared 2y ago
Moores law is the inverse
- Retric 2y ago
  Moores law doesn’t help when you’re spending money today rather than 10 years from now.

max_ 2y ago

All I am saying is that there are other strategies to explore. It's ok to have deep learning. But the herding towards deep learning is just too much. Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions" Scientific progress is made by researchers exploring other alternatives. Herding can only slow down progress.

jerpint 2y ago
The imagenet results’ “main” contribution was showing that scaling the compute on large datasets worked surprisingly well. They didn’t contribute any new models as much as they showed that scaling is what helped these models learn and obtain SOTA. It’s no surprise that industry has been pushing for more compute ever since.
andsoitis 2y ago
> All I am saying is that there are other strategies to explore. Are people not exploring other strategies?
- max_ 2y ago
  Do you know anyone in AI doing research on something that isn't Neural Networks? They are very few. Too few.
dxbydt 2y ago
> All I am saying is that there are other strategies to explore The “other strategies” have been completely sidelined. Even back when we had no transformers, only CNNs and RNNs, the other strategies were tossed aside because they didn’t show much promise in cifar or mnist or whatever benchmarks the NN community deemed the most suitable sota. Even before that, when we only had decision trees and catboost, people had already dismissed loess/splines etc. You can get a flavor of the dismissals here - https://news.ycombinator.com/item?id=19145706 https://news.ycombinator.com/item?id=19145706 At this point that horse has left the barn.
lostmsu 2y ago
> Herding can only slow down progress. You don't know that either. What if there are actually no alternatives.
- max_ 2y ago
  There are solutions. The human brain is one example. Isn't that what we are trying to simulate?

axus 2y ago

So much is being learned, and some stock prices have gone up. If nothing else, the people receiving coding suggestions and AI-generated artworks appreciate the billions of dollars and million programmer-hours spent.