23 ms·
Sutton argues that algorithm development needs to be shaped by the assumption that compute power will continue growing exponentially into the future. At this po
by KKKKkkkk1 7y ago
Sutton argues that algorithm development needs to be shaped by the assumption that compute power will continue growing exponentially into the future. At this point, it is commonly believed that this is not the case.
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
- Veedrac 7y agoI'm fairly sure the consensus from the hardware side, especially considering how companies like NVIDIA, Intel and Arm are reacting, is that we still have a lot of exponential growth to go, especially for AI. A factor 1000 increase in model sizes seems within reason within 10ish years; I go over some arguments in the link below. https://www.reddit.com/r/MachineLearning/comments/ds1xvc/d_deep_learning_has_a_size_problem_we_need_to/f6nnq8d/ https://www.reddit.com/r/MachineLearning/comments/ds1xvc/d_d...