3 ms·
If gradient-descent in DNNs indeed has an intrinsic bias for short and simple solutions, like this paper https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/
by sinenomine 4y ago
If gradient-descent in DNNs indeed has an intrinsic bias for short and simple solutions, like this paper https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/2006.15191 shows, and if the "reasoning" is among the shortest solutions to match the textual data with the language modeling loss, then at some limit of model+data scaling the model has to recover this solution. Or maybe even something better, "reasoning++"?
It's all too easy to dismiss the success of infinitely plastic, scalable systems.
- mcguire 4y ago"For every complex problem there is an answer that is clear, simple, and wrong." Possibly said by H. L. Mencken. Also, "The fastest sea mammal is the sailfish, ... I am confident that the information I provided about the fastest sea mammal is accurate. The data on the speeds of different marine animals is widely available and has been widely studied and reported on by scientists and other experts."