3 ms·

And I don't see any fundamental reason to believe that language models can learn or reason at all.

• sinenomine 4y ago
  If gradient-descent in DNNs indeed has an intrinsic bias for short and simple solutions, like this paper https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/2006.15191 shows, and if the "reasoning" is among the shortest solutions to match the textual data with the language modeling loss, then at some limit of model+data scaling the model has to recover this solution. Or maybe even something better, "reasoning++"? It's all too easy to dismiss the success of infinitely plastic, scalable systems.
  • mcguire 4y ago
    "For every complex problem there is an answer that is clear, simple, and wrong." Possibly said by H. L. Mencken. Also, "The fastest sea mammal is the sailfish, ... I am confident that the information I provided about the fastest sea mammal is accurate. The data on the speeds of different marine animals is widely available and has been widely studied and reported on by scientists and other experts."
• actually_a_dog 4y ago
  Then you need to provide a definition for "learn" and "reason" and an argument as to why. These models are trained to do the task they do, which is, by definition learning. I'll leave "reasoning" up to you, because that's a fuzzier question, but I still would like to see what your fundamental objection is.