5 ms·

> Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. I think it's more accurate t

> Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. I think it's more accurate to say that they learn math by memorizing a sequence of steps that result in a correct solution, typically by following along with some examples. Hopefully they also remember why each step contributes to the answer as this aids recall and generalization. The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly. This is just standard training. Understanding the motivation of each step helps with that memorization, and also allows you to apply that step in novel problems. > The original article is spot on that there is no AGI pathway in the current research direction. I think you're wrong. The research on grokking shows that LLMs transition from memorization to generalized circuits for problem solving if trained enough, and parametric memory generalizes their operation to many more tasks. They have now been able to achieve near perfect accuracy on comparison tasks, where GPT-4 is barely in the double digit success rate. Composition tasks are still challenging, but parametric memory is a big step in the right direction for that too. Accurate comparitive and compositional reasoning sound tantalizingly close to AGI.

shkkmo 2y ago
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Perhaps that is how you learned math, but it is nothing like how I learned math. Memorizing steps does not help, I sucked at it. What works for me us understanding the steps and why we used them. Once I understood the process and why it worked, I was able to reason my way through it. > The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly. Did you look at the types of problems presented by the ARC-AGO test? I don't see how memorization plays any role. > They have now been able to achieve near perfect accuracy on comparison tasks, where GPT-4 is barely in the double digit success rate. Then lets see how they do on the ARC test? While it is possible that generalized circuits can develop in Ls with enough training but I am pretty skeptical till we see results.
- naasking 2y ago
  > Perhaps that is how you learned math, but it is nothing like how I learned math. Memorization is literally how you learned arithmetic, multiplication tables and fractions. Everyone starts learning math by memorization, and only later start understanding why certain steps work. Some people don't advance to that point, and those that do become more adept at math.
  - pedrosorio 2y ago
    > Memorization is literally how you learned arithmetic, multiplication tables and fractions I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure". Also, I am not even sure what you mean by "memorization is how you learned fractions". What is there to memorize?
    - naasking 2y ago
      > I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure" What did you understand, exactly? You understood how to "count" using "numbers" that you also memorized? You intuitively understood that addition was counting up and subtraction was counting down, or did you memorize those words and what they meant in reference to counting? > Also, I am not even sure what you mean by "memorization is how you learned fractions". What is there to memorize? The procedure to add or subtract fractions by establishing a common denominator, for instance. The procedure for how numerators and denominators are multiplied or divided. I could go on.
      - shkkmo 2y ago
        Fractions is exactly an area of mathematics where I learned by understanding the concept and how it was represented and then would use that understanding to re-reason the procedures I had a hard time remembering. I do have the single digit multiplication table memorized now, but there was a long time where that table had gaps and I would use my understanding of how numbers worked to to calculate the result rather than remembering it. That same process still occurs for double digit number. Mathematics education, especially historically, has indeed leaned pretty heavily on memorization. That does mean thats the only way to learn math, or even a particularly good one. I personally think over reliance on memorization is part of why so many people think they hate math.
Vetch 2y ago
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes. Me and Terence Tao on the same exact math training data would not yield two mathematicians of similar skill. While it's true that memorization of properties, structure, operations and what should be applied when and where is involved, there is a much deeper component of knowing how these all relate to each other. Grasping their fundamental meaning and structure, and some people seem to be wired to be better at thinking about and picking out these subtle mathematical relations using just the description or based off of only a few examples (or be able to at all, where everyone else struggles). > I think you're wrong. The research on grokking shows that LLMs transition from memorization to generalized circuits It's worth noting that for composition, key to abstract reasoning, LLMs failed to generalize to out of domain examples on simple synthetic data. From: https://arxiv.org/abs/2405.15071 https://arxiv.org/abs/2405.15071 > The levels of generalization also vary across reasoning types: when faced with out-of-distribution examples, transformers fail to systematically generalize for composition but succeed for comparison.
- naasking 2y ago
  > Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes Everyone starts by memorizing how to do basic arithmetic on numbers, their multiplication tables and fractions. Only some then advance to understanding why those operations must work as they do. > It's worth noting that for composition, key to abstract reasoning, LLMs failed to generalize to out of domain examples on simple synthetic data. Yes, I acknowledged that when I said "Composition tasks are still challenging". Comparisons and composition are both key to abstract reasoning. Clearly parametric memory and grokking have shown a fairly dramatic improvement in comparative reasoning with only a small tweak. There is no evidence to suggest that compositional reasoning would not also fall to yet another small tweak. Maybe it will require something more dramatic, but I wouldn't bet on it. This pattern of thinking humans are special does not have a good track record. Therefore, I find the original claim that I was responding to("there is no AGI pathway in the current research direction") completely unpersuasive.

shkkmo 2y ago

> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Perhaps that is how you learned math, but it is nothing like how I learned math. Memorizing steps does not help, I sucked at it. What works for me us understanding the steps and why we used them. Once I understood the process and why it worked, I was able to reason my way through it. > The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly. Did you look at the types of problems presented by the ARC-AGO test? I don't see how memorization plays any role. > They have now been able to achieve near perfect accuracy on comparison tasks, where GPT-4 is barely in the double digit success rate. Then lets see how they do on the ARC test? While it is possible that generalized circuits can develop in Ls with enough training but I am pretty skeptical till we see results.

naasking 2y ago
> Perhaps that is how you learned math, but it is nothing like how I learned math. Memorization is literally how you learned arithmetic, multiplication tables and fractions. Everyone starts learning math by memorization, and only later start understanding why certain steps work. Some people don't advance to that point, and those that do become more adept at math.
- pedrosorio 2y ago
  > Memorization is literally how you learned arithmetic, multiplication tables and fractions I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure". Also, I am not even sure what you mean by "memorization is how you learned fractions". What is there to memorize?
  - naasking 2y ago
    > I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure" What did you understand, exactly? You understood how to "count" using "numbers" that you also memorized? You intuitively understood that addition was counting up and subtraction was counting down, or did you memorize those words and what they meant in reference to counting? > Also, I am not even sure what you mean by "memorization is how you learned fractions". What is there to memorize? The procedure to add or subtract fractions by establishing a common denominator, for instance. The procedure for how numerators and denominators are multiplied or divided. I could go on.
    - shkkmo 2y ago
      Fractions is exactly an area of mathematics where I learned by understanding the concept and how it was represented and then would use that understanding to re-reason the procedures I had a hard time remembering. I do have the single digit multiplication table memorized now, but there was a long time where that table had gaps and I would use my understanding of how numbers worked to to calculate the result rather than remembering it. That same process still occurs for double digit number. Mathematics education, especially historically, has indeed leaned pretty heavily on memorization. That does mean thats the only way to learn math, or even a particularly good one. I personally think over reliance on memorization is part of why so many people think they hate math.

Vetch 2y ago

> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes. Me and Terence Tao on the same exact math training data would not yield two mathematicians of similar skill. While it's true that memorization of properties, structure, operations and what should be applied when and where is involved, there is a much deeper component of knowing how these all relate to each other. Grasping their fundamental meaning and structure, and some people seem to be wired to be better at thinking about and picking out these subtle mathematical relations using just the description or based off of only a few examples (or be able to at all, where everyone else struggles). > I think you're wrong. The research on grokking shows that LLMs transition from memorization to generalized circuits It's worth noting that for composition, key to abstract reasoning, LLMs failed to generalize to out of domain examples on simple synthetic data. From: https://arxiv.org/abs/2405.15071 https://arxiv.org/abs/2405.15071 > The levels of generalization also vary across reasoning types: when faced with out-of-distribution examples, transformers fail to systematically generalize for composition but succeed for comparison.

naasking 2y ago
> Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes Everyone starts by memorizing how to do basic arithmetic on numbers, their multiplication tables and fractions. Only some then advance to understanding why those operations must work as they do. > It's worth noting that for composition, key to abstract reasoning, LLMs failed to generalize to out of domain examples on simple synthetic data. Yes, I acknowledged that when I said "Composition tasks are still challenging". Comparisons and composition are both key to abstract reasoning. Clearly parametric memory and grokking have shown a fairly dramatic improvement in comparative reasoning with only a small tweak. There is no evidence to suggest that compositional reasoning would not also fall to yet another small tweak. Maybe it will require something more dramatic, but I wouldn't bet on it. This pattern of thinking humans are special does not have a good track record. Therefore, I find the original claim that I was responding to("there is no AGI pathway in the current research direction") completely unpersuasive.