4 ms·

Is that meaningfully different from the study methods of the past? That doesn't require an LLM, and using one risks correctness in exchange for speed. You might

Is that meaningfully different from the study methods of the past? That doesn't require an LLM, and using one risks correctness in exchange for speed. You might not even get that speed if you're stuck in the weeds often enough. A decade ago, a google search for study guides written by another professor would have been slightly slower. A decade before that, you'd be even slower fumbling through several books. Every single word could at least be trusted. You don't get that from an LLM.

winnit 2mo ago
When studying a mathematics text, it is good practice to question what you are reading, trying to prove things to yourself etc., which IMO makes the problem of trusted sources much less than it is in things which you are not able to verify e.g. historical accounts.
Jtarii 2mo ago
If you have an actual mathematics professor to ask questions to then sure do that, but most people don't have that luxury. Also the error rate of frontier LLMs on textbook highschool/college level mathematics is going to be extremely low. It is also significantly more engaging and fun.
- tcp_handshaker 2mo ago
  >> the error rate of frontier LLMs on textbook highschool/college level mathematics is going to be extremely low. The scientific basis for this statement is unclear.
  - refactor_master 2mo ago
    AIME and HMMT are high school math olympiad competitions now used to benchmark AI. Frontier models score 95-99% https://benchlm.ai/blog/posts/aime-hmmt-competition-math https://benchlm.ai/blog/posts/aime-hmmt-competition-math
    - batshit_beaver 2mo ago
      They score 99% but then can’t explain _why_ their solution works with any accuracy. Not as useful for learning as one might think.
      - fl4regun 2mo ago
        are you referring to this section? > Verification is different from generation: Models scoring 98 can solve problems but can't always explain why their approach works at the level a human mathematician would. It doesn't say it can't explain why with any accuracy, it just says it can't *always* explain at the level of a mathematician, but most of us don't have such a mathematician at our beck and call to answer our questions anyways (thinking of the perspective of a self-learner outside of formal education)
        batshit_beaver 2mo ago
        It’s not just math. Anecdotally, LLMs struggle the same way with software engineering where the code they write is correct (compiles and passes tests), but reasoning is wrong often enough to eliminate most trust in these models’ ability to explain codebases or even features they themselves produce. It’s not about the _level_ of the supposed intelligence where a model struggles to summarize things succinctly or simply enough (responding to the “pocket mathematician” comment) or can’t grasp certain concepts at all (if so, how tf is it able to apply them?). It’s that by their design LLMs have no concept of truth and no concept of causality. They guess with every single inference and it’s very hard as a user to understand which guesses are more or less certain, since, you know, confidence ratings aren’t part of these models’ design either.
  - vonneumannstan 2mo ago
    >The scientific basis for this statement is unclear. Is it? Actually this seems like one of the MOST clear aspects of LLM performance we can measure.
- johnnyanmac 2mo ago
  >most people don't have that luxury. I wonder how long before LLMs need to cash in and even simple prompts are beyond the non-existent budget of a student. Which means it will need to be subsidized by education... and we're back in the teacher loop once again. Or perhaps libraries. That would be neat. >It is also significantly more engaging and fun. That won't last long either. I remember when phone apps were the "engaging and fun ways to learn". Half life of 2 years, and we're already seeing people lose the fun factor.
cornholio 2mo ago
> Is that meaningfully different from the study methods of the past? The fundamental service a teacher provides is personalized feedback, quickly identifying where you are stuck and focusing the explanations and exercises on that area, drastically increasing the speed and quality of learning versus the self-supervised route. The lack of this closed loop effectively killed the high hopes that were placed in e-learning and MOOCs 15-20 years ago, TV learning in the 1960s and many other failed revolutions, seems every generation has its own version. It appears to me LLMs have a real potential to close this loop and become the failed educational revolution of our own generation.
- alper 2mo ago
  > personalized feedback, quickly identifying where you are stuck and focusing the explanations and exercises on that area, This has been a huge blocker when I tried to study advanced math myself. Many of the exercise books don't have worked out answers, so often you're either stuck or you have to hunt a variety of sources online for solutions and advice. It kills flow.
defmacr0 2mo ago
I certainly would only trust my textbook as the authoritative source, but i can see that in the absence of an expert teacher it's nice to have something that can critique a proof. Imo the fact that it's hard to verify that your own proof is correct is one of the main barriers in self-studying math, especially if one is at a level where one is not completely fluent in applying the various techniques. This also applies to judging answers to open-ended questions in any other field.
- nicf 2mo ago
  I'm a private math tutor specializing in exactly this sort of material, and I agree with this very strongly. Knowing what "counts" as a proof is one of the most common gaps I see in students who come to me after self-studying, and most students do need some back-and-forth with an expert to really get that skill down. I imagine that LLM's could be very helpful for this if they were used judiciously!

winnit 2mo ago

When studying a mathematics text, it is good practice to question what you are reading, trying to prove things to yourself etc., which IMO makes the problem of trusted sources much less than it is in things which you are not able to verify e.g. historical accounts.

Jtarii 2mo ago

If you have an actual mathematics professor to ask questions to then sure do that, but most people don't have that luxury. Also the error rate of frontier LLMs on textbook highschool/college level mathematics is going to be extremely low. It is also significantly more engaging and fun.

tcp_handshaker 2mo ago
>> the error rate of frontier LLMs on textbook highschool/college level mathematics is going to be extremely low. The scientific basis for this statement is unclear.
- refactor_master 2mo ago
  AIME and HMMT are high school math olympiad competitions now used to benchmark AI. Frontier models score 95-99% https://benchlm.ai/blog/posts/aime-hmmt-competition-math https://benchlm.ai/blog/posts/aime-hmmt-competition-math
  - batshit_beaver 2mo ago
    They score 99% but then can’t explain _why_ their solution works with any accuracy. Not as useful for learning as one might think.
    - fl4regun 2mo ago
      are you referring to this section? > Verification is different from generation: Models scoring 98 can solve problems but can't always explain why their approach works at the level a human mathematician would. It doesn't say it can't explain why with any accuracy, it just says it can't *always* explain at the level of a mathematician, but most of us don't have such a mathematician at our beck and call to answer our questions anyways (thinking of the perspective of a self-learner outside of formal education)
      - batshit_beaver 2mo ago
        It’s not just math. Anecdotally, LLMs struggle the same way with software engineering where the code they write is correct (compiles and passes tests), but reasoning is wrong often enough to eliminate most trust in these models’ ability to explain codebases or even features they themselves produce. It’s not about the _level_ of the supposed intelligence where a model struggles to summarize things succinctly or simply enough (responding to the “pocket mathematician” comment) or can’t grasp certain concepts at all (if so, how tf is it able to apply them?). It’s that by their design LLMs have no concept of truth and no concept of causality. They guess with every single inference and it’s very hard as a user to understand which guesses are more or less certain, since, you know, confidence ratings aren’t part of these models’ design either.
- vonneumannstan 2mo ago
  >The scientific basis for this statement is unclear. Is it? Actually this seems like one of the MOST clear aspects of LLM performance we can measure.
johnnyanmac 2mo ago
>most people don't have that luxury. I wonder how long before LLMs need to cash in and even simple prompts are beyond the non-existent budget of a student. Which means it will need to be subsidized by education... and we're back in the teacher loop once again. Or perhaps libraries. That would be neat. >It is also significantly more engaging and fun. That won't last long either. I remember when phone apps were the "engaging and fun ways to learn". Half life of 2 years, and we're already seeing people lose the fun factor.

cornholio 2mo ago

> Is that meaningfully different from the study methods of the past? The fundamental service a teacher provides is personalized feedback, quickly identifying where you are stuck and focusing the explanations and exercises on that area, drastically increasing the speed and quality of learning versus the self-supervised route. The lack of this closed loop effectively killed the high hopes that were placed in e-learning and MOOCs 15-20 years ago, TV learning in the 1960s and many other failed revolutions, seems every generation has its own version. It appears to me LLMs have a real potential to close this loop and become the failed educational revolution of our own generation.

alper 2mo ago
> personalized feedback, quickly identifying where you are stuck and focusing the explanations and exercises on that area, This has been a huge blocker when I tried to study advanced math myself. Many of the exercise books don't have worked out answers, so often you're either stuck or you have to hunt a variety of sources online for solutions and advice. It kills flow.

defmacr0 2mo ago

I certainly would only trust my textbook as the authoritative source, but i can see that in the absence of an expert teacher it's nice to have something that can critique a proof. Imo the fact that it's hard to verify that your own proof is correct is one of the main barriers in self-studying math, especially if one is at a level where one is not completely fluent in applying the various techniques. This also applies to judging answers to open-ended questions in any other field.

nicf 2mo ago
I'm a private math tutor specializing in exactly this sort of material, and I agree with this very strongly. Knowing what "counts" as a proof is one of the most common gaps I see in students who come to me after self-studying, and most students do need some back-and-forth with an expert to really get that skill down. I imagine that LLM's could be very helpful for this if they were used judiciously!