4 ms·
It's wild to me that, of all the things to call LLMs out for, this piece has chosen to include math tutoring. I've been doing Math Academy for a bit over 6 mont
by tptacek 1y ago
It's wild to me that, of all the things to call LLMs out for, this piece has chosen to include math tutoring. I've been doing Math Academy for a bit over 6 months now, going from (essentially) Algebra II through Calc II (integration by parts, arc lengths, Taylor expansions) and LLMs have been a huge part of what has made that effective:
* Clear explanation of concepts that respond to questions and reformulate when things bounce
* Step-by-step verification of solutions, spotting exactly where calculations have gone
* Instantaneously generating new problem sets to reinforce concepts
LLMs are probably not going to live up to all sorts of claims their proponents make. But I don't think you can ever have tried to use an LLM in a math course and reach the conclusion that it's "demoware" for that application. At what point, over 6 months of continuous work, does it stop being a "demo"?
- empath75 1y agoIt seems very hard to maintain the belief that LLMs are useless in the face of the fact that millions of people are using them. It's very much "nobody goes there anymore, it's too crowded"
- tptacek 1y agoI think you'd be crazy to say LLMs are blockchain-style hype when it comes to software development but I don't begrudge anybody who believes they're not currently workable for the kinds of problems they work on; I think reasonable people can disagree about how ready for prime time they are for production software development. But for math tutoring? If you claim LLM math tutoring is demoware, you're very clearly telling on yourself.
- tjr 1y agoI have seen LLMs fabricate bogus calculations; I personally would be hesitant to use an LLM as my one and only source of math learning, but I suppose using it in conjunction with something like Math Academy mitigates that issue? You've clearly had good success here, but any problem areas with the LLM to watch out for?
- badsectoracula 1y agoI wouldn't trust the LLM's raw output to be correct, but math is provable and if there was a filter between the LLM's output (which would be in some more rigid/structured format, not free form text) and whatever the user sees that tries to prove the LLM's output is correct (and try again if it goes wrong[0]), then i can see LLMs being perfectly fine for that. In fact i'd say in general anything that LLMs produce that can be "statically checked" in some way, can be fine to rely on. You most likely need more than a chat interface though, but i think in general it is plausible for such solutions to exit. [0] hopefully it wont end up always failing, ending up in an infinite loop :-P
- csmeyer 1y ago(OP) In my post, I actually ask the question of whether a student would _want_ to interact with the tutor, not if the tutor is capable of providing good instruction. These are drastically different critiques.
- rsynnott 1y agoOn that basis you'll also be adopting TCM, and homeopathy, and dowsing, and all major religions simultaneously, and all major fad diets simultaneously? Like, "lots of people like this thing and think it is helping them" is not terribly strong evidence that it is actually helping them. It's not a good argument.
- simonw 1y agoHow about "lots of people like this thing" where many of those people are credible professionals who I have respected from LONG before they started using LLMs?
- rsynnott 1y agoAgain, it's not _hugely_ strong evidence. Linus Pauling won _two_ Nobel prizes, was unquestionably brilliant... but that doesn't mean I'm going to start megadosing vitamin C anytime soon (https://en.wikipedia.org/wiki/Linus_Pauling#Medical_research_and_vitamin_C_advocacy https://en.wikipedia.org/wiki/Linus_Pauling#Medical_research...). (See also Newton and alchemy, and the list goes on.)
- simonw 1y agoThe experts I respect on this are people who's expertise is in software development, so when they say "this stuff helps me do what I do better and faster" I trust them more than if they said "I've started megadosing vitamin C and it's amazing". It helps that what they're discovering matches my own personal experience as well.
- kiitos 1y agothere are just as many equally-qualified experts who have been consistently giving exactly the opposite feedback/signal, which you never seem to acknowledge or incorporate into your comments
- deleted 1y ago[deleted]
- simonw 1y agoThis https://www.mathacademy.com/ https://www.mathacademy.com/ ? Interesting, hadn't seen that before. I've been thinking I'd like to brush up on a bunch of those topics.
- tptacek 1y agoWholeheartedly recommend it, just remember we're not the core market for it (that's high school students, though the curricula goes all the way through the normal college math sequence). Minutes later In case I've spooked anyone, they have an adult course series (Foundations I, II, and III) that's accelerated by trimming out all the material their authors believe are important only for things like school placement exams; the modal adult Math Academy person is doing I, II, and III as a leadup to their Math for Machine Learning course, which is linear algebra and multivariable calc. I think it's one of the three most mindblowing learning resources I've ever used. One of the other three: Lingua Latina Familia Romana. In both cases, I have the uncanny certainty that I am operating at the limit of my ability to acquire and retain new information, which is a fun place to be.
- getnormality 1y agoIt's nice that you think it's clear and responsive, but I think it [1] needs to be validated by an expert in both the material and education. Or we need some way to show that people have actually learned the topic. People sometimes prefer explanations that are intuitive and familiar but not accurate. Meanwhile, there are math education resources like iXL that maybe cost a little money but the lessons and practice problems are fully curated by human experts (AFAICT). I'm not saying these resources are perfect either, but as a mathematician who has experimented a lot with LLMs, including in supposed tutoring modes, they make a lot of mistakes and take a lot of shortcuts that should materially decrease their effectiveness as tutors. [1] LLM-based tutoring (edit: footnote added to clarify)
- simonw 1y agoWhat makes you think https://www.mathacademy.com/faq https://www.mathacademy.com/faq hadn't been evaluated by experts? That appears to be their whole thing, and they've been in business for longer than LLMs have been around.
- voidhorse 1y agoI think parent was clearly referring to LLM use, and not math academy.
- getnormality 1y agoI think before that question is useful to ask, we have to know if that FAQ even says anything about LLM-based tutoring. After a few minutes of research, I can't find any evidence that Math Academy offers LLM-based tutoring.
- simonw 1y agoThis was linked from the homepage: https://www.mathacademy.com/how-our-ai-works https://www.mathacademy.com/how-our-ai-works But more importantly if tptacek says they use LLMs and is a user of the platform that's good enough for me.
- pjc50 1y agoIs it always correct?
- AnotherGoodName 1y agoI've used LLMs to try to help digest some advanced maths. Eg. "Explain the number field seive with lots of numeric examples". Yes the numeric examples often don't work. The consequences of this though are similar to a failed web search. As in it's not a big deal and when it does work it's very helpful. Maths is one of those things with so much objectivity that even the LLM usually realizes it has failed to create a numeric example. "Here the numeric example breaks down since we cannot find a congruence of squares in this example without finding more B-smooth numbers in step 1". Ok that's a shame, i would have loved to see an end to end numeric example. I think people get too hung up on any possibility of LLMs not being perfect while still being extremely helpful.
- lomase 1y agoA LLM can't "realize" anything. Unless you are saying that LLMs are aware.
- AnotherGoodName 1y agoIt's a term i used to explain that in 'thinking' mode LLMs will read their own output and call out things like incorrect math statements before posting to the user. Now you probably want a debate about the term 'thinking' mode but i cbf with that. It's pretty clear what was meant and semantic arguments suck. Don't do that.
- voidhorse 1y agoGenerating problems is fantastic, but I'd caution on overreliance in the other two cases. Basically all of the cognitive science literature on learning that I am aware of says that the more you do directly and the less hand holding you are given, the better your acquisition and long term retention. In particular, having the LLM elaborate concepts for you is probably one of the worst things you can do when it comes to learning. Struggling through that elaboration process yourself is going to make the learning stick much more strongly, at least if all of the research is to be believed.
- tptacek 1y agoI understand that. The core of the pedagogical approach here is Math Academy, not LLMs. (Math Academy isn't an LLM; it's a spaced-repetition accelerated curriculum centered on graded problem set submissions). But the LLM functions exactly the way a tutor would in a math course, and for that application, LLMs have become extremely effective; arguably more effective than human tutors.
- dingnuts 1y agoIsn't this moving the goalposts? It's great that you're learning but MathAcademy appears to be a whole product that may incorporate an LLM but is much more, and it's a paid product none of us can evaluate. It's not possible to tell from looking at their site, or from your comment, what content is generated, or how it is verified before being used as teaching material. There are probably smart ways to incorporate LLM output into an application like the one you're lauding but your comment is a little like responding "but my cake tastes good" to someone who says you shouldn't eat raw flour.
- tptacek 1y agoYou're confused. Math Academy isn't LLM-based.
- unshavedyak 1y agoOfftopic, but do you have any comparison to math academy to something like Khan, or other platforms? MA seems a bit expensive for someone just wanting to improve a general skill, but perhaps it's well worth it? I thought Khan was also investing in similar AI offerings, so i'm curious how they intersect
- tptacek 1y agoKhan never clicked for me, and while the cost of Math Academy is below my noise floor (when you back it out to $/hr of engagement) as an adult professional in his prime earning years, I should also add that the cost is also a motivator: I've never been tempted to take a break, in part because I'm on the meter.
- benterix 1y agoWhile I agree, on an unrelated note - I knew I know your nick from somewhere... And then I realized[0]. [0] https://ludic.mataroa.blog/blog/contra-ptaceks-terrible-article-on-ai/ https://ludic.mataroa.blog/blog/contra-ptaceks-terrible-arti...
- tptacek 1y agoI had a conversation with that person a couple weeks ago. They're nice. I think we both would tweak (if just a little bit) how we presented our articles with the benefit of hindsight. For the record, I'm a systems programmer and a security person and I don't work for an AI company (you can Six Degrees of Sam Altman any startup to AI now if you want to make the claim, but if you try I'm just going to say "Sir, This Is A Wendy's".)
- kasey_junk 1y agoI on the other hand am the mental case that drinks rocket fuel.
- benterix 1y ago> I think we both would tweak (if just a little bit) how we presented our articles with the benefit of hindsight. Maybe you could present a joint statement of some kind, that would be interesting. I enjoy listening to the arguments of both camps and constantly comparing them to the actual state of things - and my conclusion is, sorry for the cliche, the only constant is change.
- j45 1y agoAbsolutely. This piece feels like a “I tried it out how I could” piece vs “I spent time learning how others are learning math with LLMs too” LLMs will make meaningful advances in personalized learning. Some of the frameworks might evolve along the way.
- thegrim33 1y agoSo .. a person who doesn't know X, is using LLMs to learn X, yet is able to judge that LLMs are doing a good job at teaching X, even though the person doesn't know X?
- tptacek 1y agoYou're confused. Math Academy is not an LLM.
- simonw 1y agoThere are many, many things in life where you can evaluate if you are learning the thing despite not having access to an expert guide who can verify what you are learning. Cooking: does the food taste better as you learn more? Programming: are you able to build functioning software that does what you want it to do, better than you could earlier on in your path? Fixing a broken dishwasher: does the dishwasher work again now? The idea that learning only works if you have an expert on hand to verify that you are learning is one of those things that seems obviously true until you think harder about it.