4 ms·

Has anyone used any of the "intelligent tutoring systems" mentioned in this Wikipedia article?[0] Lots of comments here are saying "we'll have an intelligent t

Has anyone used any of the "intelligent tutoring systems" mentioned in this Wikipedia article?[0] Lots of comments here are saying "we'll have an intelligent tutoring system one day, based on LLMs" -- but there are many such systems that already claim to be getting good results now.[1] I'd be interested to hear from anyone who's used one. [0] https://en.wikipedia.org/wiki/Intelligent_tutoring_system https://en.wikipedia.org/wiki/Intelligent_tutoring_system [1] https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-digital-tutor-four-months-to-total-technical-expertise https://www.lesswrong.com/posts/vbWBJGWyWyKyoxLBe/darpa-digi...

trane_project 4y ago
Going through the list, it seems like all of them only work on very specialized subjects and/or are not published. I have read about the DARPA tutor, but I don't think it's been made publically available. So I doubt anyone here has used it, unless they were in that specific branch of the military at the time it was used. I don't see how LLMs deliver on this. I think they could speed up how the curriculum is generated, and I have played around with ChatGPT to do just that, but the questions only need to be generated once. It's the guiding of students through the exercises that needs to be personalized, not the exercises themselves. As for their effectiveness, I am the author of Trane (see my other comment). It's worked pretty well for me, but I am splitting my time between my job, working on the software, creating materials, and actual practice, so I can't really say what the upper limits are until I can focus on just the last part.
admsmz 4y ago
I am a customer of mathacademy.com, an intelligent tutoring system with a focus on math. Right now they are putting the finishing touches on their courses like abstract algebra and discrete mathematics. They already have linear algebra and calculus. It absolutely works as advertised and I highly recommend it.
- SunghoYahng 4y ago
  I took a look at the site you mentioned, but I don't see how it's useful for learning. It uses AI to determine what the learner's current level of knowledge is, but the way the learner learns seems completely typical. Then I may as well read classic textbooks. It doesn't seem to me to be that meaningful to determine the learner's current level of knowledge.
  - admsmz 4y ago
    Can you explain your objections? I don’t really understand this comment.
    - lern_too_spel 4y ago
      The benefit from 1-on-1 tutoring comes not just from finding what the student knows. There are any number of standardized tests that can determine that. Khan Academy does this for free and also adds lessons for mastery based learning. The benefit comes from understanding how the student misunderstood a concept being taught to them and then fixing that misconception.
      - travisjungroth 4y ago
        I think this claim is too strong. I’ve given and received more 1:1 instruction than most, plus my share of group instruction. The correcting misunderstanding is a big deal. Don’t get me wrong. The big downside is it’s very dependent on teacher ability. I’d say the bigger factor, and luckily one that scales well, is teaching the right thing to the student at that moment. You either have unmet dependencies of knowledge, needless repetition, or the right thing. You could also call this fast, slow, just right. There’s a huge variance of what people already know, even if they’ve been through the same classes. This means if you put a group of people together, the way to teach the most to a group is to teach at a crawl. You’re not teaching to the bottom of the class, you’re teaching to the bottom of the class at any given moment. 1:1 you can just fly in comparison because you can scale up and down the time per topic easily 10x for new material. If you include there may be review, it’s totally reasonable to think one person may need 1 minute (check) or 100 minutes (learn). The thing about this is that the math is really against you at any number greater than 1. Even 2 is a step change (also for social reasons). So trying to get a smaller class isn’t nearly as effective as springing for 1:1. I mean, you could even say it’s 2 SD better.

trane_project 4y ago

Going through the list, it seems like all of them only work on very specialized subjects and/or are not published. I have read about the DARPA tutor, but I don't think it's been made publically available. So I doubt anyone here has used it, unless they were in that specific branch of the military at the time it was used. I don't see how LLMs deliver on this. I think they could speed up how the curriculum is generated, and I have played around with ChatGPT to do just that, but the questions only need to be generated once. It's the guiding of students through the exercises that needs to be personalized, not the exercises themselves. As for their effectiveness, I am the author of Trane (see my other comment). It's worked pretty well for me, but I am splitting my time between my job, working on the software, creating materials, and actual practice, so I can't really say what the upper limits are until I can focus on just the last part.

admsmz 4y ago

I am a customer of mathacademy.com, an intelligent tutoring system with a focus on math. Right now they are putting the finishing touches on their courses like abstract algebra and discrete mathematics. They already have linear algebra and calculus. It absolutely works as advertised and I highly recommend it.

SunghoYahng 4y ago
I took a look at the site you mentioned, but I don't see how it's useful for learning. It uses AI to determine what the learner's current level of knowledge is, but the way the learner learns seems completely typical. Then I may as well read classic textbooks. It doesn't seem to me to be that meaningful to determine the learner's current level of knowledge.
- admsmz 4y ago
  Can you explain your objections? I don’t really understand this comment.
  - lern_too_spel 4y ago
    The benefit from 1-on-1 tutoring comes not just from finding what the student knows. There are any number of standardized tests that can determine that. Khan Academy does this for free and also adds lessons for mastery based learning. The benefit comes from understanding how the student misunderstood a concept being taught to them and then fixing that misconception.
    - travisjungroth 4y ago
      I think this claim is too strong. I’ve given and received more 1:1 instruction than most, plus my share of group instruction. The correcting misunderstanding is a big deal. Don’t get me wrong. The big downside is it’s very dependent on teacher ability. I’d say the bigger factor, and luckily one that scales well, is teaching the right thing to the student at that moment. You either have unmet dependencies of knowledge, needless repetition, or the right thing. You could also call this fast, slow, just right. There’s a huge variance of what people already know, even if they’ve been through the same classes. This means if you put a group of people together, the way to teach the most to a group is to teach at a crawl. You’re not teaching to the bottom of the class, you’re teaching to the bottom of the class at any given moment. 1:1 you can just fly in comparison because you can scale up and down the time per topic easily 10x for new material. If you include there may be review, it’s totally reasonable to think one person may need 1 minute (check) or 100 minutes (learn). The thing about this is that the math is really against you at any number greater than 1. Even 2 is a step change (also for social reasons). So trying to get a smaller class isn’t nearly as effective as springing for 1:1. I mean, you could even say it’s 2 SD better.