4 ms·

While this article presents a system it doesn't present any results. Does this modification to SRS help? Which type of student does it help? How large is the ef

While this article presents a system it doesn't present any results. Does this modification to SRS help? Which type of student does it help? How large is the effect if there is one? I haven't read the larger pdf of which this is a part so perhaps someone who has can provide a pointer to some results.

JustinSkycak 2y ago
The main idea is that this approach makes spaced repetition feasible in something like mathematics. Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. Moreover, in addition to making spaced repetition feasible, it minimizes the amount of review (subject to the condition that you're getting sufficient review) which allows you to make really fast progress. We (Math Academy) don't have any official academic studies out at the moment, but if you want some kind of more concrete evidence of learning efficiency, you can read more online about our original in-school program in Pasadena where we have 6th graders start in Prealgebra, and then learn the entirety of high school math (Algebra 1, Geometry, Algebra 2, Precalculus) by 8th grade, and then in 8th grade they learn AP Calculus BC and take the AP exam. The AP scores started off decent while doing manual teaching, but the year we started using our automated system (of which the SRS described here is a component), the AP Calculus BC exam scores rose, with most students passing the exam and most students who passed receiving the maximum score possible (5 out of 5). Four other students took AP Calculus BC on our system that year, unaffiliated with our Pasadena school program, completely independent of a classroom, and all but one of them scored a perfect 5 on the AP exam (the other one received a 4). Even some seemingly impossible things started happening like some highly motivated 6th graders (who started midway through Prealgebra) completing all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) and starting AP Calculus BC within a single school year. Funny enough, the first time Jason & Sandy (MA founders) saw a 6th grader receiving AP Calculus BC tasks, Jason's reaction was "WTF is happening with the model, why is this kid getting calculus tasks, he placed into Prealgebra last fall, this doesn't make any sense," but I looked into it only to find that it was legit -- this kid completed all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) within a single school year. Anyway, some links if you're interested: * https://www.mathacademy.us/press https://www.mathacademy.us/press * https://www.reddit.com/r/homeschool/comments/16hn9f5/comment/k91ip0y https://www.reddit.com/r/homeschool/comments/16hn9f5/comment... * https://x.com/justinskycak/status/1810482435940913502 https://x.com/justinskycak/status/1810482435940913502 * https://x.com/justinskycak/status/1812557234028839193 https://x.com/justinskycak/status/1812557234028839193 Again, I realize these are not official academic studies, but we're completely overloaded in startup grind mode right now and have so many fish to fry with the product that we just don't have the time for academic pursuits at the moment, let alone much sleep. Happy to answer any follow-up questions that you might have, though.
- wantsanagent 2y ago
  Thanks for the detailed response! Fwiw I bet you could get some free academic labor just by offering to let a local PhD student have access to your internal data. (Obviously still not zero effort)
- krosaen 2y ago
  I'm excited about learning DAG + spaced repetition - fully sold this is an improvement over Anki > Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. I think this may be an overstatement. I've used Anki to learn and retain a lot of math including linear algebra (e.g worked through several chapters of Strang's books and its exercises). While it's not perfect, and I would love to have a DAG, what ends up happening in my experience is the more basic topics I understand and related topics end up being proven understood enough that they are backed off from review for a long time. So I might not be asked about a basic topic / problem for a year. This seems to prevent being overloaded with duplicate / repeat cards. However, if I do find that I am faced with a more advanced card that I have forgotten some of the foundations on, it is frustrating to not be able to easily be challenged smartly up through its ancestors. If I keep up with anki every day (10-20 minutes) this doesn't happen, but if I take 2-3 months off and come back to my deck, I can be faced with this problem and have to sometimes go manually digging for relevant background topics. So that's why I would love to have all this stuff get smarter, and am now reading / following you and math academy's work.
  - JustinSkycak 2y ago
    These are fair points; I guess the feasibility of spaced repetition comes down to what you consider a repetition. If you're considering a single quick problem on a topic to be a repetition, and you don't have too many topics, and you're not plowing through them too quickly, then I can see it being feasible as you're saying. Math Academy is built with a different context in mind: -- 1. we have tons of different topics (for instance, over 300 in our AP Calculus BC course -- and that's just one course; students who stick with us past their first course and continue taking more courses on our system can easily accumulate thousands of topics on which they need to maintain their knowledge) -- 2. each repetition consists of multiple questions on a topic (after an initial lesson task consisting of somewhere in the ballpark of 10 questions, future repetitions are review tasks in the range of 3-5 review questions) -- 3. students are getting through a new topic every 20 minutes or so on average (our AP Calculus BC course is estimated to take about 6000 minutes, and that includes time spent on review / quizzes / etc., and 6000 minutes / 300 topics = 20 minutes/topic) Just some back-of-the envelope math in our context: say you learn 3 new topics per day (an hour-long session), and each review takes you about 4 questions. Then, as a rough estimate, pretty quickly you reach a point where you've got 12 review questions based on yesterday's lessons, + another 12 review questions on topics you reviewed for the first time last week, + another 12 review questions on topics you reviewed for the second time a couple weeks ago, + another 12 review questions on topics you reviewed for the third time a month ago, + another 12 review questions on topics you reviewed for the fourth time a couple months ago, ... That's already 60 review questions/day, already past the point where you’re spending every hour-long session entirely on review, which means that your progress grinds to a halt in terms of learning new material. So, in our context at least, raw spaced repetition just doesn't work out in a way that’s feasible – student will get absolutely crushed by a tsunami of review unless we unless we take measures to drastically cut down the amount of review (i.e., fractional implicit repetition + repetition compression). I hear your point about needing some refresher if you take 2-3 months off and come back, though. Similar thing happens with Math Academy students if they take time off and come back, but the way we solve that problem is we just recommend they take a new diagnostic to refresh their knowledge profile. Basically, it just peels back their "knowledge frontier" to a point where they can pick up and continue learning smoothly with our standard approach. (In my experience, that’s what you’d ideally do as a teacher after summer vacation – you know your students have forgotten a lot of material, so you have them take a beginning-of-the-year knowledge evaluation and go from there.)

JustinSkycak 2y ago

The main idea is that this approach makes spaced repetition feasible in something like mathematics. Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. Moreover, in addition to making spaced repetition feasible, it minimizes the amount of review (subject to the condition that you're getting sufficient review) which allows you to make really fast progress. We (Math Academy) don't have any official academic studies out at the moment, but if you want some kind of more concrete evidence of learning efficiency, you can read more online about our original in-school program in Pasadena where we have 6th graders start in Prealgebra, and then learn the entirety of high school math (Algebra 1, Geometry, Algebra 2, Precalculus) by 8th grade, and then in 8th grade they learn AP Calculus BC and take the AP exam. The AP scores started off decent while doing manual teaching, but the year we started using our automated system (of which the SRS described here is a component), the AP Calculus BC exam scores rose, with most students passing the exam and most students who passed receiving the maximum score possible (5 out of 5). Four other students took AP Calculus BC on our system that year, unaffiliated with our Pasadena school program, completely independent of a classroom, and all but one of them scored a perfect 5 on the AP exam (the other one received a 4). Even some seemingly impossible things started happening like some highly motivated 6th graders (who started midway through Prealgebra) completing all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) and starting AP Calculus BC within a single school year. Funny enough, the first time Jason & Sandy (MA founders) saw a 6th grader receiving AP Calculus BC tasks, Jason's reaction was "WTF is happening with the model, why is this kid getting calculus tasks, he placed into Prealgebra last fall, this doesn't make any sense," but I looked into it only to find that it was legit -- this kid completed all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) within a single school year. Anyway, some links if you're interested: * https://www.mathacademy.us/press https://www.mathacademy.us/press * https://www.reddit.com/r/homeschool/comments/16hn9f5/comment/k91ip0y https://www.reddit.com/r/homeschool/comments/16hn9f5/comment... * https://x.com/justinskycak/status/1810482435940913502 https://x.com/justinskycak/status/1810482435940913502 * https://x.com/justinskycak/status/1812557234028839193 https://x.com/justinskycak/status/1812557234028839193 Again, I realize these are not official academic studies, but we're completely overloaded in startup grind mode right now and have so many fish to fry with the product that we just don't have the time for academic pursuits at the moment, let alone much sleep. Happy to answer any follow-up questions that you might have, though.

wantsanagent 2y ago
Thanks for the detailed response! Fwiw I bet you could get some free academic labor just by offering to let a local PhD student have access to your internal data. (Obviously still not zero effort)
krosaen 2y ago
I'm excited about learning DAG + spaced repetition - fully sold this is an improvement over Anki > Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. I think this may be an overstatement. I've used Anki to learn and retain a lot of math including linear algebra (e.g worked through several chapters of Strang's books and its exercises). While it's not perfect, and I would love to have a DAG, what ends up happening in my experience is the more basic topics I understand and related topics end up being proven understood enough that they are backed off from review for a long time. So I might not be asked about a basic topic / problem for a year. This seems to prevent being overloaded with duplicate / repeat cards. However, if I do find that I am faced with a more advanced card that I have forgotten some of the foundations on, it is frustrating to not be able to easily be challenged smartly up through its ancestors. If I keep up with anki every day (10-20 minutes) this doesn't happen, but if I take 2-3 months off and come back to my deck, I can be faced with this problem and have to sometimes go manually digging for relevant background topics. So that's why I would love to have all this stuff get smarter, and am now reading / following you and math academy's work.
- JustinSkycak 2y ago
  These are fair points; I guess the feasibility of spaced repetition comes down to what you consider a repetition. If you're considering a single quick problem on a topic to be a repetition, and you don't have too many topics, and you're not plowing through them too quickly, then I can see it being feasible as you're saying. Math Academy is built with a different context in mind: -- 1. we have tons of different topics (for instance, over 300 in our AP Calculus BC course -- and that's just one course; students who stick with us past their first course and continue taking more courses on our system can easily accumulate thousands of topics on which they need to maintain their knowledge) -- 2. each repetition consists of multiple questions on a topic (after an initial lesson task consisting of somewhere in the ballpark of 10 questions, future repetitions are review tasks in the range of 3-5 review questions) -- 3. students are getting through a new topic every 20 minutes or so on average (our AP Calculus BC course is estimated to take about 6000 minutes, and that includes time spent on review / quizzes / etc., and 6000 minutes / 300 topics = 20 minutes/topic) Just some back-of-the envelope math in our context: say you learn 3 new topics per day (an hour-long session), and each review takes you about 4 questions. Then, as a rough estimate, pretty quickly you reach a point where you've got 12 review questions based on yesterday's lessons, + another 12 review questions on topics you reviewed for the first time last week, + another 12 review questions on topics you reviewed for the second time a couple weeks ago, + another 12 review questions on topics you reviewed for the third time a month ago, + another 12 review questions on topics you reviewed for the fourth time a couple months ago, ... That's already 60 review questions/day, already past the point where you’re spending every hour-long session entirely on review, which means that your progress grinds to a halt in terms of learning new material. So, in our context at least, raw spaced repetition just doesn't work out in a way that’s feasible – student will get absolutely crushed by a tsunami of review unless we unless we take measures to drastically cut down the amount of review (i.e., fractional implicit repetition + repetition compression). I hear your point about needing some refresher if you take 2-3 months off and come back, though. Similar thing happens with Math Academy students if they take time off and come back, but the way we solve that problem is we just recommend they take a new diagnostic to refresh their knowledge profile. Basically, it just peels back their "knowledge frontier" to a point where they can pick up and continue learning smoothly with our standard approach. (In my experience, that’s what you’d ideally do as a teacher after summer vacation – you know your students have forgotten a lot of material, so you have them take a beginning-of-the-year knowledge evaluation and go from there.)