14 ms·

Individualized Spaced Repetition in Hierarchical Knowledge Structures

178 points by JustinSkycak 2y ago

hiAndrewQuinn 2y ago
Now these are the insights I'm looking for when it comes to "second generation" SRS software suites. When you focus your attention on a specific niche, like mathematics here, you can get some serious improvements in retention just by using the shape of the subject to your advantage like this.
AlchemistCamp 2y ago
Anki contributor here: I’ve been using SRS on and off since 2006 and this post addresses one of the key failure modes I’ve seen in SRS usage—they tend to be fantastic for discrete pieces of information, like the names of capital cities, the pronunciation of various symbols or various physical constants. Most my usage of SRS in the early days was languages. For that domain, SRS is useful for learning an alphabet (or syllabary), for scaffolding enough common words to start understanding graded readers and for a few other tasks. They’re terrible as a primary learning strategy, though! Too much of language is highly nuanced and context dependent and the only way to absorb all the unwritten rules, the common collocations and the precise boundaries of word meanings is through extensive input. This means encountering the same words and structures in countless different variations and contexts, not the same few sentences drilled repeatedly (at least past the beginning stages). By paying attention to the specifics of learning math and tailoring the SRS to match it, it’s possible to make a far more efficient system than anyone could with an Anki deck. I’d love to see similar efforts in other subjects!
- dotancohen 2y ago
  Anki is great for building vocabulary. It's not suitable for actually learning the language. I use it constantly, averaging over 200 cards a day for over fifteen years.
  - kiba 2y ago
    I wish I have the same dedication, but alas Anki hadn't sufficiently proven useful to me that I would use it on a day to day basis.
    - dotancohen 2y ago
      If you have an Android device, go install Ankidroid. It's absolutely terrific, letting you fill in those wasted seconds when you're waiting in line, sitting on the toilet, waiting for the pot to boil. Two or three cards at a time, dozens of times per day, adds up. You could treat yourself to three cards after every git commit, for instance. Takes literally less than ten seconds to review three cards.
      - kiba 2y ago
        The problem isn't the convenience but that the value in my deck hasn't sufficiently manifested itself in real world applications. This is my own fault rather than the app's fault.
        Bluestein 2y ago
        I both second Ankidroid (oustanding) ... ... and concur when failing (as of yet) to find a way to use the system with other types of knowledge, sadly.-
  - boppo1 2y ago
    How many new cards a day do you set?
    - dotancohen 2y ago
      Depends on what I'm actively learning at any given time. Right now it's 4, sometimes it could be closer to 20.
- gwd 2y ago
  A couple of problems I found with Anki for languages: 1. The words aren't in context, which leads to "flashcard blindness", where you see a word in context, know that you recognize it, but can't remember what it means. 2. The theory behind spaced repetition is that you learn most efficiently when you try to remember something just when you're about to forget it. But that means that if the algorithm is working properly, every single card is hard. This makes motivation a problem, because you know studying is going to be a grind, not fun. 3. While "being able to remember it after thinking a second or two" might be fine when studying for an exam, or in many other contexts where memorization might be important, that's too slow for languages. What you want for a language is "know it immediately without having to think about it". 4. The "scheduled review" system is too inflexible. Some days you get only a handful of cards to review, some days you get dozens. It's hard to tell when you're starting out how many cards you should be adding each day such that the number of cards match the amount of time / effort you have to study. Furthermore, if you skip a single day, you have twice as much the next day; and if life happens and you end up missing a week or a month, you come back with a giant jumble of cards, half of which you've forgotten, and it's really difficult to dig your way out of it. By a strange coincidence, in 2019 I also started on an alternate to Anki to help myself study Mandarin. It has some similarities to his system, in that there's multi-level knowledge; but it's different in that instead of having a fixed schedule, it has the concept of "difficulty" and "study value" for each word / grammar concept, and the algorithm tries to give you a full "readunit" of "native input" which will balance the two. "Spaced repetition" emerges naturally from the model, and if you go away for a week (or 6 months), it knows you've forgotten some things, so it gradually refreshes your memory. And because you're reading actual native text, there's something which pulls you in. It's in closed beta now; the first public language (MVP) will be in Biblical Greek, but the second one (if it happens, maybe in a year or two) will be in Mandarin; and hopefully there will be other ones after that. There's a sign-up form you can use to be notified for updates. https://www.laleolanguage.com https://www.laleolanguage.com
  - Bluestein 2y ago
    > But that means that if the algorithm is working properly, every single card is hard. Could the algo be tweaked so as to regulate the "closeness to the edge", ergo, the difficulty? PS. Then again, something tells me what makes SR so effective *might just be* (or be related to) that "difficulty".-
- jwells89 2y ago
  For this reason, I’ve seen it recommended in various language learning guides to use Anki in a manner that’s complementary to consumption of media in the target language. The general idea is to do Anki-only for the earliest bootstrapping stages (super basic vocab) and switch to media+Anki as early as possible. The media gives both context and real experience with the language while Anki serves to commit the words you’ve read/heard (even those not used quite as commonly) to memory. I’ve only just dipped my toe into this method but have seen many reports of success with it from others.
  - bunderbunder 2y ago
    You might find, as you get deeper into the language learning journey, that this approach to practice is most popular among an extremely vocal group of intermediate language learners. But people who've been doing it for a long time tend to gravitate toward the parent poster's opinion. I've also noticed that assiduously sentence mining your input and then drilling it in Anki is much more popular among people whose media consumption is almost exclusively video content. This resonates with the work of SLA researchers such as Paul Noble who've found that extensive reading is easily the fastest way to grow vocabulary in a new language. There's even some (scant) evidence that SRS combined with extensive reading is less effective than extensive reading alone for long-term vocabulary growth and retention. It suggests, to me at least, that heavy reliance on SRS for vocabulary retention past the beginner stages of learning a language may not be a universal optimal path, so much as a backup option for people who don't enjoy reading. Like AlchemistCamp I do still like it a lot for specific things, and I'm not here to wag fingers at people who enjoy it (Above all else, make sure you're enjoying your study routine! Getting there should be 100% of the fun.), but I do still worry that many Anki users have a serious case of golden hammer syndrome.
    - jwells89 2y ago
      For sure, by no means am I married to the method and am very flexible and open to trying various approaches. I do believe however that a little bit of structure can be helpful in the beginner to intermediate levels, where it's less likely that one has formed a habit/urges to watch/listen to foreign media without subtitles or read untranslated print media. It's a real slog when you don't have a decent arsenal of words to work with.
isaacfrond 2y ago
There lots of knowledge types where Anki might not be the ideal choice. For example, - itemized lists of information. E.g. The 5 reasons for the civil war, type of questions. What to do, if you miss one item? Your card as a whole gets over exposed, while the missed item might get underexposed. - Related information, e.g., history dates. When using Anki the students tends to latch on to incidental facts, e.g., Treaty of Paris, ah I remember it has two repeating digits, yes, 1899. While it would be more useful for a student to think, ah, that was after the Cuban independence struggles in the 1890's. Any user will recognize that this effect is very strong. You tend to remember cards by the most trivial of things. - Hierarchical knowledge. Things like chess openings. How do you put that in Anki cards. It's all a kludge. Where to put the variants, etc. - Knowledge networks. Things like medical information (where Anki is hugely popular). But typically, you get cards with massive amounts of information because you have lots of linked information, (symptoms, causes, treatment, pharmaceuticals, etc. ) - Even in language learning. We have the useful fiction that there is a 1-1 relationship between words in two languages, but there almost never is. - Learned knowledge tends to be linked to Anki. It happens often that you can remember things when doing Anki, while being at a loss when needing the information in the real world. Add to that, that Anki does nothing for conceptual understanding. You really need to learn a subject before memorizing it, but in the learning phase Anki is not helpful. So in short, yes, Anki is the best tool there is to help learning, but I'm sure better tools can be made, especially when targeted to a particular knowledge field.
- gwervc 2y ago
  > - Even in language learning. We have the useful fiction that there is a 1-1 relationship between words in two languages, but there almost never is. It's not really a problem: space repetition is a way to build familiarity with vocabulary. Once it is done, more finer uses can be inferred from context. The value lies in breadth not depth.
  - gwd 2y ago
    A couple of problems I found with Anki for languages: 1. The words aren't in context, which leads to "flashcard blindness", where you see a word in context, know that you recognize it, but can't remember what it means. 2. The theory behind spaced repetition is that you learn most efficiently when you try to remember something just when you're about to forget it. But that means that if the algorithm is working properly, every single card is hard. This makes motivation a problem, because you know studying is going to be a grind, not fun. 3. While "being able to remember it after thinking a second or two" might be fine when studying for an exam, or in many other contexts where memorization might be important, that's too slow for languages. What you want for a language is "know it immediately without having to think about it". 4. The "scheduled review" system is too inflexible. Some days you get only a handful of cards to review, some days you get dozens. It's hard to tell when you're starting out how many cards you should be adding each day such that the number of cards match the amount of time / effort you have to study. Furthermore, if you skip a single day, you have twice as much the next day; and if life happens and you end up missing a week or a month, you come back with a giant jumble of cards, half of which you've forgotten, and it's really difficult to dig your way out of it. By a strange coincidence, in 2019 I also started on an alternate to Anki to help myself study Mandarin. It has some similarities to his system, in that there's multi-level knowledge; but it's different in that instead of having a fixed schedule, it has the concept of "difficulty" and "study value" for each word / grammar concept, and the algorithm tries to give you a full "readunit" of "native input" which will balance the two. "Spaced repetition" emerges naturally from the model, and if you go away for a week (or 6 months), it knows you've forgotten some things, so it gradually refreshes your memory. It's in closed beta now; the first public language (MVP) will be in Biblical Greek, but the second one (if it happens, maybe in a year or two) will be in Mandarin; and hopefully there will be other ones after that. There's a sign-up form you can use to be notified for updates. https://www.laleolanguage.com https://www.laleolanguage.com
hereme888 2y ago
This is why my initial flashcards on a topic would be cover a breadth of individual details, but later I'd start creating concept based flashcards where I'd ask things like "how does it make sense that....?" After somewhat memorizing foundational knowledge, understanding concepts and having frameworks of thinking for a topic will further increase critical thinking while reducing the amount of cognitive effort required for critical thinking about unfamiliar situations. I hope one day these more efficient models that "clean up" the review schedule by linking/identifying related flashcards become publicly available.
azeirah 2y ago
I really hope to see more research like this. I feel like LLMs have insanely high pedagogic value, and that's because I've been using them to teach myself difficult subjects and review my understanding. The issue with the kind of system that the author is proposing for curious self-driven learning is that the SRS is optimized for a given curriculum. Many people including myself use flashcards to guide (and sort of "create") their own curriculum. Flashcards with SRS are really good for many things, but it's difficult to generalize them for thus usecase. I'd really like to see some prototypes of people integrating LLM intelligence in creating and adjusting cards on the fly. It's clearly something LLMs excel at, especially the larger closed-source ones like Claude 3.5 sonnet
- kiba 2y ago
  I really hope to see more research like this. I feel like LLMs have insanely high pedagogic value, and that's because I've been using them to teach myself difficult subjects and review my understanding. How do you know if the LLM isn't hallucinating?
  - EnigmaFlare 2y ago
    For mathy subjects you usually have to verify things yourself so you'd detect any mistakes as you try to work things out or makes sense of the information. You can't learn math, physics or theory for engineering by remembering and believing things. It would be different for "bag of random facts" subjects though.
  - knighthack 2y ago
    This fear that LLMs are hallucinating constantly, and therefore entirely unreliable, is unfounded. Sure, that'll happen if you're going into extreme limits of areas of knowledge - but if you're dipping your foot into superficial waters of known areas, it's reasonably fine (and perhaps even more sound than the hotter areas of Wikipedia). E.g. if you're programming in the most well-known languages (say Python), and you're only asking beginner questions (which is where high pedagogic value lies) it's unlikely that the program will hallucinate all that deeply to mislead you. Yes, you won't know, but simple/mid questions are easy to check. You shouldn't of course head to LLMs for the most arcane scientific/mathematical knowledge and conclusions; if you do that, that's on you, because offering expertise is not what LLMs are designed for, and the disclaimers all warn against it. But you shouldn't throw out the baby with the bathwater.
ofirg 2y ago
Our company (Language Zen) has this patent https://patents.google.com/patent/EP1891619A2/en https://patents.google.com/patent/EP1891619A2/en and we have done work to teach hierarchical knowledge for language learning
- ofirg 2y ago
  sorry wrong patent link: https://www.patentsencyclopedia.com/app/20160035249 https://www.patentsencyclopedia.com/app/20160035249
  - deleted 2y ago
    [deleted]
- AlchemistCamp 2y ago
  How revolting. Nothing in what was described in your updated link below looks even remotely novel even years before the 2016 filing date. I say this as someone who was deeply involved in the industry in the late 2000s. I also contributed to Anki in that time frame.
  - rmbyrro 2y ago
    This patent is just for the business owners vanity. It's clearly not defensible in courts. Even if it was, they probably don't have the money required to pursue court enforcement of the patent. More so when any reasonable lawyer will tell them they'll get their case dismissed anyway.
  - blackbrokkoli 2y ago
    Yeah I don't understand how this can be a patent. For example Skinner's Learning Machine as well as any automatic implementation of Pimsleur's method, both described in the SIXTIES, would fall under this (according to my reading, I'm not a patent lawyer, hamdulillah). Absurd.
iandanforth 2y ago
While this article presents a system it doesn't present any results. Does this modification to SRS help? Which type of student does it help? How large is the effect if there is one? I haven't read the larger pdf of which this is a part so perhaps someone who has can provide a pointer to some results.
- JustinSkycak 2y ago
  The main idea is that this approach makes spaced repetition feasible in something like mathematics. Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. Moreover, in addition to making spaced repetition feasible, it minimizes the amount of review (subject to the condition that you're getting sufficient review) which allows you to make really fast progress. We (Math Academy) don't have any official academic studies out at the moment, but if you want some kind of more concrete evidence of learning efficiency, you can read more online about our original in-school program in Pasadena where we have 6th graders start in Prealgebra, and then learn the entirety of high school math (Algebra 1, Geometry, Algebra 2, Precalculus) by 8th grade, and then in 8th grade they learn AP Calculus BC and take the AP exam. The AP scores started off decent while doing manual teaching, but the year we started using our automated system (of which the SRS described here is a component), the AP Calculus BC exam scores rose, with most students passing the exam and most students who passed receiving the maximum score possible (5 out of 5). Four other students took AP Calculus BC on our system that year, unaffiliated with our Pasadena school program, completely independent of a classroom, and all but one of them scored a perfect 5 on the AP exam (the other one received a 4). Even some seemingly impossible things started happening like some highly motivated 6th graders (who started midway through Prealgebra) completing all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) and starting AP Calculus BC within a single school year. Funny enough, the first time Jason & Sandy (MA founders) saw a 6th grader receiving AP Calculus BC tasks, Jason's reaction was "WTF is happening with the model, why is this kid getting calculus tasks, he placed into Prealgebra last fall, this doesn't make any sense," but I looked into it only to find that it was legit -- this kid completed all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) within a single school year. Anyway, some links if you're interested: * https://www.mathacademy.us/press https://www.mathacademy.us/press * https://www.reddit.com/r/homeschool/comments/16hn9f5/comment/k91ip0y https://www.reddit.com/r/homeschool/comments/16hn9f5/comment... * https://x.com/justinskycak/status/1810482435940913502 https://x.com/justinskycak/status/1810482435940913502 * https://x.com/justinskycak/status/1812557234028839193 https://x.com/justinskycak/status/1812557234028839193 Again, I realize these are not official academic studies, but we're completely overloaded in startup grind mode right now and have so many fish to fry with the product that we just don't have the time for academic pursuits at the moment, let alone much sleep. Happy to answer any follow-up questions that you might have, though.
asoneth 2y ago
More than a decade ago I worked as a research programmer on a similar AI tutoring product for hierarchical skills like mathematics at Carnegie Mellon (in conjunction with the Pittsburgh Science of Learning Center and Carnegie Learning). The system would prompt students with problems that incorporated dozens of sub-skills, each of which incorporated other sub-skills and so on. If the student gets the problem right, all requisite sub-skills are marked as reasonably strong. If they get it wrong then the statistical model concludes that at least one sub-skill must be weak/faulty/missing, and when selecting subsequent problems the system would incorporate problems with different subsets of the original sub-skills to identify the missing sub-skill(s), with the goal being to spend time reinforcing only those skills that the student has not mastered. I no longer work in that domain, but I recall it being amazingly efficient -- it often only took a dozen questions to assess a student's understanding of thousands of skills.
- BenoitP 2y ago
  Seems quite close to the 'active learning' ML topic: let the model choose the question that will bring the best information return. And this kind of decomposition using simple knowledge pieces (logical axioms?) is IMHO what we have to do to bring LLMs to senses. These pieces should light up in the intermediate embeddings inside the LLM. It won't really be more intelligent, but it'll model reality better.
- owenpalmer 2y ago
  Reminds me of using git-bisect
- ahmni_dev 2y ago
  Agreed that this article is strongly related to Intelligent Tutoring Systems like the Carnegie Tutor. I recently posted a brief literature review video of the field for those interested in the bigger picture around the article: https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz The TLDR of the relevant sections (see video description for relevant timestamps) - Models like ACT-R represent a model of the domain and a model of the student - Knowledge tracing is an ML-heavy field that predicts skills, state, etc (like parent says) - Recommender Systems use that knowledge to intelligently recommend next problems
graboid 2y ago
Say you have concepts/items/cards A, B and C, with A -> B -> C (C encompasses B, B encompasses A, keeping the notation from the article). As I understand it, the article advocates for showing C first, then you can assume that you also know B and A to at least some part, and save yourself the repetitions for these. Intuitively, I would have guessed the opposite approach to be the best: Show A first, suspend B until A is learned (by some measure), then show B, etc. That means no repetitions to skip, but also you get less failures (and thus additional repetitions) that occur as follows: you are shown C, but don't know B anymore, and thus cannot answer and have to repeat C. If you are shown C before B, you kinda make C less atomic (you might have to actively recall both, B and C to answer it), showing B before C makes C more atomic, as you will have B more mentally present/internalized and can focus on what C adds to B.
- JustinSkycak 2y ago
  Hey! Good questions. I'll try to clarify: 1. First want to clarify that the learner is first introduced to the topics through mastery learning (i.e., not given a topic until they've seen and mastered the prereqs). So, they would explicitly learn A before learning B, and explicitly learn B before learning C. It's only in the review phase when we do all this stuff with "knocking out" repetitions implicitly. 2. When you say "then you can assume that you also know B and A to at least some part," I want to emphasize that if C encompasses B and B encompasses A in the sense of a full encompassing that would account for a full repetition, then doing C fully exercises B and A as component skills. Not just exercises them "to some part." For instance, topic C might be solving equations of the form "ax+b=cx+d," topic B might be solving equations "ax+b=c," and topic A might be solving equations "ax=b." 3. This scenario should never happen: "you are shown C, but don't know B anymore, and thus cannot answer and have to repeat C." There are both theoretical and practical safeguards. 3a-- Theoretical: if you are at risk of forgetting B in the near future, then you'll have a repetition due on B right now, which means you're going to review it right now (by "knocking it out" with some more advanced topic if possible, but if that's not possible, we're going to give you an explicit review of B itself. In general, if a repetition is due, we're not going to wait for an "implicit knock-out" opportunity to open up and let you forget it while we wait. We'll just say "okay, guess we can't knock this one out implicitly, so we'll give it to you explicitly." 3b-- Practical: suppose that for whatever reason, the review timing is a little miscalibrated and a student ends up having forgotten more of B than we'd like when they're shown C. Even then, they haven't forgotten B completely, and they can refresh on B pretty easily. Often, that refresher is within C itself: for instance, if you're learning to solve equations of the form "ax+b=cx+d," then the explanation is going to include a thorough reminder of how to solve "ax+b=c." And even in other cases where that reminder might not be as thorough, if you're too fuzzy on B to follow the explanation in C, then you can just refer back to the content where you learned B and freshen up: "Huh, that thing in C is familiar but it involves B and I forgot how you do some part of B... okay, look back at B's lesson... ah yeah, that's right, that's how you do it. Okay, back to C." And then the act of solving problems in C solidifies your refreshed memory on B. Anyway, I think I've clarified all your questions? But please do let me know if you have any follow-up questions or I've misinterpreted anything about what you're asking. Happy to discuss further.
jchanimal 2y ago
Is anyone building a personal social friend graph / calendar hangout maximizer with spaced repetition?
- owenpalmer 2y ago
  Would hangouts equate to reviews? An SRS is for review minimization, not maximization. Bob: Well Owen, I really enjoyed hanging out with you. Me: presses "Easy" on the SRS Also me: See you in 9 months, Bob
  - jchanimal 2y ago
    Seems like a reasonable way to solve the packing problem. You could override to set priority levels for people, but the background process makes sure no one person goes too long without a meetup.
pessimizer 2y ago
I've been wanting to request an Anki feature for a while that is in this vein (and would enable this style): allowing cards within a deck to be tagged with a trigger, which would activate when all (or some portion of) cards within a deck also tagged with that trigger reach "maturity." When activated, it can add a tagged batch of cards from the deck as new cards for review, or more generally it could change their state (from/to Young, Mature, Suspended, etc.). This would allow cards to be automatically introduced in groups, rather than the "just give me 10 new cards every day" thing. It could also allow a default suspended deck style, where all but a few cards begin suspended, and they are unsuspended in an orderly way based on one's progress with the active cards. That could be a pattern that works for drilling hierarchies of knowledge. There's a deck style called the KOFI (conjugation first) system for learning Romance conjugations that would really benefit from that, which now depends on people manipulating the decks manually with the aid of an abused deck description field describing that manipulation. Anki is pretty hostile to order, because spaced repetition is all about moving individual cards optimally and without regard to other cards, but the order in which information is introduced can be important. At this point you can only choose between introducing cards in a linear order, or completely at random. There's no way to deviate from that without the user having to learn to juggle things manually. A deck creator can't just suggest a schedule programmatically, even if the deck was designed around that schedule. The other half of this, where cards that duplicate combinations of other cards are automatically strengthened when their related cards are strengthened (or vice versa) is interesting. Seems like a neuron model.
- AlexErrant 2y ago
  How do you feel about hierarchical tags? No, I don't mean in the way that Anki has a "file tree/directory" kind of structure for tags and decks, but like, a separate "concept" hierarchical graph that organizes the tags. For example, consider two tags, "Fractions" and "Prime Numbers". One should know Fractions before they study Prime Factors, and one could represent that using a drag/drop UI like <insert generic mindmap tool>. This "concept hierarchy" would organize using tags. This way you could still syntactically tag cards with, say, "Prealgebra/Fractions" and "Prealgebra/Prime Numbers". In this way one could have a "syntax" tree and a "semantic" tree/graph. (File trees can't represent graphs, as you may know.) One problem with making cards directly hierarchical with each other is that people have many cards, and organizing them individually can be a pain (and questionable usage of time).
JuliusSu 2y ago
Hi Jason, we had discussions at Caltech on some of these ideas many years back. We do have a patent, now lapsed, on doing spaced repetition on knowledge graphs: https://patentimages.storage.googleapis.com/ff/ae/6d/34838593506b08/US8832117.pdf https://patentimages.storage.googleapis.com/ff/ae/6d/3483859... Math pedagogy in the US leans to understanding being more important than practice. This is wrong. As you point out, understanding more often comes from practice. To correct this imbalance, software like your company is developing is useful, as are platforms like ALEKS, iReady, etc. The challenge is addressing the social aspect of learning -- the benefits students get from working and learning together. Is it still a classroom if all the students are doing different things? Of course, students come in with different abilities and levels of preparation. Differentiation that works in a class setting is very hard. Software is the easier part. Students who are motivated to learn and can grow with guidance from a computerized tutoring environment -- helping them is also the easier part.
- jayro 2y ago
  Hi Julius, it's been a while! Yep, it's an extremely challenging problem. The initial version of the Math Academy system has been primarily geared toward individual learners, which for a variety of technical and business reasons was the only realistic way to get off the ground. However, we're implementing a variety of features that will give teachers the ability to direct their class's learning progression while still allowing the system to adapt to and meet each student's individual needs. In some cases, it will provide critical remediation while for others it will offer topic extensions and challenge problems. Additionally, the system will offer a variety of differentiated group and class projects that are unlocked as students demonstrate mastery of the requisite skills.
bhjgjy 2y ago
[flagged]

178 points by JustinSkycak 2y ago

Now these are the insights I'm looking for when it comes to "second generation" SRS software suites. When you focus your attention on a specific niche, like mathematics here, you can get some serious improvements in retention just by using the shape of the subject to your advantage like this.

AlchemistCamp 2y ago

Anki contributor here: I’ve been using SRS on and off since 2006 and this post addresses one of the key failure modes I’ve seen in SRS usage—they tend to be fantastic for discrete pieces of information, like the names of capital cities, the pronunciation of various symbols or various physical constants. Most my usage of SRS in the early days was languages. For that domain, SRS is useful for learning an alphabet (or syllabary), for scaffolding enough common words to start understanding graded readers and for a few other tasks. They’re terrible as a primary learning strategy, though! Too much of language is highly nuanced and context dependent and the only way to absorb all the unwritten rules, the common collocations and the precise boundaries of word meanings is through extensive input. This means encountering the same words and structures in countless different variations and contexts, not the same few sentences drilled repeatedly (at least past the beginning stages). By paying attention to the specifics of learning math and tailoring the SRS to match it, it’s possible to make a far more efficient system than anyone could with an Anki deck. I’d love to see similar efforts in other subjects!

dotancohen 2y ago
Anki is great for building vocabulary. It's not suitable for actually learning the language. I use it constantly, averaging over 200 cards a day for over fifteen years.
- kiba 2y ago
  I wish I have the same dedication, but alas Anki hadn't sufficiently proven useful to me that I would use it on a day to day basis.
  - dotancohen 2y ago
    If you have an Android device, go install Ankidroid. It's absolutely terrific, letting you fill in those wasted seconds when you're waiting in line, sitting on the toilet, waiting for the pot to boil. Two or three cards at a time, dozens of times per day, adds up. You could treat yourself to three cards after every git commit, for instance. Takes literally less than ten seconds to review three cards.
    - kiba 2y ago
      The problem isn't the convenience but that the value in my deck hasn't sufficiently manifested itself in real world applications. This is my own fault rather than the app's fault.
      - Bluestein 2y ago
        I both second Ankidroid (oustanding) ... ... and concur when failing (as of yet) to find a way to use the system with other types of knowledge, sadly.-
- boppo1 2y ago
  How many new cards a day do you set?
  - dotancohen 2y ago
    Depends on what I'm actively learning at any given time. Right now it's 4, sometimes it could be closer to 20.
gwd 2y ago
A couple of problems I found with Anki for languages: 1. The words aren't in context, which leads to "flashcard blindness", where you see a word in context, know that you recognize it, but can't remember what it means. 2. The theory behind spaced repetition is that you learn most efficiently when you try to remember something just when you're about to forget it. But that means that if the algorithm is working properly, every single card is hard. This makes motivation a problem, because you know studying is going to be a grind, not fun. 3. While "being able to remember it after thinking a second or two" might be fine when studying for an exam, or in many other contexts where memorization might be important, that's too slow for languages. What you want for a language is "know it immediately without having to think about it". 4. The "scheduled review" system is too inflexible. Some days you get only a handful of cards to review, some days you get dozens. It's hard to tell when you're starting out how many cards you should be adding each day such that the number of cards match the amount of time / effort you have to study. Furthermore, if you skip a single day, you have twice as much the next day; and if life happens and you end up missing a week or a month, you come back with a giant jumble of cards, half of which you've forgotten, and it's really difficult to dig your way out of it. By a strange coincidence, in 2019 I also started on an alternate to Anki to help myself study Mandarin. It has some similarities to his system, in that there's multi-level knowledge; but it's different in that instead of having a fixed schedule, it has the concept of "difficulty" and "study value" for each word / grammar concept, and the algorithm tries to give you a full "readunit" of "native input" which will balance the two. "Spaced repetition" emerges naturally from the model, and if you go away for a week (or 6 months), it knows you've forgotten some things, so it gradually refreshes your memory. And because you're reading actual native text, there's something which pulls you in. It's in closed beta now; the first public language (MVP) will be in Biblical Greek, but the second one (if it happens, maybe in a year or two) will be in Mandarin; and hopefully there will be other ones after that. There's a sign-up form you can use to be notified for updates. https://www.laleolanguage.com https://www.laleolanguage.com
- Bluestein 2y ago
  > But that means that if the algorithm is working properly, every single card is hard. Could the algo be tweaked so as to regulate the "closeness to the edge", ergo, the difficulty? PS. Then again, something tells me what makes SR so effective *might just be* (or be related to) that "difficulty".-
jwells89 2y ago
For this reason, I’ve seen it recommended in various language learning guides to use Anki in a manner that’s complementary to consumption of media in the target language. The general idea is to do Anki-only for the earliest bootstrapping stages (super basic vocab) and switch to media+Anki as early as possible. The media gives both context and real experience with the language while Anki serves to commit the words you’ve read/heard (even those not used quite as commonly) to memory. I’ve only just dipped my toe into this method but have seen many reports of success with it from others.
- bunderbunder 2y ago
  You might find, as you get deeper into the language learning journey, that this approach to practice is most popular among an extremely vocal group of intermediate language learners. But people who've been doing it for a long time tend to gravitate toward the parent poster's opinion. I've also noticed that assiduously sentence mining your input and then drilling it in Anki is much more popular among people whose media consumption is almost exclusively video content. This resonates with the work of SLA researchers such as Paul Noble who've found that extensive reading is easily the fastest way to grow vocabulary in a new language. There's even some (scant) evidence that SRS combined with extensive reading is less effective than extensive reading alone for long-term vocabulary growth and retention. It suggests, to me at least, that heavy reliance on SRS for vocabulary retention past the beginner stages of learning a language may not be a universal optimal path, so much as a backup option for people who don't enjoy reading. Like AlchemistCamp I do still like it a lot for specific things, and I'm not here to wag fingers at people who enjoy it (Above all else, make sure you're enjoying your study routine! Getting there should be 100% of the fun.), but I do still worry that many Anki users have a serious case of golden hammer syndrome.
  - jwells89 2y ago
    For sure, by no means am I married to the method and am very flexible and open to trying various approaches. I do believe however that a little bit of structure can be helpful in the beginner to intermediate levels, where it's less likely that one has formed a habit/urges to watch/listen to foreign media without subtitles or read untranslated print media. It's a real slog when you don't have a decent arsenal of words to work with.

isaacfrond 2y ago

There lots of knowledge types where Anki might not be the ideal choice. For example, - itemized lists of information. E.g. The 5 reasons for the civil war, type of questions. What to do, if you miss one item? Your card as a whole gets over exposed, while the missed item might get underexposed. - Related information, e.g., history dates. When using Anki the students tends to latch on to incidental facts, e.g., Treaty of Paris, ah I remember it has two repeating digits, yes, 1899. While it would be more useful for a student to think, ah, that was after the Cuban independence struggles in the 1890's. Any user will recognize that this effect is very strong. You tend to remember cards by the most trivial of things. - Hierarchical knowledge. Things like chess openings. How do you put that in Anki cards. It's all a kludge. Where to put the variants, etc. - Knowledge networks. Things like medical information (where Anki is hugely popular). But typically, you get cards with massive amounts of information because you have lots of linked information, (symptoms, causes, treatment, pharmaceuticals, etc. ) - Even in language learning. We have the useful fiction that there is a 1-1 relationship between words in two languages, but there almost never is. - Learned knowledge tends to be linked to Anki. It happens often that you can remember things when doing Anki, while being at a loss when needing the information in the real world. Add to that, that Anki does nothing for conceptual understanding. You really need to learn a subject before memorizing it, but in the learning phase Anki is not helpful. So in short, yes, Anki is the best tool there is to help learning, but I'm sure better tools can be made, especially when targeted to a particular knowledge field.

gwervc 2y ago
> - Even in language learning. We have the useful fiction that there is a 1-1 relationship between words in two languages, but there almost never is. It's not really a problem: space repetition is a way to build familiarity with vocabulary. Once it is done, more finer uses can be inferred from context. The value lies in breadth not depth.
- gwd 2y ago
  A couple of problems I found with Anki for languages: 1. The words aren't in context, which leads to "flashcard blindness", where you see a word in context, know that you recognize it, but can't remember what it means. 2. The theory behind spaced repetition is that you learn most efficiently when you try to remember something just when you're about to forget it. But that means that if the algorithm is working properly, every single card is hard. This makes motivation a problem, because you know studying is going to be a grind, not fun. 3. While "being able to remember it after thinking a second or two" might be fine when studying for an exam, or in many other contexts where memorization might be important, that's too slow for languages. What you want for a language is "know it immediately without having to think about it". 4. The "scheduled review" system is too inflexible. Some days you get only a handful of cards to review, some days you get dozens. It's hard to tell when you're starting out how many cards you should be adding each day such that the number of cards match the amount of time / effort you have to study. Furthermore, if you skip a single day, you have twice as much the next day; and if life happens and you end up missing a week or a month, you come back with a giant jumble of cards, half of which you've forgotten, and it's really difficult to dig your way out of it. By a strange coincidence, in 2019 I also started on an alternate to Anki to help myself study Mandarin. It has some similarities to his system, in that there's multi-level knowledge; but it's different in that instead of having a fixed schedule, it has the concept of "difficulty" and "study value" for each word / grammar concept, and the algorithm tries to give you a full "readunit" of "native input" which will balance the two. "Spaced repetition" emerges naturally from the model, and if you go away for a week (or 6 months), it knows you've forgotten some things, so it gradually refreshes your memory. It's in closed beta now; the first public language (MVP) will be in Biblical Greek, but the second one (if it happens, maybe in a year or two) will be in Mandarin; and hopefully there will be other ones after that. There's a sign-up form you can use to be notified for updates. https://www.laleolanguage.com https://www.laleolanguage.com

hereme888 2y ago

This is why my initial flashcards on a topic would be cover a breadth of individual details, but later I'd start creating concept based flashcards where I'd ask things like "how does it make sense that....?" After somewhat memorizing foundational knowledge, understanding concepts and having frameworks of thinking for a topic will further increase critical thinking while reducing the amount of cognitive effort required for critical thinking about unfamiliar situations. I hope one day these more efficient models that "clean up" the review schedule by linking/identifying related flashcards become publicly available.

azeirah 2y ago

I really hope to see more research like this. I feel like LLMs have insanely high pedagogic value, and that's because I've been using them to teach myself difficult subjects and review my understanding. The issue with the kind of system that the author is proposing for curious self-driven learning is that the SRS is optimized for a given curriculum. Many people including myself use flashcards to guide (and sort of "create") their own curriculum. Flashcards with SRS are really good for many things, but it's difficult to generalize them for thus usecase. I'd really like to see some prototypes of people integrating LLM intelligence in creating and adjusting cards on the fly. It's clearly something LLMs excel at, especially the larger closed-source ones like Claude 3.5 sonnet

kiba 2y ago
I really hope to see more research like this. I feel like LLMs have insanely high pedagogic value, and that's because I've been using them to teach myself difficult subjects and review my understanding. How do you know if the LLM isn't hallucinating?
- EnigmaFlare 2y ago
  For mathy subjects you usually have to verify things yourself so you'd detect any mistakes as you try to work things out or makes sense of the information. You can't learn math, physics or theory for engineering by remembering and believing things. It would be different for "bag of random facts" subjects though.
- knighthack 2y ago
  This fear that LLMs are hallucinating constantly, and therefore entirely unreliable, is unfounded. Sure, that'll happen if you're going into extreme limits of areas of knowledge - but if you're dipping your foot into superficial waters of known areas, it's reasonably fine (and perhaps even more sound than the hotter areas of Wikipedia). E.g. if you're programming in the most well-known languages (say Python), and you're only asking beginner questions (which is where high pedagogic value lies) it's unlikely that the program will hallucinate all that deeply to mislead you. Yes, you won't know, but simple/mid questions are easy to check. You shouldn't of course head to LLMs for the most arcane scientific/mathematical knowledge and conclusions; if you do that, that's on you, because offering expertise is not what LLMs are designed for, and the disclaimers all warn against it. But you shouldn't throw out the baby with the bathwater.

ofirg 2y ago

Our company (Language Zen) has this patent https://patents.google.com/patent/EP1891619A2/en https://patents.google.com/patent/EP1891619A2/en and we have done work to teach hierarchical knowledge for language learning

ofirg 2y ago
sorry wrong patent link: https://www.patentsencyclopedia.com/app/20160035249 https://www.patentsencyclopedia.com/app/20160035249
- deleted 2y ago
  [deleted]
AlchemistCamp 2y ago
How revolting. Nothing in what was described in your updated link below looks even remotely novel even years before the 2016 filing date. I say this as someone who was deeply involved in the industry in the late 2000s. I also contributed to Anki in that time frame.
- rmbyrro 2y ago
  This patent is just for the business owners vanity. It's clearly not defensible in courts. Even if it was, they probably don't have the money required to pursue court enforcement of the patent. More so when any reasonable lawyer will tell them they'll get their case dismissed anyway.
- blackbrokkoli 2y ago
  Yeah I don't understand how this can be a patent. For example Skinner's Learning Machine as well as any automatic implementation of Pimsleur's method, both described in the SIXTIES, would fall under this (according to my reading, I'm not a patent lawyer, hamdulillah). Absurd.

iandanforth 2y ago

While this article presents a system it doesn't present any results. Does this modification to SRS help? Which type of student does it help? How large is the effect if there is one? I haven't read the larger pdf of which this is a part so perhaps someone who has can provide a pointer to some results.

JustinSkycak 2y ago
The main idea is that this approach makes spaced repetition feasible in something like mathematics. Without this approach, spaced repetition wouldn't even be feasible because after a short while, you'd be continually overloaded with too many reviews to really make any progress learning new material. Moreover, in addition to making spaced repetition feasible, it minimizes the amount of review (subject to the condition that you're getting sufficient review) which allows you to make really fast progress. We (Math Academy) don't have any official academic studies out at the moment, but if you want some kind of more concrete evidence of learning efficiency, you can read more online about our original in-school program in Pasadena where we have 6th graders start in Prealgebra, and then learn the entirety of high school math (Algebra 1, Geometry, Algebra 2, Precalculus) by 8th grade, and then in 8th grade they learn AP Calculus BC and take the AP exam. The AP scores started off decent while doing manual teaching, but the year we started using our automated system (of which the SRS described here is a component), the AP Calculus BC exam scores rose, with most students passing the exam and most students who passed receiving the maximum score possible (5 out of 5). Four other students took AP Calculus BC on our system that year, unaffiliated with our Pasadena school program, completely independent of a classroom, and all but one of them scored a perfect 5 on the AP exam (the other one received a 4). Even some seemingly impossible things started happening like some highly motivated 6th graders (who started midway through Prealgebra) completing all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) and starting AP Calculus BC within a single school year. Funny enough, the first time Jason & Sandy (MA founders) saw a 6th grader receiving AP Calculus BC tasks, Jason's reaction was "WTF is happening with the model, why is this kid getting calculus tasks, he placed into Prealgebra last fall, this doesn't make any sense," but I looked into it only to find that it was legit -- this kid completed all of what is typically high school math (Algebra I, Geometry, Algebra II, Precalculus) within a single school year. Anyway, some links if you're interested: * https://www.mathacademy.us/press https://www.mathacademy.us/press * https://www.reddit.com/r/homeschool/comments/16hn9f5/comment/k91ip0y https://www.reddit.com/r/homeschool/comments/16hn9f5/comment... * https://x.com/justinskycak/status/1810482435940913502 https://x.com/justinskycak/status/1810482435940913502 * https://x.com/justinskycak/status/1812557234028839193 https://x.com/justinskycak/status/1812557234028839193 Again, I realize these are not official academic studies, but we're completely overloaded in startup grind mode right now and have so many fish to fry with the product that we just don't have the time for academic pursuits at the moment, let alone much sleep. Happy to answer any follow-up questions that you might have, though.

asoneth 2y ago

More than a decade ago I worked as a research programmer on a similar AI tutoring product for hierarchical skills like mathematics at Carnegie Mellon (in conjunction with the Pittsburgh Science of Learning Center and Carnegie Learning). The system would prompt students with problems that incorporated dozens of sub-skills, each of which incorporated other sub-skills and so on. If the student gets the problem right, all requisite sub-skills are marked as reasonably strong. If they get it wrong then the statistical model concludes that at least one sub-skill must be weak/faulty/missing, and when selecting subsequent problems the system would incorporate problems with different subsets of the original sub-skills to identify the missing sub-skill(s), with the goal being to spend time reinforcing only those skills that the student has not mastered. I no longer work in that domain, but I recall it being amazingly efficient -- it often only took a dozen questions to assess a student's understanding of thousands of skills.

BenoitP 2y ago
Seems quite close to the 'active learning' ML topic: let the model choose the question that will bring the best information return. And this kind of decomposition using simple knowledge pieces (logical axioms?) is IMHO what we have to do to bring LLMs to senses. These pieces should light up in the intermediate embeddings inside the LLM. It won't really be more intelligent, but it'll model reality better.
owenpalmer 2y ago
Reminds me of using git-bisect
ahmni_dev 2y ago
Agreed that this article is strongly related to Intelligent Tutoring Systems like the Carnegie Tutor. I recently posted a brief literature review video of the field for those interested in the bigger picture around the article: https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz https://youtu.be/SDb9-LGRrCU?si=t6FM3832-_OwwpSz The TLDR of the relevant sections (see video description for relevant timestamps) - Models like ACT-R represent a model of the domain and a model of the student - Knowledge tracing is an ML-heavy field that predicts skills, state, etc (like parent says) - Recommender Systems use that knowledge to intelligently recommend next problems

graboid 2y ago

Say you have concepts/items/cards A, B and C, with A -> B -> C (C encompasses B, B encompasses A, keeping the notation from the article). As I understand it, the article advocates for showing C first, then you can assume that you also know B and A to at least some part, and save yourself the repetitions for these. Intuitively, I would have guessed the opposite approach to be the best: Show A first, suspend B until A is learned (by some measure), then show B, etc. That means no repetitions to skip, but also you get less failures (and thus additional repetitions) that occur as follows: you are shown C, but don't know B anymore, and thus cannot answer and have to repeat C. If you are shown C before B, you kinda make C less atomic (you might have to actively recall both, B and C to answer it), showing B before C makes C more atomic, as you will have B more mentally present/internalized and can focus on what C adds to B.

JustinSkycak 2y ago
Hey! Good questions. I'll try to clarify: 1. First want to clarify that the learner is first introduced to the topics through mastery learning (i.e., not given a topic until they've seen and mastered the prereqs). So, they would explicitly learn A before learning B, and explicitly learn B before learning C. It's only in the review phase when we do all this stuff with "knocking out" repetitions implicitly. 2. When you say "then you can assume that you also know B and A to at least some part," I want to emphasize that if C encompasses B and B encompasses A in the sense of a full encompassing that would account for a full repetition, then doing C fully exercises B and A as component skills. Not just exercises them "to some part." For instance, topic C might be solving equations of the form "ax+b=cx+d," topic B might be solving equations "ax+b=c," and topic A might be solving equations "ax=b." 3. This scenario should never happen: "you are shown C, but don't know B anymore, and thus cannot answer and have to repeat C." There are both theoretical and practical safeguards. 3a-- Theoretical: if you are at risk of forgetting B in the near future, then you'll have a repetition due on B right now, which means you're going to review it right now (by "knocking it out" with some more advanced topic if possible, but if that's not possible, we're going to give you an explicit review of B itself. In general, if a repetition is due, we're not going to wait for an "implicit knock-out" opportunity to open up and let you forget it while we wait. We'll just say "okay, guess we can't knock this one out implicitly, so we'll give it to you explicitly." 3b-- Practical: suppose that for whatever reason, the review timing is a little miscalibrated and a student ends up having forgotten more of B than we'd like when they're shown C. Even then, they haven't forgotten B completely, and they can refresh on B pretty easily. Often, that refresher is within C itself: for instance, if you're learning to solve equations of the form "ax+b=cx+d," then the explanation is going to include a thorough reminder of how to solve "ax+b=c." And even in other cases where that reminder might not be as thorough, if you're too fuzzy on B to follow the explanation in C, then you can just refer back to the content where you learned B and freshen up: "Huh, that thing in C is familiar but it involves B and I forgot how you do some part of B... okay, look back at B's lesson... ah yeah, that's right, that's how you do it. Okay, back to C." And then the act of solving problems in C solidifies your refreshed memory on B. Anyway, I think I've clarified all your questions? But please do let me know if you have any follow-up questions or I've misinterpreted anything about what you're asking. Happy to discuss further.

jchanimal 2y ago

Is anyone building a personal social friend graph / calendar hangout maximizer with spaced repetition?

owenpalmer 2y ago
Would hangouts equate to reviews? An SRS is for review minimization, not maximization. Bob: Well Owen, I really enjoyed hanging out with you. Me: presses "Easy" on the SRS Also me: See you in 9 months, Bob
- jchanimal 2y ago
  Seems like a reasonable way to solve the packing problem. You could override to set priority levels for people, but the background process makes sure no one person goes too long without a meetup.

pessimizer 2y ago

I've been wanting to request an Anki feature for a while that is in this vein (and would enable this style): allowing cards within a deck to be tagged with a trigger, which would activate when all (or some portion of) cards within a deck also tagged with that trigger reach "maturity." When activated, it can add a tagged batch of cards from the deck as new cards for review, or more generally it could change their state (from/to Young, Mature, Suspended, etc.). This would allow cards to be automatically introduced in groups, rather than the "just give me 10 new cards every day" thing. It could also allow a default suspended deck style, where all but a few cards begin suspended, and they are unsuspended in an orderly way based on one's progress with the active cards. That could be a pattern that works for drilling hierarchies of knowledge. There's a deck style called the KOFI (conjugation first) system for learning Romance conjugations that would really benefit from that, which now depends on people manipulating the decks manually with the aid of an abused deck description field describing that manipulation. Anki is pretty hostile to order, because spaced repetition is all about moving individual cards optimally and without regard to other cards, but the order in which information is introduced can be important. At this point you can only choose between introducing cards in a linear order, or completely at random. There's no way to deviate from that without the user having to learn to juggle things manually. A deck creator can't just suggest a schedule programmatically, even if the deck was designed around that schedule. The other half of this, where cards that duplicate combinations of other cards are automatically strengthened when their related cards are strengthened (or vice versa) is interesting. Seems like a neuron model.

AlexErrant 2y ago
How do you feel about hierarchical tags? No, I don't mean in the way that Anki has a "file tree/directory" kind of structure for tags and decks, but like, a separate "concept" hierarchical graph that organizes the tags. For example, consider two tags, "Fractions" and "Prime Numbers". One should know Fractions before they study Prime Factors, and one could represent that using a drag/drop UI like <insert generic mindmap tool>. This "concept hierarchy" would organize using tags. This way you could still syntactically tag cards with, say, "Prealgebra/Fractions" and "Prealgebra/Prime Numbers". In this way one could have a "syntax" tree and a "semantic" tree/graph. (File trees can't represent graphs, as you may know.) One problem with making cards directly hierarchical with each other is that people have many cards, and organizing them individually can be a pain (and questionable usage of time).

JuliusSu 2y ago

Hi Jason, we had discussions at Caltech on some of these ideas many years back. We do have a patent, now lapsed, on doing spaced repetition on knowledge graphs: https://patentimages.storage.googleapis.com/ff/ae/6d/34838593506b08/US8832117.pdf https://patentimages.storage.googleapis.com/ff/ae/6d/3483859... Math pedagogy in the US leans to understanding being more important than practice. This is wrong. As you point out, understanding more often comes from practice. To correct this imbalance, software like your company is developing is useful, as are platforms like ALEKS, iReady, etc. The challenge is addressing the social aspect of learning -- the benefits students get from working and learning together. Is it still a classroom if all the students are doing different things? Of course, students come in with different abilities and levels of preparation. Differentiation that works in a class setting is very hard. Software is the easier part. Students who are motivated to learn and can grow with guidance from a computerized tutoring environment -- helping them is also the easier part.

jayro 2y ago
Hi Julius, it's been a while! Yep, it's an extremely challenging problem. The initial version of the Math Academy system has been primarily geared toward individual learners, which for a variety of technical and business reasons was the only realistic way to get off the ground. However, we're implementing a variety of features that will give teachers the ability to direct their class's learning progression while still allowing the system to adapt to and meet each student's individual needs. In some cases, it will provide critical remediation while for others it will offer topic extensions and challenge problems. Additionally, the system will offer a variety of differentiated group and class projects that are unlocked as students demonstrate mastery of the requisite skills.

bhjgjy 2y ago

[flagged]