8 ms·

Want to Remember Everything You'll Ever Learn? Surrender to This Algorithm

258 points by mshafrir 17y ago

btilly 17y ago
In grad school I managed to take advantage of the pacing effect in an educational setting. I was teaching linear algebra. What I did was make the homework incremental - 1/3 of homework on today's material, 1/3 on the previous week, and 1/3 anything in the course. Those thirds were in increasing order of difficulty. I also started every class with a question/answer period. The rules were simple, the questioning will last at least 10 minutes, and you don't want me to ask the questions. :-) Anything that had come up in the questions that seemed to be a point of confusion was sure to be added to the next homework set. I won't go into what else I did with that class, but the end result is worth thinking about. First note that I gave a ridiculously hard final. Other grad students who saw it thought that the class would bomb. Secondly they aced the test. What do I mean by aced? Well I had a bonus question which fellow grad students thought nobody would get. 70% of the class got that question, and a good fraction were over 100% on the test. So they must have studied hard, right? Nope. I ran into some students several months later. They told me that they tried to study for the final and stopped after a few minutes because it was useless, they knew everything. And several months later they still knew much of the material cold! The thing is that none of what I did was very radical. The principles have been known for a century. Psychologists have been trying to get people to listen for that long. I learned about it in the 80s from a university course I watched on TV. (British Columbia had a TV channel devoted to lectures for correspondence courses.) Yet, despite how dramatic the effects are, nobody listens and nobody takes advantage of it.
- kiba 17y ago
  This remind me of a drawing book called "Drawing on the Right Side of the Brain". It teaches people how to draw realistically using techniques like contour drawing among other stuff. However, middle school art teachers seem stuck in their little art project. I am not sure about high school, but it doesn't look like they teach the skills.
  - cesare 17y ago
    > It teaches people how to draw realistically using techniques like contour drawing among other stuff. A great book. One of the exercises is to copy a picture turned 180 degrees, so that you're drawing exactly what you're seeing without the interference of your mind's prior knowledge of the subject.
    - juvenn 17y ago
      Yeh, I've remembered it. And I'm thinking on intro it to my best friend.
  - diiq 17y ago
    Not in high school, and not in collegiate art school. I have a copy that I pass around via samizdat, because the mandatory drawing classes here are useless to students without the basic principles. (Also worth noting: 'The Memory Book' by Lucas and Loraynne. Now back in print, an falling-apart copy got me through high school. I can still memorize a deck of cards in under ten minutes.)
  - gila 17y ago
    I took art in high school and my teacher used this technique, although getting kids to adhere to the style wasn't very easy at first. I guess my teacher had an advantage with a psychology degree though.
- tocomment 17y ago
  If you made this story into a blog post, people would listen. (I'm actually serious, it's an interesting story and a lot of people might benefit from hearing more of the details and the thought process behind it.
  - btilly 17y ago
    I'll think about it. I won't get around to it in the near future though, sorry.
    - bjelkeman-again 17y ago
      Please do, as I am sure there are a bunch of us who would love to hear more about it.
      - yummyfajitas 17y ago
        I'd love to hear about it as soon as possible (i.e., early enough in the semester for me to act on it). I'm already updating webassign to follow your suggestion, current material + old stuff each week. Anything else you did (beyond SRS) would also be useful to know.
        eob 17y ago
        You don't need someone to tell you what to do -- he/she already gave you the general idea. Why can't you be similarly creative with your class and run with it?
        yummyfajitas 17y ago
        I'm implementing all of btilly's specific suggestions. As for why not "be creative"? I don't see how that suggestion is helpful. Btilly wasn't creative, he implemented proven techniques. I want to do the same, I just don't know what those proven techniques are (beyond the ones he mentioned in his post).
    - 17y ago
- elai 17y ago
  Could you go into more details into SRS for math? "Fact Memorizing" type systems seem to fit well with things like supermemo (languages, law, history, biology, a bunch of sciences), but not so much with things like math (other than memorizing formulas). And did you really add homework questions like find the kernel of this subspace in the first few weeks of the course?
  - btilly 17y ago
    I'm not sure what details to go into. Being reminded of something as we're forgetting it pushes that thing into longer term memory. It also internalizes the lesson. This applies to both rote facts and intellectual ideas. If you think of it as fact memorization, then you're missing the real power. There are a lot of things you need to learn to master a mathematical topic. You need to learn definitions and terminology. You need to learn key theorems. You should be able to apply procedures and derive formulas. At a higher level you need to learn available proof techniques and the key ideas behind important proofs. Of course learning these things intellectually is not quite sufficient. You need skills as well. However failing to learn these things will keep you from learning the topic. And that is where most students fall down. This is particularly true in subjects that build on themselves like mathematics. If you missed an important point earlier, you may develop a bigger misconception about a current topic that will make it impossible to learn a future one. Anyone who has tutored students should be aware of this. As for what to review, students did not get the same questions over and over again. Instead I gave them questions from the same section over and over again. Since I only asked a few questions from the current day's section, there were always plenty of questions left to go back to for future homework sets. As a bonus doing many kinds of questions at the same time makes connections obvious. When a single homework set asks you to solve a set of equations, find a basis for a vector space, and invert a matrix, you see how they tie together.
    - juvenn 17y ago
      You reminded me of learning abstract algebra, in the last year of my college. The subject is extremely abstract, and I try hard to learn the key ideas, without realizing I need do it intellectually, instead of just thinking it more, and reading it with more care. Besides, I could not make any progress on that book, it just got me frustrated. And then I gave up, I hope to continuing my study in grad school, but I could just not get my feet in. I'm slow, I think, and I can not do well in math. So I just let math get off my life, and trying to be a computing hacker as well as an entrepreneur.
- stevenj 17y ago
  I think I learned more about retention (for me personally) by reading your comment than by reading the article. Thanks for sharing.
- imajes 17y ago
  I wish you were my teacher. That's a theory of education I've been pondering about for ages. Glad to see it formalized like this. Please consider running for office and fixing how we go about learning. :)
  - btilly 17y ago
    Actually you should be glad that I'm not your teacher. I set a number of goals for that course around learning, and forgot to set any around making the students happy. I met the goals I set but... I'm sure that if I had stayed in academia I would have sorted out those kinds of issues. However it is clear to me that I'm much happier not being an academic.
    - antipaganda 17y ago
      If I learned what you were trying to teach me, I would be happy... afterwards. Being happy during the course is not required.
  - adinobro 17y ago
    You might want to have a look at Pimsleur. The theory described is almost the same except Pimsleur was studying language learning... http://en.wikipedia.org/wiki/Pimsleur_language_learning_system http://en.wikipedia.org/wiki/Pimsleur_language_learning_syst...
- ghshephard 17y ago
  What an excellent Hacker News Post. Please do write up a bit more about your method, what your students got out of the course specifically, and whether you've had success with it in other places. You have a clean, readable, and interesting writing style. I think we can all recall a teacher or teachers in our life who truly made a difference in teaching us something - sometimes _despite_ our interest (or lack thereof) of learning it. I was forced in University to take a course in Propositional Calculus (it was a breadth requirement for computing science) - it was mid-day so I attended the class (unlike early morning classes in which my attendance was abysmal) - I went in with zero desire - no coding, or even math. But, the timing was right so I just sat in the class. The instructor spent thirteen weeks walking us through Reductio Ad absurdum proof, tautologies, conditional proofs, etc.. He gave us homework assignments that built on earlier knowledge, and just as we were getting into new material, he'd throw some earlier sequent proofs at us to solve and refresh our memories. Here is the thing - I had _no_ interest in this course, the only thing going for it was it's timing, and repetition. I was _killing_ myself trying to learn integration by parts, linear algebra, discrete math, spending 5 hours outside of class for every hour in class trying to master those other course, and basically just sitting in this one distraction course. I was a C+ student in the math courses (after _massive_ effort) - I walked out of the Propositional Calculus course with an A+. Sixteen years later I look back at University and can honestly say that the most useful course, that I _continue_ to have rock solid knowledge of PLUS use almost every single hour of the day was that stupid Philosphy 210 Propositional Calculus course. For the life of me I can't remember much of graph theory, and wouldn't be able to invert a matrix if you held a gun to my head, but, I'll sit in a meeting and see a ton of possible alternative and immediately start applying Disjunction elimination to break us out of a log jam. There really is something to this spacing method.
  - joe_the_user 17y ago
    It could also be that propositional calculus is dead simple - or at least that has always been my impression of it. Calculus and linear algebra work on complex problems in a large domain. Propositional calculus has a small domain and fairly simple theories. I think in general, the knowledge domain is important. Linear algebra is a very unified field by itself but when you get courses that involve a grab bag of algebra, linear algebra, trigonometry and calculus, I think the "stew" they present is incredibly hard to comprehend on a high level and students wind-up either failing or learning by a horrible kind of rote. Uh, this is from biased personal experience. I indeed only taught one college math course when I doing my math MA, it was "Trignometry". It was a complete failure. I still blame the "grab bag" quality of the material but perhaps I was to blame instead.
- TobiasCassell 17y ago
  Hey OP, was your technique influenced by Mihaly Csikszentmihalyi's Flow? I'm referring to the relationship between ability level and challenge. Do I see similarities with your clearly effective teaching style and some of the elements Video Game designers have almost unanimously implemented to great success?
  - butterfi 17y ago
    I'm not sure "the relationship between ability level and challenge" is what MC means by flow, if that's what you're saying. Q: "What is "the flow experience" and what does it have to do with motivation? A: The flow experience is when a person is completely involved in what he or she is doing, when the concentration is very high, when the person knows moment by moment what the next steps should be, like if you are playing tennis, you know where you want the ball to go, if you are playing a musical instrument you know what notes you want to play, every millisecond, almost. And you get feedback to what you're doing. That is, if you're playing music, you can hear whether what you are trying to do is coming out right or in tennis you see where the ball goes and so on. So there's concentration, clear goals, feedback, there is the feeling that what you can do is more or less in balance with what needs to be done, that is, challenges and skills are pretty much in balance." http://www.edutopia.org/mihaly-csikszentmihalyi-motivating-people-learn http://www.edutopia.org/mihaly-csikszentmihalyi-motivating-p...
    - TobiasCassell 17y ago
      (Although it's on the shortlist, I didn't read 'Flow' yet so I may not be able to speak with authority about his work.) Hey cool, thanks for the link. It's been a long time since I was in school but back then every homework assignment was 100% what we covered that day. So if the "lessons" became unclear many folks just got further and further behind and for many there was no recovering at some point because the teachers never wanted to deviate from their schedule. Btillys 1/3 1/3 1/3 structure really resonates with me because I can imagine I would feel confident and never get in over my head as the complexity of the work escalates. (And if I fell behind I could use his 10 minute improvisational time to get clear) But back to the confidence, Video Games Designers want the player to be challenged but also to feel confident and of course this balance has to remain throughout all levels of the game. Video Game Designers have long used MC's work to help them incorporate this balance. So, yes I do believe the relationship between ability and challenge is part of MC's agenda- there is more, of course. It is my understanding that Video Games are the most common contact most people usually have with MC's work. I made a connection between the successful feeling of playing video games with the successful learning experiences in Btillys class's and it would be interesting if Btilly is aware of MC's work or if he had ever thought of what he does as MCesque. In closing, I think you and I are saying the same thing here. Thanks for the comment..
- fburnaby 17y ago
  I'm just about to start lecturing the tutorials for a linear algebra class next week. The prof that is running the course taught it to me 4 years ago and is the reason I'm now in grad school. She did more or less the same thing that you mention, and it definitely works. I remember all of this material from back when I took the class; it's still familiar and I use it all the time. But despite being very useful for a half to two-thirds of her students, there is always about 1/3 of the class who does terrible in this course. How did you get these people to keep up? Anyway, I'm glad you posted this. It's a good pick-me-up motivator as I'm about to be trying to do the same thing!
- cgs 17y ago
  I've toyed with the idea of keeping a practice journal, but I've always been too lazy to stick to it. Your comment has inspired me though! This way, I can go back X number of days and review whatever I was working on then.
- xccx 17y ago
  is this smart.fm? http://www.techcrunch.com/2009/09/10/cerego-raises-3-4-million-for-smart-fm-launches-facebook-friend-quiz http://www.techcrunch.com/2009/09/10/cerego-raises-3-4-milli...
  - nwinter 17y ago
    Yes. It used to be called iKnow, and now iKnow is just an app on smart.fm, which is run by Cerego.
- juvenn 17y ago
  I think a good self-teacher would be better a teacher than who just hold some fancy degrees. Since good self-teachers always know what students need, and where they failed to grasp the essential, and most importantly how to get them out of obstacles. And I bet you must be a good self-teacher. I wish I have ever had you as my teacher!
  - antipaganda 17y ago
    There is one glaring fault that a self-taught teacher may have; people learn in different ways. What works for one may not work for all. So the self-taught teacher may not realise that their own personal methods are not necessarily valid for everyone.
  - eob 17y ago
    What is a "self-teacher"? The advantage of holding "fancy degrees" is usually that you have been exposed to a wide variety of viewpoints and insignt that self-taught people might have. People tend to study what they are interested in. Unfortunately, this can produce self-taught academics with horse blinders on. Imaging you only got your news from Reddit & Huffington Post, saying you were becoming a self-taught news commentator. You'd probably end up incredibly informed... about liberal viewpoints, perhaps not realizing that a large portion of the country doesn't share them. Getting taught in a formal education system (providing the system works) would expose you to a wider variety of views.
    - juvenn 17y ago
      Completely agreed, providing the system works. I'm not a native En, I mean self-taught teacher? Sorry.
- morphir 17y ago
  I recently heard that newly educated teachers come to schools with a strong desire to teach their specialties. And to transmit some of their true passion over to their students. But since being a teacher (here in Norway) means they have to administrate and organize the kids so much, they have little energy left for teaching. Meaning the teacher appears as a boring, pale fuck (not very inspiring). This is of course a organizational problem, and not a pedagogical one.
  - hyperbovine 17y ago
    Otoh I have always assumed pretty much everyone is pale in Norway. :-)
- scharan 17y ago
  Please do a more detailed version, if it makes sense to do so. You actually managed to say a lot in this little space. I had a teacher too who did this on a small scale - the first 5 mins was a summary of the previous calls. The first day of each week he summarized the entire chapter or section in the first 10 mins. This bird's eye view of the material went a long way in keeping the concepts 'live',
- msluyter 17y ago
  Seems related to this Wired article that recently was reposted on Reddit: http://www.wired.com/medtech/health/magazine/16-05/ff_wozniak?currentPage=all http://www.wired.com/medtech/health/magazine/16-05/ff_woznia... Bottom line is that to maximize retention, there's an optimal time for being re-exposed to information: right before you're about to forget it. Seems like re-covering the previous material in the class may have accomplished this.
- wsprague 17y ago
  The question of why is there so much crappy education is interesting. I think part of the answer is that society is well-served by the failings of the educational system in that widespread educational crappiness helps support the class system. I think it would be fairly straightforward to turn 90% of the population into well educated upper middle class types, using techniques like btilly describes... ... But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it? So I think that a huge function of the educational system is to educate a large part of the population badly. I think that the teacher training system serves this is as well, by selecting for mediocre teachers and then making them more mediocre via training. Additionally, in the schools themselves a lot of effort is made to classify young people into dumb and smart categories, usually unfairly (http://en.wikipedia.org/wiki/Pygmalion_effect http://en.wikipedia.org/wiki/Pygmalion_effect), and this classification stamps them for the rest of their lives and creates a population of hopeless lower class workers. If there are SOME good teachers in the system like btilly, then SOME kids go on to get out of their class, which is perfect in that (1) we need to claim that it is possible to pull yourself up by your bootstraps, and (2) we need to recruit SOME kids to grow up to fill management positions, but not too many that they all can't find careers. Historically, underemployed educated people are the people who become union leaders. The beauty of it is that the people who fail think it is their fault! (Sorry to rant about the educational / class system in general, rather than the topic of how to teach advanced math, but I couldn't stop myself.) Call me paranoid, but there you go.
  - seanb 17y ago
    Interesting idea. "But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it?" Robots? If the education system elevated more of the population, there would be a shortage of blue-collar workers, more pressure for system automation, and more well-educated minds to develop the robots. Of course, there would be no shortage of immigrants to fill the blue-collar positions, but perhaps in the long term, a "better" equilibrium would be reached.
- martincmartin 17y ago
  It's sad that professors are promoted for research, rather than either research or teaching. It amazes me that the private U.S. universities that are most highly ranked for undergraduate education are the ones with the best research.
- kalid 17y ago
  That's really awesome. I'm a huge proponent of finding the real, actual insights and lessons behind a subject (plug: I blog at betterexplained.com). The dirty little secret of education is that it's "ok" to cram and forget. Most classes reward that; education is about "being able to do something once" vs. actually remembering and applying it. I hope you write more about your findings!
- nostrademons 17y ago
  When I was learning to play the violin, my teacher had a very strict method for going through the repertoire. You had your working piece, which was supposed to be hard - it was stretching the abilities of what you could do as a violinist. You had your polishing piece, which was your previous working piece where you were finessing all the fine points of technique and musicality. You had review, which was all the other pieces you'd learned so far. And you had your preps, which were little passages (4 measures or so) from your working piece that were so hard that you played them over and over again, way slowed down, until you got them right and could speed them up and incorporate them into your working piece. I've tried to do something similar with my work so far - a working codebase that I'm just learning, a polishing codebase that I basically know my way around in, and various tweaks and bugfixes that I have to do for previous projects. It's a bit harder in the corporate world though - while Google engineers are given a lot of freedom to pick their projects, they're still subject to the needs of the business, and sometimes a project will come up that's such a great opportunity for professional visibility that I'd want to take it even if it involves working hard on something I already know well and dropping the polishing of stuff I just learned.
- pw 17y ago
  I'd suggest the book Why Don't Students Like School? (http://www.amazon.com/Why-Dont-Students-Like-School/dp/0470279303 http://www.amazon.com/Why-Dont-Students-Like-School/dp/04702...) to those interested in applying the findings of cognitive science to their own learning. I'm finding it very useful in that regard.
- araneae 17y ago
  I'm wondering what type of university you were at that let you write a final. The most flexibility given to grad students in introductory courses I've seen is, at most, to write a quiz or two. Most don't even let you write homeworks.
  - jedliu 17y ago
    Actually, at my department (a "top-10" school for CS), for each course we TA, we're expected to write at least one problem set and come up with at least one question for each exam given. There are also half-courses on intro to Unix or intro to C++ that are taught entirely by grad students. I suspect the OP was teaching an intro-level course, designed to get freshmen up to speed on basic linear algebra, filling any gaps they might have had in their high school curriculum.
cesare 17y ago
Once you've finished the article, there's a lot of reading material on the SuperMemo website: http://www.supermemo.com/english/contents.htm#Articles http://www.supermemo.com/english/contents.htm#Articles Really interesting stuff. The website and the software user interface (at least judging from the screenshots) are terrible, though.
- billswift 17y ago
  You beat to it. I've had the site on my favorites list for months, it's got more information than the article, but it wasn't anywhere near convincing enough to get me to actually spend any money. Maybe I'll go through it again and see.
  - cesare 17y ago
    Have you tried Anki? (it's open source, see GiraffeNecktie comment). I've just downloaded it. There are also some decks shared by people (mostly for natural languages) that you can download from the app itself.
    - peregrine 17y ago
      And GRE vocabulary, I don't even need to take the GRE(maybe) but I think I will study these, improve my vocabulary.
GiraffeNecktie 17y ago
This article is written as if SuperMemo were the only software that uses spaced repetition (SRS) algorithms for learning. Actually there are quite a number of other programs out there including some that are free or open source. I've used SuperMemo for years on my Palm and found it very useful for learning French but the software (at least for the Palm) hasn't been updated for many years and doesn't give the user any control whatsoever over how the program is used. For learning Chinese I've moved to a commercial product called Pleco on the Palm which is basically a dictionary but also has elaborate SRS functionality, it's really light years ahead of SuperMemo. Other more generic SRS programs include Anki and Mnemosyne. Unlike SuperMemo, it's easy to migrate your flashcards among these other applications. One other point, SRS is a huge help to learning vocabulary but learning words is only one part of mastering a language.
- cesare 17y ago
  Thanks. I was just wondering if there was a similar open source project. Anki: http://ichi2.net/anki/ http://ichi2.net/anki/ Wikipedia entry about this technique: http://en.wikipedia.org/wiki/Spaced_repetition http://en.wikipedia.org/wiki/Spaced_repetition
  - sandGorgon 17y ago
    Mnemosyne is actually very good http://www.mnemosyne-proj.org/ http://www.mnemosyne-proj.org/ it works on Win, Mac and Linux - written in Python
    - andymism 17y ago
      Thanks for the rec! The first thing I'm going to be using Mnemosyne for is scheduling guitar practice. I often forget the number of songs and scales that I actually know. Eventually, they get dusty enough that I won't remember them at all. I'm hoping this will help me to retain much better.
    - trapper 17y ago
      I use this when learning a new subject. It's insanely good
rgrieselhuber 17y ago
There is a company based here in Japan called Smart.fm (formerly iKnow) made up of really great people doing similar work.
bhellman1 17y ago
A lot of fluff here but still interesting stuff.
klipt 17y ago
There's at least one maker of language courses (Pimsleur) which use the pacing effect. Unfortunately they're audio only, which at least for me isn't nearly as effective as a combination of audio and writing.
- jwecker 17y ago
  I've used these- really brilliant. Takes the pacing effect but makes it conversational so it really seems to flow. The trade-off, of course, is that it has no idea how well _you_ have memorized various things (i.e., being prerecorded audio, it is generalized).
tocomment 17y ago
Is there someone I can try this out for free? Are there ready made flashcards for something useful to learn?
- tigerthink 17y ago
  As mentioned above, Anki and Supermemo are two free implementations. As for ready-made flashcards, here are some I made for a calculus class: http://www.spicyelephant.com/decks/523 http://www.spicyelephant.com/decks/523 http://www.spicyelephant.com/decks/577 http://www.spicyelephant.com/decks/577 http://www.spicyelephant.com/decks/659 http://www.spicyelephant.com/decks/659 You probably have to register for an account if you want the site to apply supermemo-type techniques. I don't recommend using the site in the long term because it will eventually force you to register for $20.
  - gwern 17y ago
    Supermemo isn't really free - maybe you meant 'Anki & Mnemosyne'? That's the big 2 in the Free SRS space.
    - tigerthink 17y ago
      Right. Sorry.
- r7000 17y ago
  You can try my free implementation here: http://flashcarddb.com http://flashcarddb.com. You won't be forced to pay for it to keep working.
nwinter 17y ago
This is what we do at http://www.skritter.com http://www.skritter.com for Chinese and Japanese. Spaced repetition is definitely still in the geek realm, though. Even after automating almost everything (vocab creation, addition, even grading), there are still a lot of people who don't want to practice on the computer, or aren't as interested in efficiency of the learning process. I think as spaced repetition moves away from raw SRS like SuperMemo and toward integration with classrooms, textbooks, and other software that provides content as well, we will start to see more widespread adoption and recognition of its power. There was a recent thread here about whether SRS will evolve to appeal to non-programmers: http://globalmaverick.org/archives/169-repetitio-est-mater-studiorum http://globalmaverick.org/archives/169-repetitio-est-mater-s...
idigthought 17y ago
Is there a good os x and iphone application for this? I'm looking particularly for one that supports images and LaTeX.
- gwern 17y ago
  There are a number of SRS clones already produced for the iPhone in the store (Mnemosyne, I understand, works fine on OS X); there's also a Mnemosyne clone in early development for the iPhone.
cgs 17y ago
One thing I've never bothered to do is learn an API of X language really well. Because, I thought, I'll just look up whatever I don't know. But I'm constantly stopping to look things up and it really slows me down. Now I'm beginning to think there's really some value in having a strong command of an API. Has anyone tried using an SRS tool to help with this? Or is it simply a matter of not having enough experience with X language?
- chimariko 17y ago
  I am not sure about SRS for learning API of a language. However I have recently started to use mind maps for that. Whenever I come across a new language or library I take an A4 and a pen and put down all basic concepts. It helps to grasp the big picture of the system and to see the interconnections of different blocks of code. I prefer mind maps to the standard linear up-down way of writing stuff down because the schematic representation utilized in mind mapping eases the twist between the way information is stored on paper and (presumably) in the human memory. All in all I find spending some time on studying docs useful in contrast to the pure work-by-doing approach.
- gwern 17y ago
  Actually, the SuperMemo site covers just this topic: http://www.supermemo.com/articles/programming.htm http://www.supermemo.com/articles/programming.htm tl;dr: if you will spend more than 5 minutes looking it up or trying to remember it over the next 10/20 years, then it's a net gain to memorize it in a SRS. I find this plausible, but I'm not sure 5 minutes is right - I understand from Anki users (Anki's statistics include 'how much total time ever have I spent looking at/on this particular card') that a card will cost you more than 5 minutes, and more like 10 minutes.
gfodor 17y ago
iPhone version coming out in a few weeks. (I emailed him about this.)

258 points by mshafrir 17y ago

In grad school I managed to take advantage of the pacing effect in an educational setting. I was teaching linear algebra. What I did was make the homework incremental - 1/3 of homework on today's material, 1/3 on the previous week, and 1/3 anything in the course. Those thirds were in increasing order of difficulty. I also started every class with a question/answer period. The rules were simple, the questioning will last at least 10 minutes, and you don't want me to ask the questions. :-) Anything that had come up in the questions that seemed to be a point of confusion was sure to be added to the next homework set. I won't go into what else I did with that class, but the end result is worth thinking about. First note that I gave a ridiculously hard final. Other grad students who saw it thought that the class would bomb. Secondly they aced the test. What do I mean by aced? Well I had a bonus question which fellow grad students thought nobody would get. 70% of the class got that question, and a good fraction were over 100% on the test. So they must have studied hard, right? Nope. I ran into some students several months later. They told me that they tried to study for the final and stopped after a few minutes because it was useless, they knew everything. And several months later they still knew much of the material cold! The thing is that none of what I did was very radical. The principles have been known for a century. Psychologists have been trying to get people to listen for that long. I learned about it in the 80s from a university course I watched on TV. (British Columbia had a TV channel devoted to lectures for correspondence courses.) Yet, despite how dramatic the effects are, nobody listens and nobody takes advantage of it.

kiba 17y ago
This remind me of a drawing book called "Drawing on the Right Side of the Brain". It teaches people how to draw realistically using techniques like contour drawing among other stuff. However, middle school art teachers seem stuck in their little art project. I am not sure about high school, but it doesn't look like they teach the skills.
- cesare 17y ago
  > It teaches people how to draw realistically using techniques like contour drawing among other stuff. A great book. One of the exercises is to copy a picture turned 180 degrees, so that you're drawing exactly what you're seeing without the interference of your mind's prior knowledge of the subject.
  - juvenn 17y ago
    Yeh, I've remembered it. And I'm thinking on intro it to my best friend.
- diiq 17y ago
  Not in high school, and not in collegiate art school. I have a copy that I pass around via samizdat, because the mandatory drawing classes here are useless to students without the basic principles. (Also worth noting: 'The Memory Book' by Lucas and Loraynne. Now back in print, an falling-apart copy got me through high school. I can still memorize a deck of cards in under ten minutes.)
- gila 17y ago
  I took art in high school and my teacher used this technique, although getting kids to adhere to the style wasn't very easy at first. I guess my teacher had an advantage with a psychology degree though.
tocomment 17y ago
If you made this story into a blog post, people would listen. (I'm actually serious, it's an interesting story and a lot of people might benefit from hearing more of the details and the thought process behind it.
- btilly 17y ago
  I'll think about it. I won't get around to it in the near future though, sorry.
  - bjelkeman-again 17y ago
    Please do, as I am sure there are a bunch of us who would love to hear more about it.
    - yummyfajitas 17y ago
      I'd love to hear about it as soon as possible (i.e., early enough in the semester for me to act on it). I'm already updating webassign to follow your suggestion, current material + old stuff each week. Anything else you did (beyond SRS) would also be useful to know.
      - eob 17y ago
        You don't need someone to tell you what to do -- he/she already gave you the general idea. Why can't you be similarly creative with your class and run with it?
        yummyfajitas 17y ago
        I'm implementing all of btilly's specific suggestions. As for why not "be creative"? I don't see how that suggestion is helpful. Btilly wasn't creative, he implemented proven techniques. I want to do the same, I just don't know what those proven techniques are (beyond the ones he mentioned in his post).
  - 17y ago
elai 17y ago
Could you go into more details into SRS for math? "Fact Memorizing" type systems seem to fit well with things like supermemo (languages, law, history, biology, a bunch of sciences), but not so much with things like math (other than memorizing formulas). And did you really add homework questions like find the kernel of this subspace in the first few weeks of the course?
- btilly 17y ago
  I'm not sure what details to go into. Being reminded of something as we're forgetting it pushes that thing into longer term memory. It also internalizes the lesson. This applies to both rote facts and intellectual ideas. If you think of it as fact memorization, then you're missing the real power. There are a lot of things you need to learn to master a mathematical topic. You need to learn definitions and terminology. You need to learn key theorems. You should be able to apply procedures and derive formulas. At a higher level you need to learn available proof techniques and the key ideas behind important proofs. Of course learning these things intellectually is not quite sufficient. You need skills as well. However failing to learn these things will keep you from learning the topic. And that is where most students fall down. This is particularly true in subjects that build on themselves like mathematics. If you missed an important point earlier, you may develop a bigger misconception about a current topic that will make it impossible to learn a future one. Anyone who has tutored students should be aware of this. As for what to review, students did not get the same questions over and over again. Instead I gave them questions from the same section over and over again. Since I only asked a few questions from the current day's section, there were always plenty of questions left to go back to for future homework sets. As a bonus doing many kinds of questions at the same time makes connections obvious. When a single homework set asks you to solve a set of equations, find a basis for a vector space, and invert a matrix, you see how they tie together.
  - juvenn 17y ago
    You reminded me of learning abstract algebra, in the last year of my college. The subject is extremely abstract, and I try hard to learn the key ideas, without realizing I need do it intellectually, instead of just thinking it more, and reading it with more care. Besides, I could not make any progress on that book, it just got me frustrated. And then I gave up, I hope to continuing my study in grad school, but I could just not get my feet in. I'm slow, I think, and I can not do well in math. So I just let math get off my life, and trying to be a computing hacker as well as an entrepreneur.
stevenj 17y ago
I think I learned more about retention (for me personally) by reading your comment than by reading the article. Thanks for sharing.
imajes 17y ago
I wish you were my teacher. That's a theory of education I've been pondering about for ages. Glad to see it formalized like this. Please consider running for office and fixing how we go about learning. :)
- btilly 17y ago
  Actually you should be glad that I'm not your teacher. I set a number of goals for that course around learning, and forgot to set any around making the students happy. I met the goals I set but... I'm sure that if I had stayed in academia I would have sorted out those kinds of issues. However it is clear to me that I'm much happier not being an academic.
  - antipaganda 17y ago
    If I learned what you were trying to teach me, I would be happy... afterwards. Being happy during the course is not required.
- adinobro 17y ago
  You might want to have a look at Pimsleur. The theory described is almost the same except Pimsleur was studying language learning... http://en.wikipedia.org/wiki/Pimsleur_language_learning_system http://en.wikipedia.org/wiki/Pimsleur_language_learning_syst...
ghshephard 17y ago
What an excellent Hacker News Post. Please do write up a bit more about your method, what your students got out of the course specifically, and whether you've had success with it in other places. You have a clean, readable, and interesting writing style. I think we can all recall a teacher or teachers in our life who truly made a difference in teaching us something - sometimes _despite_ our interest (or lack thereof) of learning it. I was forced in University to take a course in Propositional Calculus (it was a breadth requirement for computing science) - it was mid-day so I attended the class (unlike early morning classes in which my attendance was abysmal) - I went in with zero desire - no coding, or even math. But, the timing was right so I just sat in the class. The instructor spent thirteen weeks walking us through Reductio Ad absurdum proof, tautologies, conditional proofs, etc.. He gave us homework assignments that built on earlier knowledge, and just as we were getting into new material, he'd throw some earlier sequent proofs at us to solve and refresh our memories. Here is the thing - I had _no_ interest in this course, the only thing going for it was it's timing, and repetition. I was _killing_ myself trying to learn integration by parts, linear algebra, discrete math, spending 5 hours outside of class for every hour in class trying to master those other course, and basically just sitting in this one distraction course. I was a C+ student in the math courses (after _massive_ effort) - I walked out of the Propositional Calculus course with an A+. Sixteen years later I look back at University and can honestly say that the most useful course, that I _continue_ to have rock solid knowledge of PLUS use almost every single hour of the day was that stupid Philosphy 210 Propositional Calculus course. For the life of me I can't remember much of graph theory, and wouldn't be able to invert a matrix if you held a gun to my head, but, I'll sit in a meeting and see a ton of possible alternative and immediately start applying Disjunction elimination to break us out of a log jam. There really is something to this spacing method.
- joe_the_user 17y ago
  It could also be that propositional calculus is dead simple - or at least that has always been my impression of it. Calculus and linear algebra work on complex problems in a large domain. Propositional calculus has a small domain and fairly simple theories. I think in general, the knowledge domain is important. Linear algebra is a very unified field by itself but when you get courses that involve a grab bag of algebra, linear algebra, trigonometry and calculus, I think the "stew" they present is incredibly hard to comprehend on a high level and students wind-up either failing or learning by a horrible kind of rote. Uh, this is from biased personal experience. I indeed only taught one college math course when I doing my math MA, it was "Trignometry". It was a complete failure. I still blame the "grab bag" quality of the material but perhaps I was to blame instead.
TobiasCassell 17y ago
Hey OP, was your technique influenced by Mihaly Csikszentmihalyi's Flow? I'm referring to the relationship between ability level and challenge. Do I see similarities with your clearly effective teaching style and some of the elements Video Game designers have almost unanimously implemented to great success?
- butterfi 17y ago
  I'm not sure "the relationship between ability level and challenge" is what MC means by flow, if that's what you're saying. Q: "What is "the flow experience" and what does it have to do with motivation? A: The flow experience is when a person is completely involved in what he or she is doing, when the concentration is very high, when the person knows moment by moment what the next steps should be, like if you are playing tennis, you know where you want the ball to go, if you are playing a musical instrument you know what notes you want to play, every millisecond, almost. And you get feedback to what you're doing. That is, if you're playing music, you can hear whether what you are trying to do is coming out right or in tennis you see where the ball goes and so on. So there's concentration, clear goals, feedback, there is the feeling that what you can do is more or less in balance with what needs to be done, that is, challenges and skills are pretty much in balance." http://www.edutopia.org/mihaly-csikszentmihalyi-motivating-people-learn http://www.edutopia.org/mihaly-csikszentmihalyi-motivating-p...
  - TobiasCassell 17y ago
    (Although it's on the shortlist, I didn't read 'Flow' yet so I may not be able to speak with authority about his work.) Hey cool, thanks for the link. It's been a long time since I was in school but back then every homework assignment was 100% what we covered that day. So if the "lessons" became unclear many folks just got further and further behind and for many there was no recovering at some point because the teachers never wanted to deviate from their schedule. Btillys 1/3 1/3 1/3 structure really resonates with me because I can imagine I would feel confident and never get in over my head as the complexity of the work escalates. (And if I fell behind I could use his 10 minute improvisational time to get clear) But back to the confidence, Video Games Designers want the player to be challenged but also to feel confident and of course this balance has to remain throughout all levels of the game. Video Game Designers have long used MC's work to help them incorporate this balance. So, yes I do believe the relationship between ability and challenge is part of MC's agenda- there is more, of course. It is my understanding that Video Games are the most common contact most people usually have with MC's work. I made a connection between the successful feeling of playing video games with the successful learning experiences in Btillys class's and it would be interesting if Btilly is aware of MC's work or if he had ever thought of what he does as MCesque. In closing, I think you and I are saying the same thing here. Thanks for the comment..
fburnaby 17y ago
I'm just about to start lecturing the tutorials for a linear algebra class next week. The prof that is running the course taught it to me 4 years ago and is the reason I'm now in grad school. She did more or less the same thing that you mention, and it definitely works. I remember all of this material from back when I took the class; it's still familiar and I use it all the time. But despite being very useful for a half to two-thirds of her students, there is always about 1/3 of the class who does terrible in this course. How did you get these people to keep up? Anyway, I'm glad you posted this. It's a good pick-me-up motivator as I'm about to be trying to do the same thing!
cgs 17y ago
I've toyed with the idea of keeping a practice journal, but I've always been too lazy to stick to it. Your comment has inspired me though! This way, I can go back X number of days and review whatever I was working on then.
xccx 17y ago
is this smart.fm? http://www.techcrunch.com/2009/09/10/cerego-raises-3-4-million-for-smart-fm-launches-facebook-friend-quiz http://www.techcrunch.com/2009/09/10/cerego-raises-3-4-milli...
- nwinter 17y ago
  Yes. It used to be called iKnow, and now iKnow is just an app on smart.fm, which is run by Cerego.
juvenn 17y ago
I think a good self-teacher would be better a teacher than who just hold some fancy degrees. Since good self-teachers always know what students need, and where they failed to grasp the essential, and most importantly how to get them out of obstacles. And I bet you must be a good self-teacher. I wish I have ever had you as my teacher!
- antipaganda 17y ago
  There is one glaring fault that a self-taught teacher may have; people learn in different ways. What works for one may not work for all. So the self-taught teacher may not realise that their own personal methods are not necessarily valid for everyone.
- eob 17y ago
  What is a "self-teacher"? The advantage of holding "fancy degrees" is usually that you have been exposed to a wide variety of viewpoints and insignt that self-taught people might have. People tend to study what they are interested in. Unfortunately, this can produce self-taught academics with horse blinders on. Imaging you only got your news from Reddit & Huffington Post, saying you were becoming a self-taught news commentator. You'd probably end up incredibly informed... about liberal viewpoints, perhaps not realizing that a large portion of the country doesn't share them. Getting taught in a formal education system (providing the system works) would expose you to a wider variety of views.
  - juvenn 17y ago
    Completely agreed, providing the system works. I'm not a native En, I mean self-taught teacher? Sorry.
morphir 17y ago
I recently heard that newly educated teachers come to schools with a strong desire to teach their specialties. And to transmit some of their true passion over to their students. But since being a teacher (here in Norway) means they have to administrate and organize the kids so much, they have little energy left for teaching. Meaning the teacher appears as a boring, pale fuck (not very inspiring). This is of course a organizational problem, and not a pedagogical one.
- hyperbovine 17y ago
  Otoh I have always assumed pretty much everyone is pale in Norway. :-)
scharan 17y ago
Please do a more detailed version, if it makes sense to do so. You actually managed to say a lot in this little space. I had a teacher too who did this on a small scale - the first 5 mins was a summary of the previous calls. The first day of each week he summarized the entire chapter or section in the first 10 mins. This bird's eye view of the material went a long way in keeping the concepts 'live',
msluyter 17y ago
Seems related to this Wired article that recently was reposted on Reddit: http://www.wired.com/medtech/health/magazine/16-05/ff_wozniak?currentPage=all http://www.wired.com/medtech/health/magazine/16-05/ff_woznia... Bottom line is that to maximize retention, there's an optimal time for being re-exposed to information: right before you're about to forget it. Seems like re-covering the previous material in the class may have accomplished this.
wsprague 17y ago
The question of why is there so much crappy education is interesting. I think part of the answer is that society is well-served by the failings of the educational system in that widespread educational crappiness helps support the class system. I think it would be fairly straightforward to turn 90% of the population into well educated upper middle class types, using techniques like btilly describes... ... But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it? So I think that a huge function of the educational system is to educate a large part of the population badly. I think that the teacher training system serves this is as well, by selecting for mediocre teachers and then making them more mediocre via training. Additionally, in the schools themselves a lot of effort is made to classify young people into dumb and smart categories, usually unfairly (http://en.wikipedia.org/wiki/Pygmalion_effect http://en.wikipedia.org/wiki/Pygmalion_effect), and this classification stamps them for the rest of their lives and creates a population of hopeless lower class workers. If there are SOME good teachers in the system like btilly, then SOME kids go on to get out of their class, which is perfect in that (1) we need to claim that it is possible to pull yourself up by your bootstraps, and (2) we need to recruit SOME kids to grow up to fill management positions, but not too many that they all can't find careers. Historically, underemployed educated people are the people who become union leaders. The beauty of it is that the people who fail think it is their fault! (Sorry to rant about the educational / class system in general, rather than the topic of how to teach advanced math, but I couldn't stop myself.) Call me paranoid, but there you go.
- seanb 17y ago
  Interesting idea. "But if we did that, who would drink Coors and drive forklift for Walmart and not complain about it?" Robots? If the education system elevated more of the population, there would be a shortage of blue-collar workers, more pressure for system automation, and more well-educated minds to develop the robots. Of course, there would be no shortage of immigrants to fill the blue-collar positions, but perhaps in the long term, a "better" equilibrium would be reached.
martincmartin 17y ago
It's sad that professors are promoted for research, rather than either research or teaching. It amazes me that the private U.S. universities that are most highly ranked for undergraduate education are the ones with the best research.
kalid 17y ago
That's really awesome. I'm a huge proponent of finding the real, actual insights and lessons behind a subject (plug: I blog at betterexplained.com). The dirty little secret of education is that it's "ok" to cram and forget. Most classes reward that; education is about "being able to do something once" vs. actually remembering and applying it. I hope you write more about your findings!
nostrademons 17y ago
When I was learning to play the violin, my teacher had a very strict method for going through the repertoire. You had your working piece, which was supposed to be hard - it was stretching the abilities of what you could do as a violinist. You had your polishing piece, which was your previous working piece where you were finessing all the fine points of technique and musicality. You had review, which was all the other pieces you'd learned so far. And you had your preps, which were little passages (4 measures or so) from your working piece that were so hard that you played them over and over again, way slowed down, until you got them right and could speed them up and incorporate them into your working piece. I've tried to do something similar with my work so far - a working codebase that I'm just learning, a polishing codebase that I basically know my way around in, and various tweaks and bugfixes that I have to do for previous projects. It's a bit harder in the corporate world though - while Google engineers are given a lot of freedom to pick their projects, they're still subject to the needs of the business, and sometimes a project will come up that's such a great opportunity for professional visibility that I'd want to take it even if it involves working hard on something I already know well and dropping the polishing of stuff I just learned.
pw 17y ago
I'd suggest the book Why Don't Students Like School? (http://www.amazon.com/Why-Dont-Students-Like-School/dp/0470279303 http://www.amazon.com/Why-Dont-Students-Like-School/dp/04702...) to those interested in applying the findings of cognitive science to their own learning. I'm finding it very useful in that regard.
araneae 17y ago
I'm wondering what type of university you were at that let you write a final. The most flexibility given to grad students in introductory courses I've seen is, at most, to write a quiz or two. Most don't even let you write homeworks.
- jedliu 17y ago
  Actually, at my department (a "top-10" school for CS), for each course we TA, we're expected to write at least one problem set and come up with at least one question for each exam given. There are also half-courses on intro to Unix or intro to C++ that are taught entirely by grad students. I suspect the OP was teaching an intro-level course, designed to get freshmen up to speed on basic linear algebra, filling any gaps they might have had in their high school curriculum.

cesare 17y ago

Once you've finished the article, there's a lot of reading material on the SuperMemo website: http://www.supermemo.com/english/contents.htm#Articles http://www.supermemo.com/english/contents.htm#Articles Really interesting stuff. The website and the software user interface (at least judging from the screenshots) are terrible, though.

billswift 17y ago
You beat to it. I've had the site on my favorites list for months, it's got more information than the article, but it wasn't anywhere near convincing enough to get me to actually spend any money. Maybe I'll go through it again and see.
- cesare 17y ago
  Have you tried Anki? (it's open source, see GiraffeNecktie comment). I've just downloaded it. There are also some decks shared by people (mostly for natural languages) that you can download from the app itself.
  - peregrine 17y ago
    And GRE vocabulary, I don't even need to take the GRE(maybe) but I think I will study these, improve my vocabulary.

GiraffeNecktie 17y ago

This article is written as if SuperMemo were the only software that uses spaced repetition (SRS) algorithms for learning. Actually there are quite a number of other programs out there including some that are free or open source. I've used SuperMemo for years on my Palm and found it very useful for learning French but the software (at least for the Palm) hasn't been updated for many years and doesn't give the user any control whatsoever over how the program is used. For learning Chinese I've moved to a commercial product called Pleco on the Palm which is basically a dictionary but also has elaborate SRS functionality, it's really light years ahead of SuperMemo. Other more generic SRS programs include Anki and Mnemosyne. Unlike SuperMemo, it's easy to migrate your flashcards among these other applications. One other point, SRS is a huge help to learning vocabulary but learning words is only one part of mastering a language.

cesare 17y ago
Thanks. I was just wondering if there was a similar open source project. Anki: http://ichi2.net/anki/ http://ichi2.net/anki/ Wikipedia entry about this technique: http://en.wikipedia.org/wiki/Spaced_repetition http://en.wikipedia.org/wiki/Spaced_repetition
- sandGorgon 17y ago
  Mnemosyne is actually very good http://www.mnemosyne-proj.org/ http://www.mnemosyne-proj.org/ it works on Win, Mac and Linux - written in Python
  - andymism 17y ago
    Thanks for the rec! The first thing I'm going to be using Mnemosyne for is scheduling guitar practice. I often forget the number of songs and scales that I actually know. Eventually, they get dusty enough that I won't remember them at all. I'm hoping this will help me to retain much better.
  - trapper 17y ago
    I use this when learning a new subject. It's insanely good

rgrieselhuber 17y ago

There is a company based here in Japan called Smart.fm (formerly iKnow) made up of really great people doing similar work.

bhellman1 17y ago

A lot of fluff here but still interesting stuff.

klipt 17y ago

There's at least one maker of language courses (Pimsleur) which use the pacing effect. Unfortunately they're audio only, which at least for me isn't nearly as effective as a combination of audio and writing.

jwecker 17y ago
I've used these- really brilliant. Takes the pacing effect but makes it conversational so it really seems to flow. The trade-off, of course, is that it has no idea how well _you_ have memorized various things (i.e., being prerecorded audio, it is generalized).

tocomment 17y ago

Is there someone I can try this out for free? Are there ready made flashcards for something useful to learn?

tigerthink 17y ago
As mentioned above, Anki and Supermemo are two free implementations. As for ready-made flashcards, here are some I made for a calculus class: http://www.spicyelephant.com/decks/523 http://www.spicyelephant.com/decks/523 http://www.spicyelephant.com/decks/577 http://www.spicyelephant.com/decks/577 http://www.spicyelephant.com/decks/659 http://www.spicyelephant.com/decks/659 You probably have to register for an account if you want the site to apply supermemo-type techniques. I don't recommend using the site in the long term because it will eventually force you to register for $20.
- gwern 17y ago
  Supermemo isn't really free - maybe you meant 'Anki & Mnemosyne'? That's the big 2 in the Free SRS space.
  - tigerthink 17y ago
    Right. Sorry.
r7000 17y ago
You can try my free implementation here: http://flashcarddb.com http://flashcarddb.com. You won't be forced to pay for it to keep working.

nwinter 17y ago

This is what we do at http://www.skritter.com http://www.skritter.com for Chinese and Japanese. Spaced repetition is definitely still in the geek realm, though. Even after automating almost everything (vocab creation, addition, even grading), there are still a lot of people who don't want to practice on the computer, or aren't as interested in efficiency of the learning process. I think as spaced repetition moves away from raw SRS like SuperMemo and toward integration with classrooms, textbooks, and other software that provides content as well, we will start to see more widespread adoption and recognition of its power. There was a recent thread here about whether SRS will evolve to appeal to non-programmers: http://globalmaverick.org/archives/169-repetitio-est-mater-studiorum http://globalmaverick.org/archives/169-repetitio-est-mater-s...

idigthought 17y ago

Is there a good os x and iphone application for this? I'm looking particularly for one that supports images and LaTeX.

gwern 17y ago
There are a number of SRS clones already produced for the iPhone in the store (Mnemosyne, I understand, works fine on OS X); there's also a Mnemosyne clone in early development for the iPhone.

cgs 17y ago

One thing I've never bothered to do is learn an API of X language really well. Because, I thought, I'll just look up whatever I don't know. But I'm constantly stopping to look things up and it really slows me down. Now I'm beginning to think there's really some value in having a strong command of an API. Has anyone tried using an SRS tool to help with this? Or is it simply a matter of not having enough experience with X language?

chimariko 17y ago
I am not sure about SRS for learning API of a language. However I have recently started to use mind maps for that. Whenever I come across a new language or library I take an A4 and a pen and put down all basic concepts. It helps to grasp the big picture of the system and to see the interconnections of different blocks of code. I prefer mind maps to the standard linear up-down way of writing stuff down because the schematic representation utilized in mind mapping eases the twist between the way information is stored on paper and (presumably) in the human memory. All in all I find spending some time on studying docs useful in contrast to the pure work-by-doing approach.
gwern 17y ago
Actually, the SuperMemo site covers just this topic: http://www.supermemo.com/articles/programming.htm http://www.supermemo.com/articles/programming.htm tl;dr: if you will spend more than 5 minutes looking it up or trying to remember it over the next 10/20 years, then it's a net gain to memorize it in a SRS. I find this plausible, but I'm not sure 5 minutes is right - I understand from Anki users (Anki's statistics include 'how much total time ever have I spent looking at/on this particular card') that a card will cost you more than 5 minutes, and more like 10 minutes.

gfodor 17y ago

iPhone version coming out in a few weeks. (I emailed him about this.)