5 ms·

Glad you like my work. To quote the essay: "these kinds of results should be taken with a grain of salt. The mnemonic medium is in its early days, has many de

Glad you like my work. To quote the essay: "these kinds of results should be taken with a grain of salt. The mnemonic medium is in its early days, has many deficiencies, and needs improvement in many ways... is this medium really working? What effect is it actually having on people? ... are there blockers that make this an irredeemably bad or at best mediocre idea? How important a role does memory play in cognition, anyway?" The essay continues at great length in this vein, getting gradually into more and more detail. We have detailed discussion of common ways spaced repetition fails, of ways it may inevitably fail or needs to be redesigned (a different issue), of ways it falls short of the goal of understanding, of ways spaced repetition is a poor lens for memory systems, and so on. I haven't counted, but believe there's several thousand words in this vein. The conclusion is that: (a) we are some ways from having really good memory systems; but if we did it would (b) likely to be extremely helpful, including in lots of ways people don't commonly appreciate; and (c) be far from a panacea. The essay has a lengthy discussion of points (a) and (c), in addition to (b). I don't think it's reasonably characterized as hype.

westoncb 7y ago
I was curious to read the self-criticism section since I fall into the memory-skeptic camp myself. And I did in fact find my main concerns referenced there, though my impression is that they weren't taken seriously. One telltale sign is the condescending tone with which they're referenced: > Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious. The other signal is the convenient changeup of exemplar subjects, natural language vs. computer programming. In the above, for example, having a "broad conceptual understanding" of French is obviously useless—and yet having a broad conceptual understanding of topics in e.g. Computer Science is extremely useful because: 1) It's part of a map that allows you to jump deeply into related as areas if a demand for that knowledge actually arises. 2) The 'broad conceptual understanding' (if done well) is generally of 'basic principles,' which have an interesting property of being applicable to wide ranges of much more specific subject areas. > Another common argument against spaced repetition systems is that it's better to rely on natural repetition. A more fair angle on this argument is that it's better to rely on natural contexts instead of 'repetition'—this is the reason "brain training" apps fail, for instance. (SRS obviously isn't as bad as those, but it still certainly ranks lower than a 'natural context' in regards to imparting skill in that context.) > For instance, it's no good (but surprisingly common) for someone to memorize lots of details of a programming language they plan to use for just one small project. ... But the truths of the last paragraph also have limits. If you're learning French, ... This is another sneaky subject change. It's pointed out how using memory systems for learning programming languages is probably non-optimal. But then this next paragraph is supposed to show us the limits of the supposed non-optimality—but it never does, because it switches to talking about a natural language instead, using an argument about lacking interlocutors which is completely inapplicable to the programming languages example. In any case, there are things about this project that I like very much, but I do still suspect there are subtle flaws in the premise of what this would end up making people better at, and the tone approaches Wolfram-esque levels of grandiosity that makes it hard to take seriously.
- jacobolus 7y ago
  >> Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious. The way people learn French (or any language) is not by memorizing atomized vocabulary definitions on flash cards, but by listening to / reading a large amount of comprehensible input, which is to say, meaningful grammatically correct sentences in their context. Memorizing vocabulary might help some self-studying language students bootstrap enough very basic understanding to make a wider variety of inputs comprehensible, in the absence of a teacher or properly-designed curriculum, but in and of itself does not comprise learning the language. People who encounter a large amount of appropriate-level language in context do end up with what might be called “intuitive understanding” of language (or maybe “subconscious understanding”; “conceptual” is not a great label, IMO). They know how to recognize and later how to form grammatical sentences because they have been exposed to a wide variety of such sentences. They learn the subtle connotations of words (beyond the definitions offered in an English–French dictionary) by seeing the words used many times in context. Etc. From what I have read, speaking per se is just not that useful at the earliest stages (at least the first several months) when learning a language. >> It seems plausible, though needs further study, that the mnemonic medium can help speed up the acquisition of such chunks, and so the acquisition of mastery. This whole argument basically boils down to speculation.
pso 7y ago
I've googled generally and verbatim with various quotes from above and your name, and cannot find the essay. It sounds interesting. Can you post a link please.
reubens 7y ago
Thanks for your response Michael, and my apologies for the temerity of my initial comment. At the end of four years of Med school, I am truly Anki fatigued (and a bit fed up with learning facts about rare diseases I may never see in practise). The points you highlight here I recall being present to a degree in Quantum Country, so I really don’t have a leg to stand on- apologies for directing my frustration toward your work.
- michael_nielsen 7y ago
  Thanks, no worries. Sorry to hear you're feeling a bit fed up with Anki / parts of Med School.
DoctorOetker 7y ago
In the essay, the importance of dependency on previous concepts is touched upon multiple times. I essentially see 2 distinct "problems" or tasks that would need to be done in this specific respect (without downplaying the value of other factors): 1) it takes effort to encode the dependency relationships, 2) even if someone puts in a lot of work for a specific subject or domain to encode the dependency relationships, we need to modify the tools to take advantage of this knowledge. I would propose as a next testbed, one takes the MetaMath database (set.mm), since dependency of a theorem on other theorems or axioms can be extracted in automated fashion. So I would love to see a tool (a working title could be Anki/MetaDrill or "autodidact" was the title I had in mind) that presents me with the following exercises: type 0a, 0b: given ascii characters select LaTeX symbol (a) and vice versa (b) [for example ".-" for negation etc...] type 1: given theorem/hypothesis/axiom/definition abbreviation name: produce the statement type 2: given theorem/hypothesis/axiom/definition statement: produce the abbreviation (the reverse of exercise type 1, both can be easily extracted from an instance of a verifier that has parsed the set.mm database) type 3: given a couple of input statement, and an output statement: produce the theorem abreviation that justifies this step (can be easily done by randomly selecting a target theorem from those scheduled for the user to know, then searching all proofs in the database for steps that reference the target theorem, each reference to the theorem is a utilization or example application of the theorem) type 4: given a theorem: produce the list of theorems directly used in its proof (this one depends on the proof used, so that it is possible to find proofs that don't rely on those in the database, but is easy to generate for the proofs proposed in the database: just enumerate all the references in the proof of this theorem) type 5: given a theorem (by abbreviation or by statement): produce a proof (the previous series of exercises should provide all the prerequisites, and the last exercise above would contain a lot of hints to reproduce the proof) it is possible to find an original proof, and the MetaMath verifier can be used to check the user's proof! the exploration or presentation of new axioms, definitions, theorems, can be limited to those "in view" to the user, using the dependency graph of theorems, so only a minimal number of new concepts is introduced at a time. I.e. this prevents the user from being bombarded with uncontextualized facts like "the connected sum of a torus with a sphere is a torus" without first learning about addition of rational numbers... once a large number of people can quickly get up to speed with MetaMath, they will see the utility of spaced repitition, and understand that formalization is the way forward, since formalization automatically forces you to be explicit in dependencies, and forces people to state their beliefs as exactly as possible, or alternatively as quickly as possible (be bold), but such that others can prove errors in their conjectures by providing proof that the bold person happened to introduce an inconsistency. But the biggest advantage of logical formalization of knowledge is that all the above exercises can be automatically generated, no one needs to produce flash cards! not only can they be generated automatically, they can be generated as a function of the past performance of the user on those exercises, instead of waiting a long period of time before retesting prerequisites 2 or 3 levels deep, we can test already seen knowledge that relies on it, and upon failure schedule the immediate prerequisites it relies on 1 level deeper, most of which will succeed, but one of which will typically fail again prompting it's lower level prerequisites to be tested (2 levels deeper from the original failure) to finally prompt a last failure, upon which the 3rd level lower prerequisites are reached but all succeed, just like how a professor in an oral exam probes where exactly the student got stuck. This allows to test for gaps in working memory to be detected without explicitly testing all the lower levels, just like the blood test during the second world war: at some point an expensive but very sensitive test was developed, and instead of testing the soldiers individually, their blood was mixed then tested, and in this manner it was possible to cheaply scan for positives and only zoom in when a group had a positive. I can not find the link to the relevant wikipedia page sadly.

westoncb 7y ago

I was curious to read the self-criticism section since I fall into the memory-skeptic camp myself. And I did in fact find my main concerns referenced there, though my impression is that they weren't taken seriously. One telltale sign is the condescending tone with which they're referenced: > Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious. The other signal is the convenient changeup of exemplar subjects, natural language vs. computer programming. In the above, for example, having a "broad conceptual understanding" of French is obviously useless—and yet having a broad conceptual understanding of topics in e.g. Computer Science is extremely useful because: 1) It's part of a map that allows you to jump deeply into related as areas if a demand for that knowledge actually arises. 2) The 'broad conceptual understanding' (if done well) is generally of 'basic principles,' which have an interesting property of being applicable to wide ranges of much more specific subject areas. > Another common argument against spaced repetition systems is that it's better to rely on natural repetition. A more fair angle on this argument is that it's better to rely on natural contexts instead of 'repetition'—this is the reason "brain training" apps fail, for instance. (SRS obviously isn't as bad as those, but it still certainly ranks lower than a 'natural context' in regards to imparting skill in that context.) > For instance, it's no good (but surprisingly common) for someone to memorize lots of details of a programming language they plan to use for just one small project. ... But the truths of the last paragraph also have limits. If you're learning French, ... This is another sneaky subject change. It's pointed out how using memory systems for learning programming languages is probably non-optimal. But then this next paragraph is supposed to show us the limits of the supposed non-optimality—but it never does, because it switches to talking about a natural language instead, using an argument about lacking interlocutors which is completely inapplicable to the programming languages example. In any case, there are things about this project that I like very much, but I do still suspect there are subtle flaws in the premise of what this would end up making people better at, and the tone approaches Wolfram-esque levels of grandiosity that makes it hard to take seriously.

jacobolus 7y ago
>> Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious. The way people learn French (or any language) is not by memorizing atomized vocabulary definitions on flash cards, but by listening to / reading a large amount of comprehensible input, which is to say, meaningful grammatically correct sentences in their context. Memorizing vocabulary might help some self-studying language students bootstrap enough very basic understanding to make a wider variety of inputs comprehensible, in the absence of a teacher or properly-designed curriculum, but in and of itself does not comprise learning the language. People who encounter a large amount of appropriate-level language in context do end up with what might be called “intuitive understanding” of language (or maybe “subconscious understanding”; “conceptual” is not a great label, IMO). They know how to recognize and later how to form grammatical sentences because they have been exposed to a wide variety of such sentences. They learn the subtle connotations of words (beyond the definitions offered in an English–French dictionary) by seeing the words used many times in context. Etc. From what I have read, speaking per se is just not that useful at the earliest stages (at least the first several months) when learning a language. >> It seems plausible, though needs further study, that the mnemonic medium can help speed up the acquisition of such chunks, and so the acquisition of mastery. This whole argument basically boils down to speculation.

pso 7y ago

I've googled generally and verbatim with various quotes from above and your name, and cannot find the essay. It sounds interesting. Can you post a link please.

reubens 7y ago

Thanks for your response Michael, and my apologies for the temerity of my initial comment. At the end of four years of Med school, I am truly Anki fatigued (and a bit fed up with learning facts about rare diseases I may never see in practise). The points you highlight here I recall being present to a degree in Quantum Country, so I really don’t have a leg to stand on- apologies for directing my frustration toward your work.

michael_nielsen 7y ago
Thanks, no worries. Sorry to hear you're feeling a bit fed up with Anki / parts of Med School.

DoctorOetker 7y ago

In the essay, the importance of dependency on previous concepts is touched upon multiple times. I essentially see 2 distinct "problems" or tasks that would need to be done in this specific respect (without downplaying the value of other factors): 1) it takes effort to encode the dependency relationships, 2) even if someone puts in a lot of work for a specific subject or domain to encode the dependency relationships, we need to modify the tools to take advantage of this knowledge. I would propose as a next testbed, one takes the MetaMath database (set.mm), since dependency of a theorem on other theorems or axioms can be extracted in automated fashion. So I would love to see a tool (a working title could be Anki/MetaDrill or "autodidact" was the title I had in mind) that presents me with the following exercises: type 0a, 0b: given ascii characters select LaTeX symbol (a) and vice versa (b) [for example ".-" for negation etc...] type 1: given theorem/hypothesis/axiom/definition abbreviation name: produce the statement type 2: given theorem/hypothesis/axiom/definition statement: produce the abbreviation (the reverse of exercise type 1, both can be easily extracted from an instance of a verifier that has parsed the set.mm database) type 3: given a couple of input statement, and an output statement: produce the theorem abreviation that justifies this step (can be easily done by randomly selecting a target theorem from those scheduled for the user to know, then searching all proofs in the database for steps that reference the target theorem, each reference to the theorem is a utilization or example application of the theorem) type 4: given a theorem: produce the list of theorems directly used in its proof (this one depends on the proof used, so that it is possible to find proofs that don't rely on those in the database, but is easy to generate for the proofs proposed in the database: just enumerate all the references in the proof of this theorem) type 5: given a theorem (by abbreviation or by statement): produce a proof (the previous series of exercises should provide all the prerequisites, and the last exercise above would contain a lot of hints to reproduce the proof) it is possible to find an original proof, and the MetaMath verifier can be used to check the user's proof! the exploration or presentation of new axioms, definitions, theorems, can be limited to those "in view" to the user, using the dependency graph of theorems, so only a minimal number of new concepts is introduced at a time. I.e. this prevents the user from being bombarded with uncontextualized facts like "the connected sum of a torus with a sphere is a torus" without first learning about addition of rational numbers... once a large number of people can quickly get up to speed with MetaMath, they will see the utility of spaced repitition, and understand that formalization is the way forward, since formalization automatically forces you to be explicit in dependencies, and forces people to state their beliefs as exactly as possible, or alternatively as quickly as possible (be bold), but such that others can prove errors in their conjectures by providing proof that the bold person happened to introduce an inconsistency. But the biggest advantage of logical formalization of knowledge is that all the above exercises can be automatically generated, no one needs to produce flash cards! not only can they be generated automatically, they can be generated as a function of the past performance of the user on those exercises, instead of waiting a long period of time before retesting prerequisites 2 or 3 levels deep, we can test already seen knowledge that relies on it, and upon failure schedule the immediate prerequisites it relies on 1 level deeper, most of which will succeed, but one of which will typically fail again prompting it's lower level prerequisites to be tested (2 levels deeper from the original failure) to finally prompt a last failure, upon which the 3rd level lower prerequisites are reached but all succeed, just like how a professor in an oral exam probes where exactly the student got stuck. This allows to test for gaps in working memory to be detected without explicitly testing all the lower levels, just like the blood test during the second world war: at some point an expensive but very sensitive test was developed, and instead of testing the soldiers individually, their blood was mixed then tested, and in this manner it was possible to cheaply scan for positives and only zoom in when a group had a positive. I can not find the link to the relevant wikipedia page sadly.