7 ms·
Failing 15% of the time is the best way to learn, say scientists
- cagenut 7y agosee mom those F's on my report card show I was learning optimally
- Merrill 7y agoSo B students are learning fastest, and A students are just idling. OTOH, I graded for a Physics prof where 50/100 on his test was about average.
- kiba 7y agoB and A is just a proxy for effectiveness in scoring. However, you do want to be a "B" student when learning, just not in scoring.
- jfk13 7y agoThis rings so true to me! At school (high school, for you Americans -- I don't mean university), I was an "A" student without really trying. Unfortunately, this meant that I didn't learn to try. Eventually (at a prestigious university) I found I could no longer "coast", and studying required real work. It wasn't easy to come to grips with that reality, and I wish I'd learned earlier.
- Merrill 7y agoA's are in part due to attention to detail - reading the instructions very carefully, interpreting the questions just right, setting up problems as intended by the test writer,...
- sova 7y agoGrades are a bit backwards, because learning actually happens when we 1) make an attempt 2) see the result 3) learn from/reflect on the result. Typically school teaches us to do numbers 1 and 2, with very little emphasis on 3
- geewee 7y agoUnless this article is leaving out some major points, the whole thing seems flawed. So their machine-learning models learn best if they fail 15% of the time. Fair enough - but trying to discern anything meaningful about how often people should fail based on that, seems like quite a bit of a stretch
- kiba 7y agoYeah, the research reported mention machine learning experiment, but they hadn't validated that it also applies to people as well.
- Dumblydorr 7y agoAnecdotally, I'd say I fail most things 95% or more of the attempts. That's why we rewrite, debug, practice, drill, google, avoiding failure takes a lot of human effort.
- jstanley 7y agoIt depends what "failure" even means. If you mis-place a semicolon and have to go back and fix it, have you "failed"? I'd say no. I'd say a project has only failed if it never gets to a working state.
- newnewpdro 7y agoI think failure is defined by expectations. If you failed to succeed in the expected amount of time/effort, it's a failure. Maybe you add some tolerance of going past the estimate before classifying as a failure, but it's still rooted in the expectation. For example, if it took you ten years to pass kindergarten, you've failed.
- infogulch 7y agoHumans get way more information out of each failure than ML systems. When you or I fail just a few times we can analyse the failures and often discover huge classes of wrong behaviors, and never repeat any of them. We're also good at differentiating what parts of the failure caused it and can even learn what parts were successful. We might even test dozens of hypothesis at once in a single attempt, even if we're focusing on just one of them. A computer often only gets one bit of learning from a failure or a success: this single behavior in particular either did or did not work. My hypothesis is that we model the system we're studying and simulate many 'attempts' for every real world attempt. I.e. we grow a low-fidelity, but much faster, model of the system in our brain that we can use to make medium-low confidence predictions about the real system many times for each time we test against the real system. So when you say you fail 95% of the time, I'm saying each of those failures actually have 200 mini-successes embedded that you can still use to train your mental model.
- solicode 7y agoNot too sure about the case in the article and if relates exactly, but it made me think about spaced-reptition systems. I aim for a 80-90% success rate as I've found that to be the optimal range (arrived at that after doing this for 10+ years now with varying settings). I found other articles about this: https://eshapard.github.io/anki/target-an-80-90-percent-success-rate-in-anki.html https://eshapard.github.io/anki/target-an-80-90-percent-succ... https://vladsperspective.wordpress.com/2017/03/14/optimize-your-anki-youre-overtesting-yourself-on-too-few-cards-make-huge-gains/ https://vladsperspective.wordpress.com/2017/03/14/optimize-y...
- beefman 7y agoThe portion 15% seems to crop up suspiciously in optimization contexts... this was noted by Gell-Mann in The Quark and the Jaguar. It's roughly the portion of false warnings sounded by certain tropical birds to gain uncontested access to food. Gell-Mann speculates that it is close to 1/(2pi)...
- yorwba 7y agoIn this case, it is close to 1/2 * (1-erf(1/√2)). No relation to pi, as far as I can tell.
- jacques_chester 7y ago> The portion 15% seems to crop up suspiciously in optimization contexts That's approximately the vale of the area under one tail of a normal distribution, from one standard deviation above the mean to infinity. I'm not statistically mature enough to say whether it's just coincidence. For one thing, oodles of natural phenomena in no way follow the normal distribution.
- kd3 7y agoIt is painful to admit, but you learn best from mistakes and failure.
- bluedino 7y agoI always tell people with cooking, it's not always knowing what to do, but knowing what not to do, and when things go wrong. If you don't fuck it up, people will think your food is at least 'good'. Making it 'great' is the hard part.
- atomicity 7y agoJust because you learn more from failures doesn't mean that you will learn more in aggregate over time with more failures. If you fail too much, your brain will tell you that it's better to quit and spend your attention/time in another way.
- kd3 7y agoElon Musk recently said it well. He said something along the lines of always assuming you are wrong. The goal is to try to be less wrong all the time. Which is basically what good science is. And that guarantees long term improvement and success. So you need to fail a little to learn and get better. If you always succeeded, I think you'd never really know why you succeeded which is important knowledge and guarantees to a certain degree not making those mistakes again in the future.
- julienreszka 7y agoI've written an article about this. https://blog.julienreszka.com/2019/09/growing-success-by-learning-to-learn.html https://blog.julienreszka.com/2019/09/growing-success-by-lea... The subject is so sensitive that I often get banned from forums for saying what I know to be true.
- joker3 7y agoIn mice. Well, you get the idea.
- yorwba 7y agoThe paper is interesting because they calculate the theoretically optimal difficulty for a specific class of learning algorithms: https://dx.doi.org/10.1038/s41467-019-12552-4 https://dx.doi.org/10.1038/s41467-019-12552-4 (I think the method might be applicable for scheduling flashcards better than the rule-of-thumb spacing of Anki et al.) The Independent article is unfounded speculation about this applying to the way humans learn, without any discussion of whether the model is actually applicable. (Most things humans are trying to learn aren't binary classification tasks.)
- Mathnerd314 7y agoThe paper has the Law & Gold model of (monkey) perceptual learning as one of their cases, and has a bit of discussion about how stochastic gradient descent in ML is quantitatively similar to observed characteristics of human learning. I don't think it's unfounded speculation, except perhaps on the part of the paper.
- Mirioron 7y agoWhile human learning might be similar, is it similar enough? Humans seem to require very few data points to learn something. Just a few examples are enough for a human to grasp a pattern. As far as I know, we haven't been able to do the same in ML.
- sjy 7y agoI’m not so sure about that. Adult humans can learn new things with a remarkably small number of additional data points, but they’re drawing on a lifetime of human experience which is also data. Children tend to need more help.
- finebalance 7y agoRight. We have priors about how the world works, and that usually helps.
- orasis 7y agoShouldn’t maximum information gain be found at maximal uncertainty: 50% ?? How is this different from information theory?
- jacques_chester 7y agoLearning in this case is really about recall -- ensuring that information already captured is successfully retrieved. That's a different sense from learning as discovery, or at least, learning as search. In searching a graph of possible hypotheses, yes, it is a better rule to look for opportunities to halve the search space.
- Enginerrrd 7y agoI'd say no, because at that failure rate, there's nothing to be gained by learning over just guessing.
- sova 7y ago1/0.15 means that the optimal failure point is every 1/6 or 1/7 times. Agree?
- mkl 7y agoNo, 1/7 < 0.15 < 1/6. Whether that's an optimal failure level for human learning is a different question that this research doesn't seem to really answer.
- dmix 7y agoWasn't that "10,000 hrs of practice to master anything" paper discredited? This one sounds very similar in trying to quantify a very chaotic and qualitative process. The usefulness of such a stat on any one person is probably nil.
- dragonwriter 7y ago> Wasn't that "10,000 hrs of practice to master anything" paper discredited? No, AFAIK it literally never existed. That was, as I recall, a popular misinterpretation of what was itself an unwarranted generalization made by Malcolm Gladwell based on a paper with much more limited scope and conclusions. > This one sounds very similar in trying to quantify a very chaotic and qualitative process. The actual direct conclusion—that this error rate is optimal for a variety of machine-learning processes—does not seem to ha r the problem you describe. The suggestion in the paper that this extends to “biologically plausible” neural networks that may model animal learning also does not seem problematic in the way you describe. The news article’s claim that this is a finding of a sweet spot for human learning is, while it is a possibility suggested by the paper, simply unwarranted as a conclusion. It's certainly plausible that a quantifiable sweet spot of this type exists for some kinds of human learning at the optimization of effectiveness in a curriculum that can be dynamically scaled to individual learners could effectively be guided by it, but there is not a strong reason without actually testing in concrete human learning scenarios to believe that the particular number here is a guide to that.
- dr_dshiv 7y agoAs though getting 100% on my calculus quizzes indicated that I wasn't learning. Or, that I would learn more if I didn't study as hard and got 85% correct.
- detaro 7y agoNeither of those two examples are about what the article says.
- dr_dshiv 7y agoBut they are presenting their Goldilocks theory as something to generalize, no?
- detaro 7y agoYour quiz example is not about choosing the difficulty of the quiz, which is the variable being considered. > As though getting 100% on my calculus quizzes indicated that I wasn't learning. It doesn't say that. It says you would probably be learning faster if the test would be more difficult. Same category error with the second.
- dr_dshiv 7y ago"you get 100 per cent right all the time and there’s nothing left to learn."
- yorwba 7y agoThe learning process considered in the paper is more like a pretest before each class, and the lesson just reveals the correct answers. If you get 100% on every pretest, you're probably not learning anything new from being told the answers.
- dr_dshiv 7y agoIn that case, I'd be surprised that optimally 85% is best, if it is testing me on stuff I hadn't learned yet. 85% seems till too easy, if it is testing material I haven't had a lesson on yet.
- alexeichemenda 7y agoInterestingly, this correlates well with what is happening in the ad-tech world. Specifically in performance marketing spend, 15% of the budget is very often allocated to "new initiatives & new partners", with the thought process that it'll either allow to find a previously un-identified improvement, or it'll allow to learn what to avoid in the future on the 85% of spend.
- omarhaneef 7y agoOn a binary classification task, it is a priori true that you would likely not learn if you were right 50% or 100% of the time. This is not a function of any particular learning algorithm. If you got 100% right, you already know everything that was being tested. If you got 50% right, you don't know if you are guessing or if you should be picking up on any features. So you would expect that the rate would not be close. 50.1% would be similar to 50% for most intents and purposes. Similarly 99.99%. So you might expect that the optimal learning rate would be close to 75%/25% in general. This would apply to humans too because it is a statement of the information you need to solve the problem, not a statement about the algo. This paper finds it to be 85%/15% for a particular algorithm. Perhaps humans learn similarly, perhaps not. However, you might expect the optimal examples to be somewhere in the 65-85% range for any particular algorithm.
- deleted 7y ago[deleted]
- stjohnswarts 7y agoDude I wish it was only 15% of the time :)
- stjohnswarts 7y agoI'm sure there's some way we can tie this to the golden ratio...
- deleted 7y ago[deleted]
- viig99 7y agoBert has an 15% masking rate, seems co-related, also 90% is what works well when you are trying to do label smoothing using entropy minimisation, what's going on!
- anonytrary 7y agoI have a data point/anecdote! I like to play certain sports. After about a year of intense focus and determination, you can get good at pretty much any sport. What I've noticed though is that on a good day, I'm messing up about 15% of the time. If I mess up significantly more than that, I get discouraged and want to go home and try again the next day. If I'm not messing up enough, I feel like I'm overfitting a particular technique and should probably be messing up more to become more well-rounded.
- blankaccount 7y agoAny tips for picking up soccer? ie. do you have a tim ferris style 80/20 split of what you would work on in the first year?