4 ms·

When the flight instructor story and regression to the mean were discussed on HN a year ago[1], user "cortesoft" had a brilliant way to convince you: Yeah, I a

When the flight instructor story and regression to the mean were discussed on HN a year ago[1], user "cortesoft" had a brilliant way to convince you: Yeah, I always make an example with coin flips to show how this is true.... lets say heads is success and tails is failure. Flip 100 coins. Take the ones that 'failed' (landed tails) and scold them. Flip them again. Half improved! Praise the ones that got heads the first time. Flip them again. Half got worse :( Clearly, scolding is more effective than praising. [1] https://news.ycombinator.com/item?id=12707606 https://news.ycombinator.com/item?id=12707606

cousin_it 9y ago
This just became my favorite cautionary story about science. Thanks to you and cortesoft!
- ataturk 9y ago
  It's more of a cautionary tale about dealing with people--I can't count the number of times I've been in some kind of work situation where the people above me believed in some absolute folklore that was detrimental to the execution of the work but they would 100% argue with anyone trying to be reasonable about what was really going on. It's like a learned helplessness, or some kind of odd comfort with the status quo. They would invent from whole cloth a whole belief system about how some software worked that was completely irrational. I include IT admin people here, not just non-techies. And the thing is, I've seen it happen more than once amongst otherwise intelligent people! So then you're in a situation where the believers begin to exert their authority over you in order to control the situation and back up their beliefs instead of trying to solve the problem, up to and including threats of termination. It's as though they might suffer emotional harm if you fix the actual underlying problems that caused the whole need to create the belief system in the first place. Those behaviors are that strong and that lizard-brained. One example: A transaction issue (actually the lack of a proper transaction) in a database was causing duplicate order numbers to occur pretty frequently each day on a large, multi-user system. Instead of getting someone competent to come in to analyze and fix the issue, the people at the company developed all these elaborate procedures for how to enter orders so that they could avoid creating duplicates in the system. They had actual written procedures and even the work schedules were affected by this order entry bug. I swear they were twirling around three times and throwing salt over one shoulder before entering a new order, but only on odd-numbered Wednesdays and Fridays. I've seen similar problems involving thread local data and badly written multithreaded code, odd caching issues on networks relating to AJAX calls from certain browsers, network setup issues, you name it. There, but for the grace of God, go I.
le-mark 9y ago
Really brilliant. I had an experience in basic training shooting a 9 millimeter pistol at a target using a 'laser trainer'. I got a perfect bullseye the first shot. The drill sergeants were highly impressed. One particularly savvy sergeant (must have read this article?) Immediately said, "do that again". And I did not make a bullseye again, and everyone went about their business yelling at privates, and getting yelled at. That particular drill sergeant knew something about probabilities apparently.
linkmotif 9y ago
It’s easy to divorce morality from coin tosses, but much harder from people who ought to be concentrating and making an effort.
Ntrails 9y ago
I'm really struggling with making sense of this. Assuming that you're assigning exactly 50/50 results in each case: Half the heads, having been praised were successful again. That's implying that praise was 50% effective. Half the tails, having been scolded, were suddenly successful. That's implying that scolding was 50% effective. How is scolding more effective than praising?
- contravariant 9y ago
  The point of this example is that there is (obviously) no difference in effectiveness. However compare this simplified example with the following citation from the article: >“On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don't tell us that reinforcement works and punishment does not, because the opposite is the case.”
- dragonwriter 9y ago
  With praise, half got worse and half stayed the same. With scolding, half got better and half stayed the same. Scolding is obviously better, in terms of effect on outcomes. Though it's probably a better parallel to the flight instructor case if you have a six-sided die, and you praise on 5-6, and scold on 1-2: most of the praised get worse, most of the scolded get better, those who get neither scolding or praise have mixed results, and a few of those scolded get worse and likewise a few praised get better. The coin flip example, while in one sense more clearly showing the problem, simplifies to the point where the connection with realistic scenarios is harder to see, IMO.
  - DenisM 9y ago
    In other words Gaussian distributions will reliably revert to the mean, whereas feedback will always push to the right. Those praised, already on the right, will move against the desired direction, left, towards the mean. Those scolded, already at the left will move in the desired direction right, also towards the mean.
- 9y ago
jacinabox 9y ago
You can also consider that flight training (or any skill really) is not a situation where you can really top out the range. Cadets can always get better. When they do, the standards go up.
ythn 9y ago
What if it wasn't 50/50 though, and the flight instructor took actual measurements and found that 80% of praised cadets did worse on the second try (vs. no praise) and 80% of scolded cadets did better on the second try (vs. no scolding)? Wouldn't that indicate that indeed no praise + lots of scolding is the best approach?
- fixermark 9y ago
  It might if the flight instructor also controlled for what circumstances he hands out praise and scolding. If it's a Gaussian distribution of performance, and he praises above the 99th percentile of bad performance and scolds below the 99th percentile, that 80%-80% pattern might very well occur---but again, it'd be expected to occur, independent of the signal he's putting into the system (because the candidate's he's scolding are staggeringly unlikely to get worse, and the candidates he's praising staggeringly unlikely to get better).
turc1656 9y ago
This, and the original link, base this concept on the idea that the task being monitored is subject to random chance. But the original link is talking about an act that requires skill and something that a person is trained for. So the analogy of the coin toss doesn't make sense to me, nor does the original link. It's like saying that just because I was able to parallel park successfully this time in one maneuver without hitting the curb or either of the cars in front or behind me that I have some sort of increased chance of not being successful the next time. That seems kind of absurd, doesn't it?
- ballenf 9y ago
  Don't equate things that aren't 50-50 with not having a degree of chance. Parallel parking and aerial maneuvers are not devoid of chance playing a role in success. The impact of chance does decrease with skill, but skill doesn't remove the element of chance. So the coin is an extreme example, not an irrelevant example.
- KirinDave 9y ago
  One of the most pervasive myths we humans cling to is the idea of agency; that our skill and will an overcome random chance. While we might be able to undertake methodologies that skew chance distributions in our favor, everything around us is a game of probability we can only gently influence. Methods are MUCH more important for a career than any individual execution.
tritium 9y ago
Man, that example knocks it out of the park. I had my own retort to the basic concept behind the article, but that one tears it to shreds. Anyway, the tendency of the article pushes negative reinforcement, not just anecdotally, but also as a continuous prevailing message, instead of alternating with degrees of praise even if only on occassion. There are at least three things wrong with that on a behavioral level: a. resentment as an inversion of stockholm syndrome b. learned helplessness, and c. simple fatigue. In other words, people will hate your idea of leadership, conclude that you can never be pleased, and realize that they'll be worked to death on a treadmill of demands, if they don't jump ship sooner or later.

cousin_it 9y ago

This just became my favorite cautionary story about science. Thanks to you and cortesoft!

ataturk 9y ago
It's more of a cautionary tale about dealing with people--I can't count the number of times I've been in some kind of work situation where the people above me believed in some absolute folklore that was detrimental to the execution of the work but they would 100% argue with anyone trying to be reasonable about what was really going on. It's like a learned helplessness, or some kind of odd comfort with the status quo. They would invent from whole cloth a whole belief system about how some software worked that was completely irrational. I include IT admin people here, not just non-techies. And the thing is, I've seen it happen more than once amongst otherwise intelligent people! So then you're in a situation where the believers begin to exert their authority over you in order to control the situation and back up their beliefs instead of trying to solve the problem, up to and including threats of termination. It's as though they might suffer emotional harm if you fix the actual underlying problems that caused the whole need to create the belief system in the first place. Those behaviors are that strong and that lizard-brained. One example: A transaction issue (actually the lack of a proper transaction) in a database was causing duplicate order numbers to occur pretty frequently each day on a large, multi-user system. Instead of getting someone competent to come in to analyze and fix the issue, the people at the company developed all these elaborate procedures for how to enter orders so that they could avoid creating duplicates in the system. They had actual written procedures and even the work schedules were affected by this order entry bug. I swear they were twirling around three times and throwing salt over one shoulder before entering a new order, but only on odd-numbered Wednesdays and Fridays. I've seen similar problems involving thread local data and badly written multithreaded code, odd caching issues on networks relating to AJAX calls from certain browsers, network setup issues, you name it. There, but for the grace of God, go I.

le-mark 9y ago

Really brilliant. I had an experience in basic training shooting a 9 millimeter pistol at a target using a 'laser trainer'. I got a perfect bullseye the first shot. The drill sergeants were highly impressed. One particularly savvy sergeant (must have read this article?) Immediately said, "do that again". And I did not make a bullseye again, and everyone went about their business yelling at privates, and getting yelled at. That particular drill sergeant knew something about probabilities apparently.

linkmotif 9y ago

It’s easy to divorce morality from coin tosses, but much harder from people who ought to be concentrating and making an effort.

Ntrails 9y ago

I'm really struggling with making sense of this. Assuming that you're assigning exactly 50/50 results in each case: Half the heads, having been praised were successful again. That's implying that praise was 50% effective. Half the tails, having been scolded, were suddenly successful. That's implying that scolding was 50% effective. How is scolding more effective than praising?

contravariant 9y ago
The point of this example is that there is (obviously) no difference in effectiveness. However compare this simplified example with the following citation from the article: >“On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don't tell us that reinforcement works and punishment does not, because the opposite is the case.”
dragonwriter 9y ago
With praise, half got worse and half stayed the same. With scolding, half got better and half stayed the same. Scolding is obviously better, in terms of effect on outcomes. Though it's probably a better parallel to the flight instructor case if you have a six-sided die, and you praise on 5-6, and scold on 1-2: most of the praised get worse, most of the scolded get better, those who get neither scolding or praise have mixed results, and a few of those scolded get worse and likewise a few praised get better. The coin flip example, while in one sense more clearly showing the problem, simplifies to the point where the connection with realistic scenarios is harder to see, IMO.
- DenisM 9y ago
  In other words Gaussian distributions will reliably revert to the mean, whereas feedback will always push to the right. Those praised, already on the right, will move against the desired direction, left, towards the mean. Those scolded, already at the left will move in the desired direction right, also towards the mean.
9y ago

jacinabox 9y ago

You can also consider that flight training (or any skill really) is not a situation where you can really top out the range. Cadets can always get better. When they do, the standards go up.

ythn 9y ago

What if it wasn't 50/50 though, and the flight instructor took actual measurements and found that 80% of praised cadets did worse on the second try (vs. no praise) and 80% of scolded cadets did better on the second try (vs. no scolding)? Wouldn't that indicate that indeed no praise + lots of scolding is the best approach?

fixermark 9y ago
It might if the flight instructor also controlled for what circumstances he hands out praise and scolding. If it's a Gaussian distribution of performance, and he praises above the 99th percentile of bad performance and scolds below the 99th percentile, that 80%-80% pattern might very well occur---but again, it'd be expected to occur, independent of the signal he's putting into the system (because the candidate's he's scolding are staggeringly unlikely to get worse, and the candidates he's praising staggeringly unlikely to get better).

turc1656 9y ago

This, and the original link, base this concept on the idea that the task being monitored is subject to random chance. But the original link is talking about an act that requires skill and something that a person is trained for. So the analogy of the coin toss doesn't make sense to me, nor does the original link. It's like saying that just because I was able to parallel park successfully this time in one maneuver without hitting the curb or either of the cars in front or behind me that I have some sort of increased chance of not being successful the next time. That seems kind of absurd, doesn't it?

ballenf 9y ago
Don't equate things that aren't 50-50 with not having a degree of chance. Parallel parking and aerial maneuvers are not devoid of chance playing a role in success. The impact of chance does decrease with skill, but skill doesn't remove the element of chance. So the coin is an extreme example, not an irrelevant example.
KirinDave 9y ago
One of the most pervasive myths we humans cling to is the idea of agency; that our skill and will an overcome random chance. While we might be able to undertake methodologies that skew chance distributions in our favor, everything around us is a game of probability we can only gently influence. Methods are MUCH more important for a career than any individual execution.

tritium 9y ago

Man, that example knocks it out of the park. I had my own retort to the basic concept behind the article, but that one tears it to shreds. Anyway, the tendency of the article pushes negative reinforcement, not just anecdotally, but also as a continuous prevailing message, instead of alternating with degrees of praise even if only on occassion. There are at least three things wrong with that on a behavioral level: a. resentment as an inversion of stockholm syndrome b. learned helplessness, and c. simple fatigue. In other words, people will hate your idea of leadership, conclude that you can never be pleased, and realize that they'll be worked to death on a treadmill of demands, if they don't jump ship sooner or later.