3 ms·

Reminds me of a story I was told while learning how to design experiments and a/b tests. In World War 2, some company was experimenting with the kind of materi

Reminds me of a story I was told while learning how to design experiments and a/b tests. In World War 2, some company was experimenting with the kind of material to use for combat aircraft wings. When they did their initial trials, they saw that the planes coated with the new material came back with more bullet holes. Off hand, that would seem like a reason to dismiss the new material. But, digging into the data, they realized that planes with the old material were actually crashing when they got their first bullet hole and the planes that were making it back, even with bullet holes, all were coated with the new material.

ardy42 6y ago
I think that account gets it a bit wrong. Here's what Wikipedia has: https://en.wikipedia.org/wiki/Survivorship_bias#In_the_military https://en.wikipedia.org/wiki/Survivorship_bias#In_the_milit... > During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire.[10] The Statistical Research Group (SRG) at Columbia University, which Wald was a part of, examined the damage done to aircraft that had returned from missions and recommended adding armor to the areas that showed the least damage, based on his reasoning. This contradicted the US military’s conclusions that the most-hit areas of the plane needed additional armor.[11][12][13] Wald noted that the military only considered the aircraft that had survived their missions; any bombers which had been shot down or otherwise lost had logically also been rendered unavailable for assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Thus, Wald proposed that the Navy reinforce areas where the returning aircraft were unscathed[10]:88, since those were the areas that, if hit, would cause the plane to be lost. His work is considered seminal in the then-nascent discipline of operational research.[14]
- ringshall 6y ago
  That's coincidental; according to Wikipedia, Wald and his wife perished in a plane crash in India.
twostorytower 6y ago
That seems to just be about picking the correct success criteria. It seems here the key success criteria should be which version of the plane makes it back most often. Which plane has more bullet holes is clearly secondary to that.

ardy42 6y ago

I think that account gets it a bit wrong. Here's what Wikipedia has: https://en.wikipedia.org/wiki/Survivorship_bias#In_the_military https://en.wikipedia.org/wiki/Survivorship_bias#In_the_milit... > During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire.[10] The Statistical Research Group (SRG) at Columbia University, which Wald was a part of, examined the damage done to aircraft that had returned from missions and recommended adding armor to the areas that showed the least damage, based on his reasoning. This contradicted the US military’s conclusions that the most-hit areas of the plane needed additional armor.[11][12][13] Wald noted that the military only considered the aircraft that had survived their missions; any bombers which had been shot down or otherwise lost had logically also been rendered unavailable for assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Thus, Wald proposed that the Navy reinforce areas where the returning aircraft were unscathed[10]:88, since those were the areas that, if hit, would cause the plane to be lost. His work is considered seminal in the then-nascent discipline of operational research.[14]

ringshall 6y ago
That's coincidental; according to Wikipedia, Wald and his wife perished in a plane crash in India.

twostorytower 6y ago

That seems to just be about picking the correct success criteria. It seems here the key success criteria should be which version of the plane makes it back most often. Which plane has more bullet holes is clearly secondary to that.