6 ms·
Note that this is not some sort of unexpected effect due to humans not behaving mathematically ideally; this is what happens when your intuitions about 1, 2 and
by lalaithion 7y ago
Note that this is not some sort of unexpected effect due to humans not behaving mathematically ideally; this is what happens when your intuitions about 1, 2 and 3 dimensions are applied to higher dimensional spaces. Consider the goal of being in the middle 50% of a random value on one dimension; you have a 50% chance. But if you have two dimensions, and you want to be in the middle 50% for both of them, it's a 25% chance. And if you have 3, it's a ~12% chance. This already is counterintuitive, but when you ramp it up to 10 dimensions, it's a ~0.01% chance. That means that if you have a thousand people, only one of them (on average) will be in the middle 50% of all of them. Even the praised end state of the air force, with 9 dimensions and a required support of 90% of individuals of each dimension, implies that only 38% of individuals will be supported overall. (Granted, human dimensions aren't wholly uncorrelated.)
- stygiansonic 7y agoGreat explanation. This is an example of the curse of dimensionality. An intuitive way to think about this is that each dimension you add is another “chance” for one of them to fall outside of the range. Another way to think about it is the often-cited unit n-dimensional sphere. If you were to uniformly sample points from within this n-dimensional sphere, as n increased, the proportion of points lying near the surface of the sphere would increase.
- Balgair 7y agoNumberphile has a good one on how these things can get crazy quickly: https://www.youtube.com/watch?v=mceaM2_zQd8 https://www.youtube.com/watch?v=mceaM2_zQd8
- 6gvONxR4sf7o 7y agoI find it more intuitive to to discard the spatial intuition and think in terms of events. You want something in the X% norm? Think about it like a X%-heads biased coin flip. If you flip it enough times, it'd be shocking to get all heads. Even if the coin flips (dimensions) are correlated, many coin flips (dimensions) means at least some tails (dimension outside the norm).
- clintonc 7y agoThat last bit seems to be the Crux of why this is so surprising -- being in the middle 50% on some dimension correlates positively with being in the middle 50% on the other dimensions, rather than each dimension being independent as in your calculation. It's difficult for me to reconcile that, for the 400 in the study within 5% of average height, each was a standard deviation away from the mean in some other measurement.
- Retric 7y agoIt’s two different effects. First being within a range is not the same as being at the center of the range. Someone 4% above average height should have ~50% of their dimensions larger than that. Similarly someone at 4% below average should have around 50% of their dimensions below that. At best someone in the exact middle of the range only has so much buffer to work with. Second the correlation is less significant than your assuming. My legs are the same length as some people a full foot shorter than I am.
- Talanes 7y agoEvery time I get my height measured by a new doctor, they feel the need to tell me that my legs are short for my height.
- kristianp 7y agoI don't recall ever having my height measured. Is that a U.S. doctor thing?
- joe-collins 7y agoVisits often start with a basic vitals check, but as an adult they've been content to simply ask my height, while measuring my weight.
- clintonc 7y agoSure, but being in the middle 10 percentile in height(or some other dimension) would serve to "normalize" the sample; so 400 people are close to the center of the larger range. Despite being close to the middle of the range, some dimension is far from the center of it's range. Yes, the correlation is not as strong as I would assuming -- that was really the point of my comment. You are a sample size of one, so your anecdote doesn't mean much. However, based on this work, apparently almost everyone has a similar anecdote: after normalizing for height, there is another common dimension which is "unusually" large or small.
- user5994461 7y agoShould note that the study seems to be working on an extremely narrow range, the 30% in middle to quote the article. So measurements in the 35-65% percentile from what I understand. In layman's term. It's so narrow that there are more people 1 inch off than there are people within the expected height. It's crazy. Probability is talking in terms of standard deviations nowadays. They are selecting less than half a standard deviation, it's hyper selective. I'm curious how many people would fit the norm if the study was looking at 1 standard deviation. Surely a lot more. For reference. Selecting the 30% on six metrics is keeping less than 0.01% of participants. Selecting the 68% (one deviation) on six metrics is keeping 10% of participants. It's night and day. Should be even more in practice because measurements are correlated.
- whiddershins 7y ago1 of 10,000 is .01% right?
- saeranv 7y agoWas going to make the same comment, should be 0.1 people out of 1000.
- MauranKilom 7y agoOr to work the example in the article: Being in the middle 30% of any three traits has a 2.7% chance (matching the quoted 3%). For ten traits it's a 1e-6 chance.
- saeranv 7y agoThis comment was what made this article make sense to me. Just to add to add to the mathematical intuition here (please correct me if I'm wrong): if you're thinking of it as a unit line/square/cube then total n-dim area is 1^n, and the portion in the 50% range is (1/2)^n, where n = number of dimensions. So that should simplify to 2^(-n). Note this works out to 10th dimension as 0.09765% or 0.1 person per 1000. ETA: As one of the comments below points out, you can also model it as a binomial distribution. Probability of getting all heads, given p=0.5 is (n!)/((n-k)!k!) / 2^n. Since n=k since we're looking to get all heads at all times, this also simplifies to 2^(-n).
- spappal 7y agoP(simultaneously within middle 50 % of n independent distributions) = 0.5^n = 2^(-n), like my parent comment says. But interestingly, every relevant comment here either got something wrong about the final frequency or percentage, or corrected the wrong thing. (Parent comment, grandparent, 2 aunts and 1 cousin.) 0.5^10 = 2^-10 = 1/(2^10) = 1/1024 [i.e. exactly 1 person per 1024] ~= 0.00097656 ~= 0.09766 %.
- saeranv 7y agoHah, you are correct. I even had access to numpy and still managed to get this wrong.
- azinman2 7y agoThis is only true if each dimension is independent and identically distributed random variables, which is doubtfully true for humans. That said, the overall intuition is probably more correct than incorrect...
- kube-system 7y agoThey are neither fully independent nor fully dependent. It falls somewhere in the middle. The same will apply, just in a slightly less dramatic effect.
- JackFr 7y agoBut surely the engineers were who were inventing jets and rocket guidance systems and radar understood probabilities of independent events. The point, which the author almost misses completely, is that they dramatically overestimated how correlated the measurements were. Beyond that, while there's no accounting for taste, I find it to be an appallingly bad article. It's poor man's Malcolm Gladwell.
- kube-system 7y agoUnderstanding the probabilities of independent events is a different thing than identifying them. It is very common for engineering projects to encounter problems caused by an issue which was understood but was not identified. Most engineering failures fall into this category. Humans simply make mistakes.
- mrtnmcc 7y agonit: 0.01% is 1 in ten thousand.
- testerhn 7y agohow did you derive 38%?