7 ms·

Why the Normal Distribution Is Vanishing

23 points by Futurebot 11y ago

sz4kerto 11y ago
This looks bullshit to me. He says the normal distribution is vanishing, and to prove himself, he brings up some charts which do not look like a normal distribution. Like the first smartphone-related chart: "You’re looking to buy a new smartphone. What does your demand distribution look like? Is it normally distributed? Absolutely not. It looks like this:" [ non-Gaussian 'curve' ] Okay, what is a 'demand distribution'? I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc.
- pzone 11y ago
  Pretty sure author is just making stuff up. There's no content to this at all. One has no ground to make such empirical claims without showing any data backing it up.
- LoSboccacc 11y ago
  whole article comes only with 'hunches' and not a single data point researched. it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones. one could argue that those are sold in lower volumes, but then that'd be a gaussian distribution also, of course economics stats are not a good fit for a gaussian. they tend to stabilize around exponential. that's why the price pyramid is a pyramid. I have to concede that by looking at some data, it seems there is a exponential distribution going on for all price ranges except at the highest end, but that's more a hint of the product price/features space being discontinuous than we being in a post scarcity society where gaussian/exponential distributions suddenly vanish. but anyway, the fact remains that the data is out there, even for the other field, this on smartphone is but one of the many he didn't research preferring to follow intuition. and human brain is very bad at applying intuition to statistics. http://image.slidesharecdn.com/tomiahonen-engagewithmobile-wsa-abudhabi2015-150220083444-conversion-gate01/95/tomi-ahonen-engage-with-mobile-at-wsamobile-global-congress-abu-dhabi-38-638.jpg?cb=1424442956 http://image.slidesharecdn.com/tomiahonen-engagewithmobile-w... http://4.bp.blogspot.com/-uY7kRAl6XfA/URibddfrTqI/AAAAAAAAPLI/pnN9ivxjtt0/s1600/Smartphone+Sales+Distibution+by+ASP.png http://4.bp.blogspot.com/-uY7kRAl6XfA/URibddfrTqI/AAAAAAAAPL...
  - coldtea 11y ago
    >it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones. He absolutely does not ignore that fact, in fact he specifically says that the better end of the market is either iOS or high-end Android phones. Obviously (and all market data point to that), the Android phones that sell more are not the high end units, but the cheapo units that dominate the "free with subscription", "prepaid" etc market.
- ekianjo 11y ago
  > This looks bullshit to me It's obviously bullshit since the guy analyses ZERO data and says the normality is dead. Obligatory facepalm. And obviously Android smartphones pricing look NOTHING like what he has drawn, there are cheap as well as high end Android devices out there at about every price point.
- omginternets 11y ago
  This is what's commonly -- and pejoratively -- referred to in the sciences as "logic without evidence".
  - commentzorro 11y ago
    > "logic without evidence" When I was in school we called these people "philosophy majors." ;-)
    - omginternets 11y ago
      I know you're saying this in jest, but a good philosopher will account for empirical knowledge in his arguments, when it exists. I know it's cool to rip on philosophers, but they're incredibly useful to the sciences: they're usually the first ones to point out circular (or otherwise fallacious) reasoning.
- coldtea 11y ago
  >I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc. And I'm pretty sure it does look like that, judging from all the people I know (most either have a crappy whatever smartphone or a high end iOS/Android model, with the middle ground, which is the main bulk of a normal distribution under-represented). It also matches all information I've seen about the mobile market, user behavior (e.g. how iOS dominates profits and mobile views while still having a much smaller userbase, etc).
ddebernardy 11y ago
> The world’s shift from Normal, Gaussian distributions of demand towards bifurcated, two-tiered distributions is a natural consequence of our shift from a world governed by scarcity to one governed by abundance. FWIW, others (e.g. Steve Keen, of "Debunking Economics" fame) would convincingly argue that the world never fit into economist make believe normal distributions to begin with.
pjc50 11y ago
This article hints at the inequality implications, but I think they're likely to become increasingly important. We're already seeing this with creative works: the vast majority of works earn zero or near-zero, apart from a few massive global hits. Inequality obviously also affects people's ability and willingness to pay for things. The lower end products are not just cheap but increasingly "free": either ad-surveillance-supported or paid for by the exploitative gambling-like "free to play" systems. Again, it's well known in F2P that most of the revenue comes from a tiny fraction of users.
princeb 11y ago
a single agent's preference might not be normal, but the average agent's preference is surely asymptotically normal! (as the number of agents tend towards infinity) [edit: of course not everything is asymptotically normal. a sequence of Cauchys do not tend towards the normal distribution.]
catnaroek 11y ago
The normal distribution is important because it arises naturally when the preconditions of the central limit theorem hold. But you still have to use your brain - you can't unquestioningly assume that any random variable (or sample or whatever) you will stumble upon will be approximately normally distributed.
- lordnacho 11y ago
  This is the right answer. Quite a lot of things are not normally distributed. Often this is because there's more than one generating mechanism. For instance you may have a financial market where normally things go up and down smoothly, people are reacting to news as normal. But then there are also panics, where people are not behaving normally. There's also cases where the past influences the future, and drawings are not identically and independently distributed (though Wikipedia tells me there's weak variants of the CLT).
  - FabHK 11y ago
    The conditions under which the Central Limit Theorem hold are complicated (in particular, weaker and weaker conditions have been shown to be sufficient even fairly recently). But intuitively, they boil down to two conditions that are fairly sensible: 1. The individual observations can't be too extreme. Bounded variance, for example, is sufficient, but weaker conditions suffice. 2. The observations must be somewhat independent. Independence is a strong condition and sufficient, but again weaker conditions suffice. If 1) doesn't hold, a massive outliers could swamp all other observations. If 2) doesn't hold, for example if one underlying factor influences all observations, all bets are off: the observations could "conspire" to be fairly large (or fairly small), thus giving rise to fat tails inconsistent with the normal distribution. (Having IID (independent identically distributed) observations, btw, is not sufficient, if the distribution itself is too fat tailed.)
    - catnaroek 11y ago
      Nice. I was aware that you could weaken independence, but not bounded variance. Without bounded variance, I thought you would get something like Student's t.
afsina 11y ago
Author tries to find some anecdotes that would verify his claims, without actual data of course. I would like to know for example, what is the actual distribution of money people paying for a cellphone (without contract) in different countries? Or the distribution of topics that people are looking in Facebook and elsewhere in Internet. Answer would probably not be a Gaussian but quite different than what author claims.
metafunctor 11y ago
This article may look like science, but it's actually just an opinion. Don't be fooled by the graphs.
- deleted 11y ago
  [deleted]
sageikosa 11y ago
I was really hoping for something about how marketing identifies consumer market segments and custom tailors product offerings and prices to make the best margin they think they can get. Even so, I'd guess that for commoditized products, there would still be a normalizable distribution of prices offered within the confines of a targeted market segment.
baldfat 11y ago
Data is wrong on examples. Mobile Phone = "Good Enough Android" and "iOS and Flagship Android" http://www.gartner.com/newsroom/id/2996817 http://www.gartner.com/newsroom/id/2996817 Apple had 15% of Worldwide phone sales in 2014. Samsung had 24% which is 39% for both and the sales and demand are fairly similar to what people are willing to pay. He had data he just didn't want to use.
blfr 11y ago
This could be a high-end news source like the New York Times; it could also be a small, niche site like Stratechery. He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end. There's still a normal distribution with relatively little demand for vile content (shock sites), more for gossip sites, a lot for Buzzfeed and other click bait factories, including what fills many Facebook feeds, then less for NYT and the like, finally very little for niche, truly high quality sources (blogs run by hobbyists, hackers, professionals, academics).
- coldtea 11y ago
  >He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end. That's beside the point (not to mention that when it comes to perceived quality, the vast majority of people would call NYT high quality, even if they don't read it). The point is most will either go to few specific sources or to a few generic aggregators, instead of breaking their surfing across a large distribution of sites (as people did more in mid-nineties).
coldtea 11y ago
The author is making an observation based on market knowledge and long term trends. Yes, he's not mouth-feeding us with the relevant data points -- we can search for them ourselves and see whether they match his observations or not. Once we get over this, in the same way that we don't expect any conversation or comment in real life to come with charts and figures, we can discuss and try to verify where his assumptions/deductions are correct and whether they match the data that we have access to. Being unsubstantiated is a different thing from being wrong.
ableal 11y ago
If anyone else is curious about the second picture, I tineyed ( http://tineye.com/search/863a041e2607e3405d2dcec1fb148bce4713e8f5/ http://tineye.com/search/863a041e2607e3405d2dcec1fb148bce471... ;-) it and found two agreeing attributions. One from a 2010 piece by Terry Tao: https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-a-non-technical-article-on-universality/ https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-... ' Fig 23. A "living histogram" of 143 students at the University of Connecticut, with females in white and males in black. (Source: The Hartford Courant, “Reaching New Heights,” November 23, 1996) Compare this non-bell-shaped distribution with Figure 5. ' And another also mentioning U. Connecticut and crediting the photographers in the image itself: http://image.slidesharecdn.com/visualiationofquantitativeinformation-110321181118-phpapp01/95/visualiation-of-quantitative-information-39-728.jpg?cb=1300731563 http://image.slidesharecdn.com/visualiationofquantitativeinf...

23 points by Futurebot 11y ago

This looks bullshit to me. He says the normal distribution is vanishing, and to prove himself, he brings up some charts which do not look like a normal distribution. Like the first smartphone-related chart: "You’re looking to buy a new smartphone. What does your demand distribution look like? Is it normally distributed? Absolutely not. It looks like this:" [ non-Gaussian 'curve' ] Okay, what is a 'demand distribution'? I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc.

pzone 11y ago
Pretty sure author is just making stuff up. There's no content to this at all. One has no ground to make such empirical claims without showing any data backing it up.
LoSboccacc 11y ago
whole article comes only with 'hunches' and not a single data point researched. it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones. one could argue that those are sold in lower volumes, but then that'd be a gaussian distribution also, of course economics stats are not a good fit for a gaussian. they tend to stabilize around exponential. that's why the price pyramid is a pyramid. I have to concede that by looking at some data, it seems there is a exponential distribution going on for all price ranges except at the highest end, but that's more a hint of the product price/features space being discontinuous than we being in a post scarcity society where gaussian/exponential distributions suddenly vanish. but anyway, the fact remains that the data is out there, even for the other field, this on smartphone is but one of the many he didn't research preferring to follow intuition. and human brain is very bad at applying intuition to statistics. http://image.slidesharecdn.com/tomiahonen-engagewithmobile-wsa-abudhabi2015-150220083444-conversion-gate01/95/tomi-ahonen-engage-with-mobile-at-wsamobile-global-congress-abu-dhabi-38-638.jpg?cb=1424442956 http://image.slidesharecdn.com/tomiahonen-engagewithmobile-w... http://4.bp.blogspot.com/-uY7kRAl6XfA/URibddfrTqI/AAAAAAAAPLI/pnN9ivxjtt0/s1600/Smartphone+Sales+Distibution+by+ASP.png http://4.bp.blogspot.com/-uY7kRAl6XfA/URibddfrTqI/AAAAAAAAPL...
- coldtea 11y ago
  >it conveniently ignores many facts and makes up some, like Androids being the cheaper, low quality choice, ignoring there are devices sold for as much as iPhones. He absolutely does not ignore that fact, in fact he specifically says that the better end of the market is either iOS or high-end Android phones. Obviously (and all market data point to that), the Android phones that sell more are not the high end units, but the cheapo units that dominate the "free with subscription", "prepaid" etc market.
ekianjo 11y ago
> This looks bullshit to me It's obviously bullshit since the guy analyses ZERO data and says the normality is dead. Obligatory facepalm. And obviously Android smartphones pricing look NOTHING like what he has drawn, there are cheap as well as high end Android devices out there at about every price point.
omginternets 11y ago
This is what's commonly -- and pejoratively -- referred to in the sciences as "logic without evidence".
- commentzorro 11y ago
  > "logic without evidence" When I was in school we called these people "philosophy majors." ;-)
  - omginternets 11y ago
    I know you're saying this in jest, but a good philosopher will account for empirical knowledge in his arguments, when it exists. I know it's cool to rip on philosophers, but they're incredibly useful to the sciences: they're usually the first ones to point out circular (or otherwise fallacious) reasoning.
coldtea 11y ago
>I am pretty sure the market demand for smartphones does not look like the graph he's showing. Etc. etc. And I'm pretty sure it does look like that, judging from all the people I know (most either have a crappy whatever smartphone or a high end iOS/Android model, with the middle ground, which is the main bulk of a normal distribution under-represented). It also matches all information I've seen about the mobile market, user behavior (e.g. how iOS dominates profits and mobile views while still having a much smaller userbase, etc).

ddebernardy 11y ago

> The world’s shift from Normal, Gaussian distributions of demand towards bifurcated, two-tiered distributions is a natural consequence of our shift from a world governed by scarcity to one governed by abundance. FWIW, others (e.g. Steve Keen, of "Debunking Economics" fame) would convincingly argue that the world never fit into economist make believe normal distributions to begin with.

pjc50 11y ago

This article hints at the inequality implications, but I think they're likely to become increasingly important. We're already seeing this with creative works: the vast majority of works earn zero or near-zero, apart from a few massive global hits. Inequality obviously also affects people's ability and willingness to pay for things. The lower end products are not just cheap but increasingly "free": either ad-surveillance-supported or paid for by the exploitative gambling-like "free to play" systems. Again, it's well known in F2P that most of the revenue comes from a tiny fraction of users.

princeb 11y ago

a single agent's preference might not be normal, but the average agent's preference is surely asymptotically normal! (as the number of agents tend towards infinity) [edit: of course not everything is asymptotically normal. a sequence of Cauchys do not tend towards the normal distribution.]

catnaroek 11y ago

The normal distribution is important because it arises naturally when the preconditions of the central limit theorem hold. But you still have to use your brain - you can't unquestioningly assume that any random variable (or sample or whatever) you will stumble upon will be approximately normally distributed.

lordnacho 11y ago
This is the right answer. Quite a lot of things are not normally distributed. Often this is because there's more than one generating mechanism. For instance you may have a financial market where normally things go up and down smoothly, people are reacting to news as normal. But then there are also panics, where people are not behaving normally. There's also cases where the past influences the future, and drawings are not identically and independently distributed (though Wikipedia tells me there's weak variants of the CLT).
- FabHK 11y ago
  The conditions under which the Central Limit Theorem hold are complicated (in particular, weaker and weaker conditions have been shown to be sufficient even fairly recently). But intuitively, they boil down to two conditions that are fairly sensible: 1. The individual observations can't be too extreme. Bounded variance, for example, is sufficient, but weaker conditions suffice. 2. The observations must be somewhat independent. Independence is a strong condition and sufficient, but again weaker conditions suffice. If 1) doesn't hold, a massive outliers could swamp all other observations. If 2) doesn't hold, for example if one underlying factor influences all observations, all bets are off: the observations could "conspire" to be fairly large (or fairly small), thus giving rise to fat tails inconsistent with the normal distribution. (Having IID (independent identically distributed) observations, btw, is not sufficient, if the distribution itself is too fat tailed.)
  - catnaroek 11y ago
    Nice. I was aware that you could weaken independence, but not bounded variance. Without bounded variance, I thought you would get something like Student's t.

afsina 11y ago

Author tries to find some anecdotes that would verify his claims, without actual data of course. I would like to know for example, what is the actual distribution of money people paying for a cellphone (without contract) in different countries? Or the distribution of topics that people are looking in Facebook and elsewhere in Internet. Answer would probably not be a Gaussian but quite different than what author claims.

metafunctor 11y ago

This article may look like science, but it's actually just an opinion. Don't be fooled by the graphs.

deleted 11y ago
[deleted]

sageikosa 11y ago

I was really hoping for something about how marketing identifies consumer market segments and custom tailors product offerings and prices to make the best margin they think they can get. Even so, I'd guess that for commoditized products, there would still be a normalizable distribution of prices offered within the confines of a targeted market segment.

baldfat 11y ago

Data is wrong on examples. Mobile Phone = "Good Enough Android" and "iOS and Flagship Android" http://www.gartner.com/newsroom/id/2996817 http://www.gartner.com/newsroom/id/2996817 Apple had 15% of Worldwide phone sales in 2014. Samsung had 24% which is 39% for both and the sales and demand are fairly similar to what people are willing to pay. He had data he just didn't want to use.

blfr 11y ago

This could be a high-end news source like the New York Times; it could also be a small, niche site like Stratechery. He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end. There's still a normal distribution with relatively little demand for vile content (shock sites), more for gossip sites, a lot for Buzzfeed and other click bait factories, including what fills many Facebook feeds, then less for NYT and the like, finally very little for niche, truly high quality sources (blogs run by hobbyists, hackers, professionals, academics).

coldtea 11y ago
>He's massaging the examples here. Just because NYT is somewhat above the sewers of the Internet doesn't make it high-end. That's beside the point (not to mention that when it comes to perceived quality, the vast majority of people would call NYT high quality, even if they don't read it). The point is most will either go to few specific sources or to a few generic aggregators, instead of breaking their surfing across a large distribution of sites (as people did more in mid-nineties).

coldtea 11y ago

The author is making an observation based on market knowledge and long term trends. Yes, he's not mouth-feeding us with the relevant data points -- we can search for them ourselves and see whether they match his observations or not. Once we get over this, in the same way that we don't expect any conversation or comment in real life to come with charts and figures, we can discuss and try to verify where his assumptions/deductions are correct and whether they match the data that we have access to. Being unsubstantiated is a different thing from being wrong.

ableal 11y ago

If anyone else is curious about the second picture, I tineyed ( http://tineye.com/search/863a041e2607e3405d2dcec1fb148bce4713e8f5/ http://tineye.com/search/863a041e2607e3405d2dcec1fb148bce471... ;-) it and found two agreeing attributions. One from a 2010 piece by Terry Tao: https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-a-non-technical-article-on-universality/ https://terrytao.wordpress.com/2010/09/14/a-second-draft-of-... ' Fig 23. A "living histogram" of 143 students at the University of Connecticut, with females in white and males in black. (Source: The Hartford Courant, “Reaching New Heights,” November 23, 1996) Compare this non-bell-shaped distribution with Figure 5. ' And another also mentioning U. Connecticut and crediting the photographers in the image itself: http://image.slidesharecdn.com/visualiationofquantitativeinformation-110321181118-phpapp01/95/visualiation-of-quantitative-information-39-728.jpg?cb=1300731563 http://image.slidesharecdn.com/visualiationofquantitativeinf...