3 ms·

I sometimes think that progress in the 21st century will be summed up as: "The realization that the normal distribution is not the only way to model data". Tal

I sometimes think that progress in the 21st century will be summed up as: "The realization that the normal distribution is not the only way to model data". Taleb's favorite topic is the "black swan event" which is something that the normal distribution, and the idea of standard deviation, don't model that well. In a normal distribution very extreme events should only happen once in the lifetime of several universes. Of course assuming variation inline with a Gaussian process is at the heart of how the Black-Sholes model calculates risk/volatility/etc. Benoit Mandelbrot argued that financial markets follow a distribution much more similar to the Cauchy distribution (specifically the Levy distribution) rather than a Gaussian. The problem of course is that the Cauchy distribution is pathological in that it doesn't have a mean or variance, you can calculate similar properties for it (location and scale), but it doesn't obey the central limit theorem so in practice it can be very strange to work with. The normal distribution is fantastic in that it does appear frequently in nature, is very well behaved, and has been extensively studied. However a great amount of future progress is going to come from wrestling with more challenging distributions, and paying more attention to when assumptions of normality need to be questioned. Of course one of the challenges of this is that the normal distribution is baked into a very large number of our existing statistical tools.

jerf 13y ago
This is actually what I expected to read: "The standard deviation is useful because with the average and the standard deviation, one can fully characterize a normal distribution. However, the standard deviation is less useful a statistical summary the farther away from 'normal' you get, and in reality, there is no such thing as a normal distribution, as a true normal distribution is defined on the entire real number line from negative infinity to positive infinity. Reality always provides some bound, and it's often quite distorted from Guassian. For instance, a 'normal' distribution averaging 2 with a standard deviation of 1.4, bounded by 0, is quite non-Gaussian in many important ways! (Not least of which is that you're going to have to do something to replace the missing probability...) "People rarely check how closely their data conform to the standard distribution; indeed, many people blindly apply the standard deviation to their data regardless of its distribution! The resulting number is often more obfuscatory than helpful, to the extent that it crowded out more useful summaries. "It's a useful metric when treated carefully, but it is rare to encounter it treated carefully. Science courses would be well-served to stop teaching it in favor of a stronger emphasis on multiple distributions. (Multiple distributions are usually touched upon, but implicitly our curricula overfavor the Gaussian distribution and end up accidentally implicitly convincing students its the only one.)" But that's just me.
- Helianthus 13y ago
  >Reality always provides some bound But... it doesn't. You ever hear about the hypothetical possibility of your atoms lining up and falling through the floor? It's hypothetical in the sense that it's really ridiculously unlikely, but there is no bound preventing it. Now the central point about different probability curves stands, but that's not what Taleb was talking about--he seems to think that it's the tool's fault if people are using it wrong--and it's also not what Homunculiheaded argued.
  - jerf 13y ago
    "But... it doesn't. You ever hear about the hypothetical possibility of your atoms lining up and falling through the floor?" A bad example; that's a very, very large sample space, such that deviations from mathematical perfection are irrelevant. They do exist, if you're precise enough (for instance, the universe is not modeled by perfectly continuous space), but I'm not inclined to argue them, because it's too easy to argue that they're irrelevant. So instead consider something more human-sized: Match a normal distribution to the height of human beings. It works very well, except in real life, the probability of a negative-height human being is zero. This is not what the Gaussian model predicts. Unfortunately, rather more science takes place in the second domain than the first. "that's not what Taleb was talking about" I'm quite aware. The fact that I commented on how I got something other than what I expected rather suggested that, I thought... The fact that this isn't precisely what Homunculiheaded said is also why I posted, rather than just upvoting....
    - Helianthus 13y ago
      >The fact that this isn't precisely what Homunculiheaded said is also why I posted Ah. I misread the following... >>This is actually what I expected to read: as agreement ("This is actually what I expected to read."). My mistake. >Unfortunately, rather more science takes place in the second domain than the first. As I said to Homunculiheaded, this is because of the relative utility of the models, which we understand--and even those that do not understand it do not make the tool's use invalid. What are we bemoaning, here, but actual misunderstanding itself? And really, what's the point of that?
Helianthus 13y ago
>"The realization that the normal distribution is not the only way to model data". Realization by who? If you understand the normal distribution you had damn well better know that there are other probability distributions. >The problem of course is that the Cauchy distribution is pathological in that it doesn't have a mean or variance, you can calculate similar properties for it (location and scale), but it doesn't obey the central limit theorem so in practice it can be very strange to work with. In other words, we're using the normal distribution as the workhorse because considering other distributions is, well, inefficient/unproductive. > However a great amount of future progress is going to come from wrestling with more challenging distributions, and paying more attention to when assumptions of normality need to be questioned What exactly is it that you think physicists have been doing for the past half century? The error accounting for CERN's experiments requires actual millions of PhD-hours. This topic's conversation is at some bizarre intersection of good intentions, concrete knowledge, and woeful ignorance. I guess I tar myself with that brush.
- jerf 13y ago
  "Realization by who? If you understand the normal distribution you had damn well better know that there are other probability distributions." See Glyptodon's post: https://news.ycombinator.com/item?id=7065067 https://news.ycombinator.com/item?id=7065067 If you hang out with mathematicians, yeah, sure, everybody knows there's a ton of distributions. Try hanging out with, say, biologists. The undergrad statistics education is basically "mumble mumble guassian mumble hideous equations mumble mumble YOU MUST DO THE CHI-SQUARED TEST mumble mumble poisson mumble hideous equations mumble YOU MUST DO THE CHI-SQUARED TEST mumble mumble CHI mumble calculus is hard mumble mumble NULL HYPOTHESIS mumble CHI CHI CHI WE WILL DRUM YOU OUT OF THIS DISCIPLINE IN DISGRACE IF YOU DON'T DO THE CHI-SQUARED TEST". I'm hardly even exaggerating! I remember being asked by someone in their third semester of using the damn test what it actually meant. Nominally, yes, the Poisson and probably a couple of others were mentioned, but believe me, the ALL CAPS part of the education does not mention them.
  - Helianthus 13y ago
    "A general science education needs far firmer statistical grounding" doesn't equate to "The standard deviation should be retired."

jerf 13y ago

This is actually what I expected to read: "The standard deviation is useful because with the average and the standard deviation, one can fully characterize a normal distribution. However, the standard deviation is less useful a statistical summary the farther away from 'normal' you get, and in reality, there is no such thing as a normal distribution, as a true normal distribution is defined on the entire real number line from negative infinity to positive infinity. Reality always provides some bound, and it's often quite distorted from Guassian. For instance, a 'normal' distribution averaging 2 with a standard deviation of 1.4, bounded by 0, is quite non-Gaussian in many important ways! (Not least of which is that you're going to have to do something to replace the missing probability...) "People rarely check how closely their data conform to the standard distribution; indeed, many people blindly apply the standard deviation to their data regardless of its distribution! The resulting number is often more obfuscatory than helpful, to the extent that it crowded out more useful summaries. "It's a useful metric when treated carefully, but it is rare to encounter it treated carefully. Science courses would be well-served to stop teaching it in favor of a stronger emphasis on multiple distributions. (Multiple distributions are usually touched upon, but implicitly our curricula overfavor the Gaussian distribution and end up accidentally implicitly convincing students its the only one.)" But that's just me.

Helianthus 13y ago
>Reality always provides some bound But... it doesn't. You ever hear about the hypothetical possibility of your atoms lining up and falling through the floor? It's hypothetical in the sense that it's really ridiculously unlikely, but there is no bound preventing it. Now the central point about different probability curves stands, but that's not what Taleb was talking about--he seems to think that it's the tool's fault if people are using it wrong--and it's also not what Homunculiheaded argued.
- jerf 13y ago
  "But... it doesn't. You ever hear about the hypothetical possibility of your atoms lining up and falling through the floor?" A bad example; that's a very, very large sample space, such that deviations from mathematical perfection are irrelevant. They do exist, if you're precise enough (for instance, the universe is not modeled by perfectly continuous space), but I'm not inclined to argue them, because it's too easy to argue that they're irrelevant. So instead consider something more human-sized: Match a normal distribution to the height of human beings. It works very well, except in real life, the probability of a negative-height human being is zero. This is not what the Gaussian model predicts. Unfortunately, rather more science takes place in the second domain than the first. "that's not what Taleb was talking about" I'm quite aware. The fact that I commented on how I got something other than what I expected rather suggested that, I thought... The fact that this isn't precisely what Homunculiheaded said is also why I posted, rather than just upvoting....
  - Helianthus 13y ago
    >The fact that this isn't precisely what Homunculiheaded said is also why I posted Ah. I misread the following... >>This is actually what I expected to read: as agreement ("This is actually what I expected to read."). My mistake. >Unfortunately, rather more science takes place in the second domain than the first. As I said to Homunculiheaded, this is because of the relative utility of the models, which we understand--and even those that do not understand it do not make the tool's use invalid. What are we bemoaning, here, but actual misunderstanding itself? And really, what's the point of that?

Helianthus 13y ago

>"The realization that the normal distribution is not the only way to model data". Realization by who? If you understand the normal distribution you had damn well better know that there are other probability distributions. >The problem of course is that the Cauchy distribution is pathological in that it doesn't have a mean or variance, you can calculate similar properties for it (location and scale), but it doesn't obey the central limit theorem so in practice it can be very strange to work with. In other words, we're using the normal distribution as the workhorse because considering other distributions is, well, inefficient/unproductive. > However a great amount of future progress is going to come from wrestling with more challenging distributions, and paying more attention to when assumptions of normality need to be questioned What exactly is it that you think physicists have been doing for the past half century? The error accounting for CERN's experiments requires actual millions of PhD-hours. This topic's conversation is at some bizarre intersection of good intentions, concrete knowledge, and woeful ignorance. I guess I tar myself with that brush.

jerf 13y ago
"Realization by who? If you understand the normal distribution you had damn well better know that there are other probability distributions." See Glyptodon's post: https://news.ycombinator.com/item?id=7065067 https://news.ycombinator.com/item?id=7065067 If you hang out with mathematicians, yeah, sure, everybody knows there's a ton of distributions. Try hanging out with, say, biologists. The undergrad statistics education is basically "mumble mumble guassian mumble hideous equations mumble mumble YOU MUST DO THE CHI-SQUARED TEST mumble mumble poisson mumble hideous equations mumble YOU MUST DO THE CHI-SQUARED TEST mumble mumble CHI mumble calculus is hard mumble mumble NULL HYPOTHESIS mumble CHI CHI CHI WE WILL DRUM YOU OUT OF THIS DISCIPLINE IN DISGRACE IF YOU DON'T DO THE CHI-SQUARED TEST". I'm hardly even exaggerating! I remember being asked by someone in their third semester of using the damn test what it actually meant. Nominally, yes, the Poisson and probably a couple of others were mentioned, but believe me, the ALL CAPS part of the education does not mention them.
- Helianthus 13y ago
  "A general science education needs far firmer statistical grounding" doesn't equate to "The standard deviation should be retired."