3 ms·

There was a similar blog post a while back about the standard deviation in statistics. About how when the SD was discovered, the original paper was full of dis

by keenerd 12y ago

There was a similar blog post a while back about the standard deviation in statistics. About how when the SD was discovered, the original paper was full of disclaimers that the SD should only be used by statisticians working with perfect distributions. The squaring would amplify the natural outliers present in real-world data and make the SD too dangerous to trust. And yet we use the SD for everything, even though the original reason for it (a computation shortcut to save hand-labor) is pretty much meaningless. I have a copy of the original paper but the paper lacks the historical perspective of the post. I've never been able to find that blog post again. Does anyone remember what I'm talking about?

rdtsc 12y ago
I kind of remember SD being used because it has the same units as the samples in the distribution.
- keenerd 12y ago
  Nope. The alternative (mean average distribution) also has the same units. The problem with the MAD was that it used an absolute value and so introduced a discontinuity. If you are a mathematician working with a system of equations, each discontinuity doubles the size of the system when taking integrals/derivatives. The SD was a clever hack to hide the discontinuity. It makes the mathematician's job of juggling equations much easier by changing the workload from O(n^2) to O(n). If you are not doing calculus on the system then the SD has much less utility. Negative utility even, because the square/square-root hack used by the SD is identical to the MAD only for perfectly normal distributions. For real-world distributions it tends to give wrong results.
- deleted 12y ago
  [deleted]
Sniffnoy 12y ago
As I understand it there are good reasons the SD is often more useful. For instance, if you are dealing with a large number of samples from your distribution, and want to apply the central limit theorem, it's the SD, not the MAD, that you need to know.
alecco 12y ago
Probably not what you refered to but close enough? http://www.leeds.ac.uk/educol/documents/00003759.htm http://www.leeds.ac.uk/educol/documents/00003759.htm http://stats.stackexchange.com/a/1516 http://stats.stackexchange.com/a/1516 And Nassim Nicholas Taleb Standard Deviation The notion of standard deviation has confused hordes of scientists; it is time to retire it from common use and replace it with the more effective one of mean deviation. https://web.archive.org/web/20140115171745/http://www.edge.org/response-detail/25401 https://web.archive.org/web/20140115171745/http://www.edge.o...
- keenerd 12y ago
  The Gorard one looks very familiar and is old enough too. Going to go with memory rot changing my other details. Thank you!

by keenerd 12y ago

rdtsc 12y ago

I kind of remember SD being used because it has the same units as the samples in the distribution.

keenerd 12y ago
Nope. The alternative (mean average distribution) also has the same units. The problem with the MAD was that it used an absolute value and so introduced a discontinuity. If you are a mathematician working with a system of equations, each discontinuity doubles the size of the system when taking integrals/derivatives. The SD was a clever hack to hide the discontinuity. It makes the mathematician's job of juggling equations much easier by changing the workload from O(n^2) to O(n). If you are not doing calculus on the system then the SD has much less utility. Negative utility even, because the square/square-root hack used by the SD is identical to the MAD only for perfectly normal distributions. For real-world distributions it tends to give wrong results.
deleted 12y ago
[deleted]

Sniffnoy 12y ago

As I understand it there are good reasons the SD is often more useful. For instance, if you are dealing with a large number of samples from your distribution, and want to apply the central limit theorem, it's the SD, not the MAD, that you need to know.

alecco 12y ago

Probably not what you refered to but close enough? http://www.leeds.ac.uk/educol/documents/00003759.htm http://www.leeds.ac.uk/educol/documents/00003759.htm http://stats.stackexchange.com/a/1516 http://stats.stackexchange.com/a/1516 And Nassim Nicholas Taleb Standard Deviation The notion of standard deviation has confused hordes of scientists; it is time to retire it from common use and replace it with the more effective one of mean deviation. https://web.archive.org/web/20140115171745/http://www.edge.org/response-detail/25401 https://web.archive.org/web/20140115171745/http://www.edge.o...

keenerd 12y ago
The Gorard one looks very familiar and is old enough too. Going to go with memory rot changing my other details. Thank you!