5 ms·

Okay, we have deviations from the mean. They are (0.1, 0.1, -0.1, -0.1). Or we have deviations from the mean (0.2, 0, 0, -0.2). I think is is pretty intuitive t

by cjfd 7y ago

Okay, we have deviations from the mean. They are (0.1, 0.1, -0.1, -0.1). Or we have deviations from the mean (0.2, 0, 0, -0.2). I think is is pretty intuitive that in the second case the deviation is larger. In the first case there are only two differing values that are 0.2 apart. In the second case there are three differing values and some are 0.2 apart and others are 0.4 apart. Also, in the first case there are two pairs of equal value and in the second case there is only one pair that is of equal value. The squaring and rooting does this for you. Of course, other functions could do a similar thing for you. The squaring and rooting has nice mathematical properties especially if you consider the variance, i.e., omitting the square root. That is much more differentiable than an absolute value. Also, consider that the correlation coefficient consists of similar formulas. They all look like the theorem of pythagoras. This ties in nicely with the fact that observables can be seen as having angles in between them where the correlation coefficient is the cosine of these angles.

chrisseaton 7y ago
I still don’t understand why we pick a metric because it’s easy to work with. Isn’t it either the right metric or not?
- deleted 7y ago
  [deleted]
- username90 7y ago
  The standard deviation is not arbitrary, it tells us what the final normal distribution would look like if you averaged a lot of independent distributions. You can read up on the central limit theorem if you want to understand more. https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Central_limit_theorem
  - chrisseaton 7y ago
    I don't want to keep going on because it sounds like I'm trolling, but when you tell me 'it tells us what the final normal distribution would look like if you averaged a lot of independent distributions' I just want to say 'why do I care about that? where did that requirement come from?'
    - username90 7y ago
      Economics of stocks is a good example, when you put a lot of stocks in one portfolio you can calculate its risk (standard deviation) and returns (average value), and see how much each piece of stock contributed to that. This is because the results of many variables looks like a normal distribution with the combined average and standard deviation. Now it gets a bit more complicated in the real world since stocks are not independent from each other, but it shouldn't be hard to understand why knowing the standard deviation of something is useful.
      - chrisseaton 7y ago
        > Now it gets a bit more complicated in the real world since stocks are not independent from each other This is also where I get frustrated. I'm taught about the CLT but every place I think I might use it, the samples aren't really IDD when you look hard enough, so it doesn't apply. Computer benchmarking (which is where I'm usually trying to apply statistics) is the classic example. One iteration changes the state of the computer for the next iteration. They aren't remotely IDD, but people still try to claim the CLT. Seems like all their stats beyond that point is broken.
        username90 7y ago
        Right, that is how statistics usually is done. Note however that you don't need IDD, independence is enough. So you assume independence, from that CLT follows which means that standard deviation is all that matters. Then the real world deviates from this based on how much your distributions fails to be independent. I'd assume that computer benchmarks have very low correlation between runs, unless you leak ram or your components don't keep a stable temperature. So in that case the CLT seems to apply just fine.
        chrisseaton 7y ago
        I can't understand this. Surely either samples are independent, or they are not? And they are not in a computer program. The state of, to just give one example, the processor cache depends on what happened in the previous run. We have all this rigour in how statistical techniques are applied, and we make black or white decisions based on the numbers that come out of them, but at their very foundation we're saying 'whatever I'll just pretend this rule applies it's probably fine'. How can we build on that flawed foundation? I don't get it!
  - jampekka 7y ago
    Normal distribution can be parameterized with almost any measures of location and dispersion. The reason why standard deviation (which is just variance in hiding) is used is because it makes the equations (that non-statisticians practically never need to look at) a bit nicer given the other (mostly arbitrary) conventions used in the maths. Put simply, standard deviation is arbitrary for most people.
- darkerside 7y ago
  I think you're right that there is an arbitrariness to this algorithm, and there are other ways of normalizing your data. And you could probably build entire new branches of statistics off of them (like non Euclidean geometry), and they'd probably even be better in some way. But, you'd be speaking a different language from everybody else. Speaking of language, have you ever thought about how all names and grammatical structures are arbitrary as well? There are parallels.
  - username90 7y ago
    No, standard deviation is special. If you have a large enough amount of independent distributions which gets averaged or added in some way, then the only thing which matters are their standard deviations and average values for what the final distribution looks like.
    - darkerside 7y ago
      I'm well out of my wheelhouse, so anyone else chiming in would be appreciated. I'm not sure what you mean by "looks like", but I would imagine other "nonstandard" measures of deviation would also result in a consistent model. My larger point is that, something can be arbitrary and still be meaningful. If people agree on a common meaning, it formalizes a communication protocol that allows for much better information sharing.
      - username90 7y ago
        "Looks like" as in the difference approaches zero as the number of times approaches infinity. This means that any distribution with standard deviation of X will behave as a normal distribution with standard deviation X when you have enough of them. Or in other words, any distribution with standard deviation of X will behave like any other distribution with standard deviation of X when you have enough of them. There is no other measure like this as there is no other equivalent to the central limit theorem. Therefore it makes sense to have this as a universal measure of random processes.
- roenxi 7y ago
  Start with the Normal or Multivariate normal. We start with this because it is the distribution that emerges naturally when we have lots of little errors adding up around a basically consistent effect (see also [0]). So basically any situation where something ought to happen but 101 things could also happen to cause small changes will turn out to be normally distributed. An exceedingly common occurrence + one that almost inevitably turns up when dealing with a practical error prone process. Now that we have a normal distribution, the mean is an obvious metric to pick because it captures the notion of 'middle' in a useful sense. Then we have proofs that we can characterise the normal with the mean and 1 other parameter (normal normal) or a matrix (multivariate normal). We call that the standard deviation^. The exact formula wasn't a coincidence. The normal can obviously be characterised by the mean and a statistic from another formula. There were a bunch of experiments tried (eg, using |x| instead of sqrt[x^2]) but it turned out that sqrt[x^2] had some other nice property that minimised some sort of error so they went with it as a standard. I forget what one, might be error of estimating the true parameters from a sample or similar. We could characterise the the normal as an infinite sum or something quirky, but when people say 'easy to work with' they mean instead of a function or something quirky we can simply pick a number. Standard Deviation isn't as important when working with non-normal distributions, although I think it still turns out to be useful. But its importance is that it characterises a normal apart from the information captured in the mean. I'm not a mathematician, YMMV, could be wrong, standard disclaimers. [0] https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Central_limit_theorem ^ I'm not going to edit this but it occurs to me that we call it the Variance. Same thing as std. dev in my opinion. ' There is a fairly subtle observation to make - Normal is characterised by mean and std. dev, but the most efficient ( https://en.wikipedia.org/wiki/Efficiency_(statistics) https://en.wikipedia.org/wiki/Efficiency_(statistics) ) unbiased estimator of the std. dev of the population is the adjusted std. dev of the sample. Therefore, accounting for mean, you can't get a more efficient characterisation of the normal than mean & std. dev. Ie, if you picked a formula other than std. dev then the most efficient estimators to characterise it would still be mean and std. dev. Don't recall if there are equally efficient choices but I think that proves there are none better. That might have been the logic for why std. dev was chosen. Just a guess.
- marcosdumay 7y ago
  The mathematical representations and operations people see at school were picked because they are easy to work on most common problems. There is not this "correct math" thing, you pick what works best.
- BeetleB 7y ago
  > Isn’t it either the right metric or not? There are many "right" metrics - this is one of them. The big picture is we want a measure/idea of how spread out the data is. One can come up with many ways to do this, and absolute value and root mean square are two of the most common. They correspond to L0 and L2 norms. There is also the L-infinity norm (just use the farthest point). The L2 norm is the most complicated of the 3, but as people pointed out, it is one of the easiest to use analytically because of its nice mathematical properties. Have a look at https://en.wikipedia.org/wiki/Statistical_dispersion https://en.wikipedia.org/wiki/Statistical_dispersion for other measures.
- dboreham 7y ago
  >Isn’t it either the right metric or not? To illustrate, perhaps an example closer to home: If someone asked you to tell them "how many lines of code in this project?" you'd start wondering things like "do I count comments?", "do I count dependent libraries?", "perhaps some metric that is equivalent of a statement count but not strictly counting lines?" and on and on. Statistical measures are like that. Someone with a decent knowledge of the problem space came up with the best stab at how to characterize the data.

by cjfd 7y ago

chrisseaton 7y ago

I still don’t understand why we pick a metric because it’s easy to work with. Isn’t it either the right metric or not?

deleted 7y ago
[deleted]
username90 7y ago
The standard deviation is not arbitrary, it tells us what the final normal distribution would look like if you averaged a lot of independent distributions. You can read up on the central limit theorem if you want to understand more. https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Central_limit_theorem
- chrisseaton 7y ago
  I don't want to keep going on because it sounds like I'm trolling, but when you tell me 'it tells us what the final normal distribution would look like if you averaged a lot of independent distributions' I just want to say 'why do I care about that? where did that requirement come from?'
  - username90 7y ago
    Economics of stocks is a good example, when you put a lot of stocks in one portfolio you can calculate its risk (standard deviation) and returns (average value), and see how much each piece of stock contributed to that. This is because the results of many variables looks like a normal distribution with the combined average and standard deviation. Now it gets a bit more complicated in the real world since stocks are not independent from each other, but it shouldn't be hard to understand why knowing the standard deviation of something is useful.
    - chrisseaton 7y ago
      > Now it gets a bit more complicated in the real world since stocks are not independent from each other This is also where I get frustrated. I'm taught about the CLT but every place I think I might use it, the samples aren't really IDD when you look hard enough, so it doesn't apply. Computer benchmarking (which is where I'm usually trying to apply statistics) is the classic example. One iteration changes the state of the computer for the next iteration. They aren't remotely IDD, but people still try to claim the CLT. Seems like all their stats beyond that point is broken.
      - username90 7y ago
        Right, that is how statistics usually is done. Note however that you don't need IDD, independence is enough. So you assume independence, from that CLT follows which means that standard deviation is all that matters. Then the real world deviates from this based on how much your distributions fails to be independent. I'd assume that computer benchmarks have very low correlation between runs, unless you leak ram or your components don't keep a stable temperature. So in that case the CLT seems to apply just fine.
        chrisseaton 7y ago
        I can't understand this. Surely either samples are independent, or they are not? And they are not in a computer program. The state of, to just give one example, the processor cache depends on what happened in the previous run. We have all this rigour in how statistical techniques are applied, and we make black or white decisions based on the numbers that come out of them, but at their very foundation we're saying 'whatever I'll just pretend this rule applies it's probably fine'. How can we build on that flawed foundation? I don't get it!
- jampekka 7y ago
  Normal distribution can be parameterized with almost any measures of location and dispersion. The reason why standard deviation (which is just variance in hiding) is used is because it makes the equations (that non-statisticians practically never need to look at) a bit nicer given the other (mostly arbitrary) conventions used in the maths. Put simply, standard deviation is arbitrary for most people.
darkerside 7y ago
I think you're right that there is an arbitrariness to this algorithm, and there are other ways of normalizing your data. And you could probably build entire new branches of statistics off of them (like non Euclidean geometry), and they'd probably even be better in some way. But, you'd be speaking a different language from everybody else. Speaking of language, have you ever thought about how all names and grammatical structures are arbitrary as well? There are parallels.
- username90 7y ago
  No, standard deviation is special. If you have a large enough amount of independent distributions which gets averaged or added in some way, then the only thing which matters are their standard deviations and average values for what the final distribution looks like.
  - darkerside 7y ago
    I'm well out of my wheelhouse, so anyone else chiming in would be appreciated. I'm not sure what you mean by "looks like", but I would imagine other "nonstandard" measures of deviation would also result in a consistent model. My larger point is that, something can be arbitrary and still be meaningful. If people agree on a common meaning, it formalizes a communication protocol that allows for much better information sharing.
    - username90 7y ago
      "Looks like" as in the difference approaches zero as the number of times approaches infinity. This means that any distribution with standard deviation of X will behave as a normal distribution with standard deviation X when you have enough of them. Or in other words, any distribution with standard deviation of X will behave like any other distribution with standard deviation of X when you have enough of them. There is no other measure like this as there is no other equivalent to the central limit theorem. Therefore it makes sense to have this as a universal measure of random processes.
roenxi 7y ago
Start with the Normal or Multivariate normal. We start with this because it is the distribution that emerges naturally when we have lots of little errors adding up around a basically consistent effect (see also [0]). So basically any situation where something ought to happen but 101 things could also happen to cause small changes will turn out to be normally distributed. An exceedingly common occurrence + one that almost inevitably turns up when dealing with a practical error prone process. Now that we have a normal distribution, the mean is an obvious metric to pick because it captures the notion of 'middle' in a useful sense. Then we have proofs that we can characterise the normal with the mean and 1 other parameter (normal normal) or a matrix (multivariate normal). We call that the standard deviation^. The exact formula wasn't a coincidence. The normal can obviously be characterised by the mean and a statistic from another formula. There were a bunch of experiments tried (eg, using |x| instead of sqrt[x^2]) but it turned out that sqrt[x^2] had some other nice property that minimised some sort of error so they went with it as a standard. I forget what one, might be error of estimating the true parameters from a sample or similar. We could characterise the the normal as an infinite sum or something quirky, but when people say 'easy to work with' they mean instead of a function or something quirky we can simply pick a number. Standard Deviation isn't as important when working with non-normal distributions, although I think it still turns out to be useful. But its importance is that it characterises a normal apart from the information captured in the mean. I'm not a mathematician, YMMV, could be wrong, standard disclaimers. [0] https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Central_limit_theorem ^ I'm not going to edit this but it occurs to me that we call it the Variance. Same thing as std. dev in my opinion. ' There is a fairly subtle observation to make - Normal is characterised by mean and std. dev, but the most efficient ( https://en.wikipedia.org/wiki/Efficiency_(statistics) https://en.wikipedia.org/wiki/Efficiency_(statistics) ) unbiased estimator of the std. dev of the population is the adjusted std. dev of the sample. Therefore, accounting for mean, you can't get a more efficient characterisation of the normal than mean & std. dev. Ie, if you picked a formula other than std. dev then the most efficient estimators to characterise it would still be mean and std. dev. Don't recall if there are equally efficient choices but I think that proves there are none better. That might have been the logic for why std. dev was chosen. Just a guess.
marcosdumay 7y ago
The mathematical representations and operations people see at school were picked because they are easy to work on most common problems. There is not this "correct math" thing, you pick what works best.
BeetleB 7y ago
> Isn’t it either the right metric or not? There are many "right" metrics - this is one of them. The big picture is we want a measure/idea of how spread out the data is. One can come up with many ways to do this, and absolute value and root mean square are two of the most common. They correspond to L0 and L2 norms. There is also the L-infinity norm (just use the farthest point). The L2 norm is the most complicated of the 3, but as people pointed out, it is one of the easiest to use analytically because of its nice mathematical properties. Have a look at https://en.wikipedia.org/wiki/Statistical_dispersion https://en.wikipedia.org/wiki/Statistical_dispersion for other measures.
dboreham 7y ago
>Isn’t it either the right metric or not? To illustrate, perhaps an example closer to home: If someone asked you to tell them "how many lines of code in this project?" you'd start wondering things like "do I count comments?", "do I count dependent libraries?", "perhaps some metric that is equivalent of a statement count but not strictly counting lines?" and on and on. Statistical measures are like that. Someone with a decent knowledge of the problem space came up with the best stab at how to characterize the data.