4 ms·
Try this? https://web.stanford.edu/class/ee486/doc/chap5.pdf https://web.stanford.edu/class/ee486/doc/chap5.pdf Edit: standard deviation is just a fancy name
by dboreham 7y ago
Try this?
https://web.stanford.edu/class/ee486/doc/chap5.pdf https://web.stanford.edu/class/ee486/doc/chap5.pdf
Edit: standard deviation is just a fancy name for root mean squared over the sample offsets. The square is there to deal with the fact that there are positive and negative offsets. The mean is the whole point of the exercise, and the root is to negate the fact that we squared earlier.
- chrisseaton 7y agoBut why not use the abs function or operator rather than squaring and rooting? This is how I always find maths and particularly statistics - I always just want to answer ‘but why’ and then people lose interest in explaining after a while.
- defen 7y agoThat exists too: https://en.wikipedia.org/wiki/Average_absolute_deviation https://en.wikipedia.org/wiki/Average_absolute_deviation Standard deviation is mathematically easier to work with though and has nice properties e.g. you can compute the standard deviation of combining a bunch of independent things by combining their individual standard deviations.
- smabie 7y agoBecause the derivative at 0 of abs is undefined.
- chrisseaton 7y agoWhy do I care about that though?
- smabie 7y agoTaylor series, calculating maxima/minima, etc. also standard deviations compose in a very nice way. All throughout math, we use squared terms instead of absolute value all the time because it’s convenient.
- cjfd 7y agoOkay, we have deviations from the mean. They are (0.1, 0.1, -0.1, -0.1). Or we have deviations from the mean (0.2, 0, 0, -0.2). I think is is pretty intuitive that in the second case the deviation is larger. In the first case there are only two differing values that are 0.2 apart. In the second case there are three differing values and some are 0.2 apart and others are 0.4 apart. Also, in the first case there are two pairs of equal value and in the second case there is only one pair that is of equal value. The squaring and rooting does this for you. Of course, other functions could do a similar thing for you. The squaring and rooting has nice mathematical properties especially if you consider the variance, i.e., omitting the square root. That is much more differentiable than an absolute value. Also, consider that the correlation coefficient consists of similar formulas. They all look like the theorem of pythagoras. This ties in nicely with the fact that observables can be seen as having angles in between them where the correlation coefficient is the cosine of these angles.
- chrisseaton 7y agoI still don’t understand why we pick a metric because it’s easy to work with. Isn’t it either the right metric or not?
- deleted 7y ago[deleted]
- username90 7y agoThe standard deviation is not arbitrary, it tells us what the final normal distribution would look like if you averaged a lot of independent distributions. You can read up on the central limit theorem if you want to understand more. https://en.wikipedia.org/wiki/Central_limit_theorem https://en.wikipedia.org/wiki/Central_limit_theorem
- chrisseaton 7y agoI don't want to keep going on because it sounds like I'm trolling, but when you tell me 'it tells us what the final normal distribution would look like if you averaged a lot of independent distributions' I just want to say 'why do I care about that? where did that requirement come from?'
- OscarCunningham 7y ago> But why not use the abs function or operator rather than squaring and rooting? You can. It's called the mean absolute deviation. Which one you use depends on your application. If you just want to give a summary of how spread out the data is then either would work. Lots of people say that we should use the mean absolute deviation as the default rather than the standard deviation.
- TeMPOraL 7y agoAnd suddenly another small subarea of maths starts to make sense for me. Thank you, and thanks to everyone commenting in this subthread. I too have a problem with math tools passed down without any surrounding context - without telling why are we using this formula, instead of any other variant from the family of formulas that would satisfy the same goals. I'd have much easier time dealing with statistics in school if someone told me that a) deviation with ABS instead of root-square is also a thing, and b) we use the root-square one because it amplifies offsets from the mean.
- zorked 7y agoBy the way, statistics as its own discipline is taught rather differently from the "here is a toolkit of magic tables and formulas and how to use it" method used when teaching it to non-statisticians. The magical bits actually make sense.
- dboreham 7y ago>I too have a problem with math tools passed down without any surrounding context Also one of my peeves. However over the years I've come to realize that in addition to teachers who omit the context, there is also a class of student that actively doesn't want to hear the context. I'm not sure exactly why this is, but I see it in my immediate family quite a bit. A sort of <cover ears, lalalala...too much detail> kind of thing perhaps due to difficulty taking on too much information?
- Isinlor 7y agoThe mean absolute deviation should be always introduced to people before introducing standard deviation. When trying to figure out what the heck is the standard deviation I also stumbled upon the mean absolute deviation. Now, to me, the standard deviation is the mean absolute deviation. It's just a version that is more sensitive to outliers. Otherwise, I still don't feel like I intuitively (visually) understand the standard deviation. I feel like the standard deviation is a black box to basically everyone without statistics degree. It also bothers me that it is so widespread, because almost nobody know how to interpret this value. It seems to me like most scientists treat the standard deviation as a magical number that allows to compare spread between datasets and otherwise doesn't mean anything on it's own. I feel like a lot more insight could be gained from the mean absolute deviation. I'm also curious how are scientific insights affected by defaulting to something that is extra-sensitive to outliers. The effect can't be big per single research, but in collective?
- acqq 7y ago> how I always find maths and particularly statistics - I always just want to answer ‘but why’ and then people lose interest in explaining after a while. Unless you try to solve more problems yourself, you won't understand the advantages or disadvantages of some methods in solving the problems. So the real answer is: construct the examples, compute and compare, don't approach it "philosophically." All the methods used today survived because they were the solutions to some problems. It doesn't mean that everybody applies them properly: for that you have to get some experience yourself. Learning about the historical development of the methods is also for me satisfying experience: e.g. logarithms aren't a concept devised for philosophical purposes, Napier developed them to save astronomers time and limit "slippery errors" of (their) calculations: https://www.thocp.net/reference/sciences/mathematics/logarithm_hist.htm https://www.thocp.net/reference/sciences/mathematics/logarit...
- pnutjam 7y agoI think you just described the common core method. The main problem is teachers don't understand all the methods they are teaching and kids pick up on this. Then the parents don't understand all the methods and the few intellectually curios help their kids figure it out; but he vast majority of people just tell their kids it's stupid.
- acqq 7y agoI didn't address the topic of what and how children learn, I've given some ideas to the one who wrote: "I have a PhD in computer science but I cannot understand long division" For that, personal experience in devising the examples, evaluating them and comparing the results is the only way to get the understanding. One can't complain that one doesn't understand Greek texts if one personally never tried to learn the Greek letters. One could have learned "about" Greek but at the end one hasn't done the inevitably necessary preconditions to actually read Greek texts. The error in thinking is believing that because one already learned about something else somewhere else he should somehow "understand" something without doing the necessary work on that something. The solution is "make your own homework" to fill the gaps. The topic of what, how and why children learn in schools is much more complex. Edit: Also, now reading the Wikipedia article: https://en.wikipedia.org/wiki/Long_division https://en.wikipedia.org/wiki/Long_division I do understand why people don't understand it: the taught notation for the process, used in many countries, is utterly confusing to me, so my reason for confusion when looking at various notations is not that I don't understand the nature of the method but that I haven't spent time analyzing what was built on top of the basically simple idea -- I can also "understand" all the notations but I'd also need more time for that. Still, having enough experience, I can claim that I understand the idea and the process even if I never learn all the notations and conventions in different countries (which part of the process you write where, what you write and what you don't when etc). The idea is simple, the conventions enforced don't have to be. Those who only learned the conventions maybe never invested any time to figure out what's behind the conventions: what is actually being done.
- marcosdumay 7y agoThe abs has a problem in that it has no derivative at 0. I imagine that's why people go with squaring and rooting.