5 ms·

The only sort of data you can trust is the data itself. Providing the mean of data is providing a sum value a person thinks is useful to convey the shape of the

by quchen 13y ago

The only sort of data you can trust is the data itself. Providing the mean of data is providing a sum value a person thinks is useful to convey the shape of the data, and the mean in particular is unsuitable at conveying shape on its own. What the author tries to get at is a precise quantity, namely the average salary for each language. However, we cannot measure that quantity directly, so taking samples (from a good source) is as good as it gets. The easiest assumption about that data is that it follows a normal distribution, which does have a mean, but also a standard deviation; both of these are required to talk about any meaning of the result. (If the normal distribution is not a good fit things may get more complicated.) What I'm saying is that providing just means is not a ballpark way of doing statistics, it is wrong. The statistical ballpark is the normal distribution, and maybe a value ensuring that it is at least somewhat appropriate to use that one.

lifeisstillgood 13y ago
So forgive the statistical ignorance, but how would one display mean and std dev in useful graphical form? (Asking seriously as it would be good to know which setting to press on gnu plot)
- route66 13y ago
  With boxplots? http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.php http://benchmarksgame.alioth.debian.org/u64q/which-programs-... (they can show a bit more though, min, max, interquartile range or mean+/- 1 stddev and mean itself) see also http://en.wikipedia.org/wiki/Box_plot http://en.wikipedia.org/wiki/Box_plot .
- merijnv 13y ago
  Not quite the same, but box-and-whisker plots (https://en.wikipedia.org/wiki/Box_plot https://en.wikipedia.org/wiki/Box_plot) are good for visualising distributions. They show the mean, minimum, maximum and the first and second quartiles. Since they show the minimum and maximum we can judge the overall spread, the mean tells us...well...the mean and first and second quartile lets us judge how close data is clustered around the mean. See http://gnuplot.sourceforge.net/demo/candlesticks.html http://gnuplot.sourceforge.net/demo/candlesticks.html for gnuplot examples.
  - lifeisstillgood 13y ago
    But box plots seem to be needing turning on their side to visualise (assuming normal distribution) I suppose I am looking for mathematically faithful whilst still Omnigraffle good looks Cheers
  - omh 13y ago
    A minor point, but I think Box Plots usually show the median rather than the mean at the centre. It would make sense perhaps to use standard deviations and means for some data, but in cases like this I think quartiles and medians make more sense.
- deleted 13y ago
  [deleted]
- ichinco 13y ago
  even just some error bars would help: http://en.wikipedia.org/wiki/Error_bar http://en.wikipedia.org/wiki/Error_bar They're typically used for confidence intervals or standard error, but you can theoretically use them with any measure of variability as long as you're clear.
- glaugh 13y ago
  Personally I prefer histograms (assuming you have all the data, not just mean and std dev). For example, here's a histogram of ages of passengers of the Titanic: https://www.statwing.com/demos/titanic#workspaces/21 https://www.statwing.com/demos/titanic#workspaces/21 Or rents in the bay area: https://www.statwing.com/open/datasets/034b4af16d50a0ad35f1aa39b4002680734e219d#workspaces/6909 https://www.statwing.com/open/datasets/034b4af16d50a0ad35f1a... Gives a real nice feel for the data. These histograms are lacking in that they don't actually have the mean line or std devs on the plot itself; my general point is that if the goal is to get across a distribution, a histogram is the best way to do it, perhaps aided by markers for mean, median, std dev, etc., as the case merits. Disclosure: I work at Statwing.
dlss 13y ago
I'm confused. I was trying to say something along the lines of "once you've played with rapleaf, you won't care what the stdev is", because rapleaf's data isn't good. The standard deviation here isn't going to be the standard deviation of programmer salaries -- it will be the standard deviation of rapleafs estimates. Which as I previously mentioned are horrible, and not particularly interesting. I agree with the author that Haskell is estimated at a lower rate because it's academic. I think the reason we see higher salaries for older languages here is that rapleaf must be using age as an input (with makes sense in some fields). > The only sort of data you can trust is the data itself. Agree. Again, here there is no data. It's gibbs sampled noise from shitty models. > Providing the mean of data is providing a sum value a person thinks is useful to convey the shape of the data, and the mean in particular is unsuitable at conveying shape on its own. I don't think anyone here is confused about the question of the mean showing the shape of data. > The easiest assumption about that data is that it follows a normal distribution Salary most certainly does not follow a normal distribution. I don't understand how you came to write what you wrote without knowing that. It's a classic example of the exponential distribution. (To be fair, salary estimates from rapleaf do seem to follow a normal distribution :p maybe you can guess why? [hint, it's a phrase I used earlier that starts with "gibbs sampled noise from" and ends with "shitty models"])

by quchen 13y ago

lifeisstillgood 13y ago

So forgive the statistical ignorance, but how would one display mean and std dev in useful graphical form? (Asking seriously as it would be good to know which setting to press on gnu plot)

route66 13y ago
With boxplots? http://benchmarksgame.alioth.debian.org/u64q/which-programs-are-fastest.php http://benchmarksgame.alioth.debian.org/u64q/which-programs-... (they can show a bit more though, min, max, interquartile range or mean+/- 1 stddev and mean itself) see also http://en.wikipedia.org/wiki/Box_plot http://en.wikipedia.org/wiki/Box_plot .
merijnv 13y ago
Not quite the same, but box-and-whisker plots (https://en.wikipedia.org/wiki/Box_plot https://en.wikipedia.org/wiki/Box_plot) are good for visualising distributions. They show the mean, minimum, maximum and the first and second quartiles. Since they show the minimum and maximum we can judge the overall spread, the mean tells us...well...the mean and first and second quartile lets us judge how close data is clustered around the mean. See http://gnuplot.sourceforge.net/demo/candlesticks.html http://gnuplot.sourceforge.net/demo/candlesticks.html for gnuplot examples.
- lifeisstillgood 13y ago
  But box plots seem to be needing turning on their side to visualise (assuming normal distribution) I suppose I am looking for mathematically faithful whilst still Omnigraffle good looks Cheers
- omh 13y ago
  A minor point, but I think Box Plots usually show the median rather than the mean at the centre. It would make sense perhaps to use standard deviations and means for some data, but in cases like this I think quartiles and medians make more sense.
deleted 13y ago
[deleted]
ichinco 13y ago
even just some error bars would help: http://en.wikipedia.org/wiki/Error_bar http://en.wikipedia.org/wiki/Error_bar They're typically used for confidence intervals or standard error, but you can theoretically use them with any measure of variability as long as you're clear.
glaugh 13y ago
Personally I prefer histograms (assuming you have all the data, not just mean and std dev). For example, here's a histogram of ages of passengers of the Titanic: https://www.statwing.com/demos/titanic#workspaces/21 https://www.statwing.com/demos/titanic#workspaces/21 Or rents in the bay area: https://www.statwing.com/open/datasets/034b4af16d50a0ad35f1aa39b4002680734e219d#workspaces/6909 https://www.statwing.com/open/datasets/034b4af16d50a0ad35f1a... Gives a real nice feel for the data. These histograms are lacking in that they don't actually have the mean line or std devs on the plot itself; my general point is that if the goal is to get across a distribution, a histogram is the best way to do it, perhaps aided by markers for mean, median, std dev, etc., as the case merits. Disclosure: I work at Statwing.

dlss 13y ago

I'm confused. I was trying to say something along the lines of "once you've played with rapleaf, you won't care what the stdev is", because rapleaf's data isn't good. The standard deviation here isn't going to be the standard deviation of programmer salaries -- it will be the standard deviation of rapleafs estimates. Which as I previously mentioned are horrible, and not particularly interesting. I agree with the author that Haskell is estimated at a lower rate because it's academic. I think the reason we see higher salaries for older languages here is that rapleaf must be using age as an input (with makes sense in some fields). > The only sort of data you can trust is the data itself. Agree. Again, here there is no data. It's gibbs sampled noise from shitty models. > Providing the mean of data is providing a sum value a person thinks is useful to convey the shape of the data, and the mean in particular is unsuitable at conveying shape on its own. I don't think anyone here is confused about the question of the mean showing the shape of data. > The easiest assumption about that data is that it follows a normal distribution Salary most certainly does not follow a normal distribution. I don't understand how you came to write what you wrote without knowing that. It's a classic example of the exponential distribution. (To be fair, salary estimates from rapleaf do seem to follow a normal distribution :p maybe you can guess why? [hint, it's a phrase I used earlier that starts with "gibbs sampled noise from" and ends with "shitty models"])