3 ms·

It's statistics 101 to not compare median to average, simply because an average is more impacted by outlier values and so a member of the general population mak

by AdamSC1 10y ago

It's statistics 101 to not compare median to average, simply because an average is more impacted by outlier values and so a member of the general population making $15m a year would lift the "average" income far more than it would lift the "median" income. The numbers set "1, 2, 3, 4, 1000" has an average of 202 but a median of 3, and 3 is clearly a better representation of those numbers as a group. Also we've got multiple data sets at play and so we shouldn't be using a "ratio" but rather a z-score. A z-score is a measure to normalize all the different distributions and see how many standard deviations away from the average each value is. It's the accepted practice for comparing across two different data sets. Udacity covers it in their data science course: https://www.udacity.com/course/viewer#!/c-st095/l-81689336/m-92314592 https://www.udacity.com/course/viewer#!/c-st095/l-81689336/m...

whorleater 10y ago
It's not necessarily wrong to compare the median to the average once a sufficiently large dataset is reached given LLN, but I'd argue that a z-score is still far too basic to capture the nuances of such a varying dataset with so many points. Some type of hierarchical model would probably be best for this dataset.
- AdamSC1 10y ago
  Fair point on the z-score still to simple but better than a ratio. As far as the LLN, also right but this assumes that PayScale has anywhere close to enough datapoints for that number to be reached which I would question. Also not sure if we can consider the LLN in the same way if you are comparing a total population with a sample here? That I guess is the other compounding factor actually, the population of Software Engineers is actually just a sample of the overall population of "the general public's income" and so that does impact this as well. We're not comparing group A to group B but rather group A to group ABC...N.

by AdamSC1 10y ago

whorleater 10y ago

It's not necessarily wrong to compare the median to the average once a sufficiently large dataset is reached given LLN, but I'd argue that a z-score is still far too basic to capture the nuances of such a varying dataset with so many points. Some type of hierarchical model would probably be best for this dataset.

AdamSC1 10y ago
Fair point on the z-score still to simple but better than a ratio. As far as the LLN, also right but this assumes that PayScale has anywhere close to enough datapoints for that number to be reached which I would question. Also not sure if we can consider the LLN in the same way if you are comparing a total population with a sample here? That I guess is the other compounding factor actually, the population of Software Engineers is actually just a sample of the overall population of "the general public's income" and so that does impact this as well. We're not comparing group A to group B but rather group A to group ABC...N.