5 ms·

> (d) Compute the mean age, and print it out as the median. It's much easier to program, and Mary Jones will probably never notice the difference. Possible imp

> (d) Compute the mean age, and print it out as the median. It's much easier to program, and Mary Jones will probably never notice the difference. Possible improvement: The median will always be a member of the actual dataset, so after calculating the mean, run through the data to find the entry which is closest to it, and return that for the "median". Could make our dastardly plan harder to uncover.

eesmith 4y ago
If there is an even number of elements then the median is usually defined as the arithmetic mean of the two middle values. Eg, the median of [1, 3, 5, 7] is 4.
- mgaunard 4y ago
  I prefer the 2-quantile myself.
- taylorius 4y ago
  That's true - maybe that gets us off the hook. :-)
- qsort 4y ago
  Depends on how you define 'median'. - Any value that minimizes the sum of absolute differences? Then all real numbers between 3 and 5 are a median of that set. - With an inequality over the PDF? The same as above. - The 1/2-quantile? Then you need to tell me if we're rounding up or down. I've seen that definition before, but it's a completely useless definition for any legitimate use case of the median as an order statistic. I have no idea where it comes from.
  - irishsultan 4y ago
    In any reasonable definition of what a median is then there is either a unique value, or the mean between the two middle elements is a median, so it's perfectly reasonable to take the average (something which you can always do and gives a unique answer).
    - bloak 4y ago
      If they are real numbers then you can take the arithmetic mean of the two middle elements, but you might prefer to define the median in such a way that the definition is applicable to any set with a total order. For example, names can be ordered alphabetically, so you can find the median of a set of names! OK, that's a silly example, but I expect someone could come up with a more sensible one. Another case might be where you're using geometric means everywhere else in your working, but then you want a median in one place, but there are an even number of elements, so why would you suddenly use an arithmetic mean of the two middle elements?
      - qsort 4y ago
        > I expect someone could come up with a more sensible one. - Imagine you hand out a survey where some answers are on a likert scale. "strongly dislike" is worse than "dislike", which is worse than "neutral" and so on. - The median ranking over a set of ranking on different criteria. - When the domain is not the real numbers. "The median employee of this company has 1.5 kids".
      - hgsgm 4y ago
  - d1sxeyes 4y ago
    https://en.wikipedia.org/wiki/Median https://en.wikipedia.org/wiki/Median has this definition. It's also the definition that I was taught at school. Oxford Languages defines it as: "denoting or relating to a value or quantity lying at the midpoint of a frequency distribution of observed values or quantities, such that there is an equal probability of falling above or below it", which also supports the idea of taking the midpoint between the two middle values. Merriam Webster define it explicitly in the way OP does (arithmetic mean of the two middle values): https://www.merriam-webster.com/dictionary/median https://www.merriam-webster.com/dictionary/median I'm not sure why you think it's a 'completely useless definition'. It would perhaps be a logical argument that data sets of even length do not have a median, but I think this would be even more useless. To the layperson (which Mary Jones undoubtedly is), providing the 'median' age as an age half of the employees are older than/younger than, is what would be expected. Your first bullet point makes sense here. For example here: 18, 20, 25, 30, [35, 40], 45, 50, 55, 60 Most people would expect the median to be calculated as 37.5. However, as you mention, any real number higher than 35 and lower than 40 would equally fulfill the definition of 'equal probability of falling above or below' assuming this is the full population. Would you then argue that there are infinitely many medians in this data set, or none? There are other issues with calculating 37.5, such as the fact it implies a higher degree of precision than it ought to. But ultimately, as a practical and usable insight into data, especially data with extreme outliers or high levels of skew, I strongly disagree that this definition is completely useless.
    - qsort 4y ago
      > Would you then argue that there are infinitely many medians in this data set By the definition I usually go by (the one with the inequalities), that dataset has infinitely many medians. There is no reason to say the midpoint is privileged. This is a better definition because it also works on sets where an order is defined but the arithmetic mean is not. If asked to report a single value with no qualifications, I would report either extreme of the interval.
      - d1sxeyes 4y ago
        > There is no reason to say the midpoint is privileged. This is a matter of opinion. I agree it's not 'privileged', but it sacrifices a minimal amount of accuracy for a usable answer. > If asked to report a single value with no qualifications, I would report either extreme of the interval. That's a very interesting answer - so for my contrived example of 18, 20, 25, 30, [35, 40], 45, 50, 55, 60, you would report 35 or 40? How about if I change the dataset a bit (so it's even more contrived): 18, 18, 18, 18, [18, 60], 60, 60, 60, 60 - would you still report either 18 or 60 (again, asking without qualifications)? I think most people would agree that the median calculated as the arithmetic mean of the two middle values (39) is misleading, but that it is less misleading than ignoring half of the data set. How would you respond to that?
  - deleted 4y ago
    [deleted]
  - eesmith 4y ago
    > I have no idea where it comes from. I went to archive.org and searched for intro books in statistics (there are a lot!). All of them gave preference to the average of the two middle values. One of them also pointed out there were valid alternatives. "Since there are 10 values [in the sorted list], the median is the average of the fifth and sixth values. However, it could be also be 4.1, or 4.9, or any other value between 4 or 5, since each of these values divides the 10 values into two equal groups." - https://archive.org/details/introductiontost0000chao/page/54/mode/2up?q=median https://archive.org/details/introductiontost0000chao/page/54... "For a sample of size n, the sample median is the ... Average of the two order statistics in the middle if n is even." - https://archive.org/details/introductiontost0000salv/page/64/mode/2up?q=median https://archive.org/details/introductiontost0000salv/page/64... "We find no middle value; therefore, the median is the mean of the two middle values" - https://archive.org/details/US_Navy_Training_Course_-_Introduction_to_Statistics_Number_Systems_and_Boolean_/page/n45/mode/2up?q=median https://archive.org/details/US_Navy_Training_Course_-_Introd... "The sample median for this collection is given by the middle observation if n is odd. If n is even, the sample median is the average of the two middle observations." - https://archive.org/details/introductiontost0000milt_g9m2/page/36/mode/2up?q=median https://archive.org/details/introductiontost0000milt_g9m2/pa... "sample median = ... the average of the middle two values if n is even" - https://archive.org/details/introductiontost0000peck_o4j9/page/152/mode/2up?q=median https://archive.org/details/introductiontost0000peck_o4j9/pa... "If there were an even number of observations (say, six instead of five), the calculation would be more complex. We would need to find the middle pair of numbers (the third and fourth observations), and then find the value that’s halfway between them, by adding the values together and dividing by two." - https://archive.org/details/introductiontost0000haan/page/n67/mode/2up?q=median https://archive.org/details/introductiontost0000haan/page/n6... "If there is an even number of values, the median is equal to the mean of the two middle values of the ordered set." - https://archive.org/details/introductiontoec0000merr/page/18/mode/2up?q=median https://archive.org/details/introductiontoec0000merr/page/18...
hhmc 4y ago
Also, the median can only be 1stddev away from the mean in any distribution, so this can further winnow the search.
- mjburgess 4y ago
  ...this isn't true: 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,10^30
  - zeristor 4y ago
    Reminds me of the distribution in salaries. I read …, 1, 10, 30 but …, 1, 10^30 is far more dramatic.
  - Chirono 4y ago
    I’d double check your working there because it certainly is true. In your example the standard deviation (2e29) is far bigger than the difference between the median and mean (5e28).
  - hhmc 4y ago
    Someone has already explained why your example is wrong, but for a general proof, for dist X, let u = mean, m = median | u - m | = |E(X - m)| <= E(|X - m|) <= E(|X - u|) <= E(sqrt((X - u)^2)) <= stddev mostly steps via Jensen's inequality
    - mjburgess 4y ago
      oops... I interpreted the claim, admittedly bizarrely, as something like: fit N(u, s) to {(x, y)...} then m will lie u+-s rather than, def. s^2 = E[X^2] - E[X]^2

eesmith 4y ago

If there is an even number of elements then the median is usually defined as the arithmetic mean of the two middle values. Eg, the median of [1, 3, 5, 7] is 4.

mgaunard 4y ago
I prefer the 2-quantile myself.
taylorius 4y ago
That's true - maybe that gets us off the hook. :-)
qsort 4y ago
Depends on how you define 'median'. - Any value that minimizes the sum of absolute differences? Then all real numbers between 3 and 5 are a median of that set. - With an inequality over the PDF? The same as above. - The 1/2-quantile? Then you need to tell me if we're rounding up or down. I've seen that definition before, but it's a completely useless definition for any legitimate use case of the median as an order statistic. I have no idea where it comes from.
- irishsultan 4y ago
  In any reasonable definition of what a median is then there is either a unique value, or the mean between the two middle elements is a median, so it's perfectly reasonable to take the average (something which you can always do and gives a unique answer).
  - bloak 4y ago
    If they are real numbers then you can take the arithmetic mean of the two middle elements, but you might prefer to define the median in such a way that the definition is applicable to any set with a total order. For example, names can be ordered alphabetically, so you can find the median of a set of names! OK, that's a silly example, but I expect someone could come up with a more sensible one. Another case might be where you're using geometric means everywhere else in your working, but then you want a median in one place, but there are an even number of elements, so why would you suddenly use an arithmetic mean of the two middle elements?
    - qsort 4y ago
      > I expect someone could come up with a more sensible one. - Imagine you hand out a survey where some answers are on a likert scale. "strongly dislike" is worse than "dislike", which is worse than "neutral" and so on. - The median ranking over a set of ranking on different criteria. - When the domain is not the real numbers. "The median employee of this company has 1.5 kids".
    - hgsgm 4y ago
- d1sxeyes 4y ago
  https://en.wikipedia.org/wiki/Median https://en.wikipedia.org/wiki/Median has this definition. It's also the definition that I was taught at school. Oxford Languages defines it as: "denoting or relating to a value or quantity lying at the midpoint of a frequency distribution of observed values or quantities, such that there is an equal probability of falling above or below it", which also supports the idea of taking the midpoint between the two middle values. Merriam Webster define it explicitly in the way OP does (arithmetic mean of the two middle values): https://www.merriam-webster.com/dictionary/median https://www.merriam-webster.com/dictionary/median I'm not sure why you think it's a 'completely useless definition'. It would perhaps be a logical argument that data sets of even length do not have a median, but I think this would be even more useless. To the layperson (which Mary Jones undoubtedly is), providing the 'median' age as an age half of the employees are older than/younger than, is what would be expected. Your first bullet point makes sense here. For example here: 18, 20, 25, 30, [35, 40], 45, 50, 55, 60 Most people would expect the median to be calculated as 37.5. However, as you mention, any real number higher than 35 and lower than 40 would equally fulfill the definition of 'equal probability of falling above or below' assuming this is the full population. Would you then argue that there are infinitely many medians in this data set, or none? There are other issues with calculating 37.5, such as the fact it implies a higher degree of precision than it ought to. But ultimately, as a practical and usable insight into data, especially data with extreme outliers or high levels of skew, I strongly disagree that this definition is completely useless.
  - qsort 4y ago
    > Would you then argue that there are infinitely many medians in this data set By the definition I usually go by (the one with the inequalities), that dataset has infinitely many medians. There is no reason to say the midpoint is privileged. This is a better definition because it also works on sets where an order is defined but the arithmetic mean is not. If asked to report a single value with no qualifications, I would report either extreme of the interval.
    - d1sxeyes 4y ago
      > There is no reason to say the midpoint is privileged. This is a matter of opinion. I agree it's not 'privileged', but it sacrifices a minimal amount of accuracy for a usable answer. > If asked to report a single value with no qualifications, I would report either extreme of the interval. That's a very interesting answer - so for my contrived example of 18, 20, 25, 30, [35, 40], 45, 50, 55, 60, you would report 35 or 40? How about if I change the dataset a bit (so it's even more contrived): 18, 18, 18, 18, [18, 60], 60, 60, 60, 60 - would you still report either 18 or 60 (again, asking without qualifications)? I think most people would agree that the median calculated as the arithmetic mean of the two middle values (39) is misleading, but that it is less misleading than ignoring half of the data set. How would you respond to that?
- deleted 4y ago
  [deleted]
- eesmith 4y ago
  > I have no idea where it comes from. I went to archive.org and searched for intro books in statistics (there are a lot!). All of them gave preference to the average of the two middle values. One of them also pointed out there were valid alternatives. "Since there are 10 values [in the sorted list], the median is the average of the fifth and sixth values. However, it could be also be 4.1, or 4.9, or any other value between 4 or 5, since each of these values divides the 10 values into two equal groups." - https://archive.org/details/introductiontost0000chao/page/54/mode/2up?q=median https://archive.org/details/introductiontost0000chao/page/54... "For a sample of size n, the sample median is the ... Average of the two order statistics in the middle if n is even." - https://archive.org/details/introductiontost0000salv/page/64/mode/2up?q=median https://archive.org/details/introductiontost0000salv/page/64... "We find no middle value; therefore, the median is the mean of the two middle values" - https://archive.org/details/US_Navy_Training_Course_-_Introduction_to_Statistics_Number_Systems_and_Boolean_/page/n45/mode/2up?q=median https://archive.org/details/US_Navy_Training_Course_-_Introd... "The sample median for this collection is given by the middle observation if n is odd. If n is even, the sample median is the average of the two middle observations." - https://archive.org/details/introductiontost0000milt_g9m2/page/36/mode/2up?q=median https://archive.org/details/introductiontost0000milt_g9m2/pa... "sample median = ... the average of the middle two values if n is even" - https://archive.org/details/introductiontost0000peck_o4j9/page/152/mode/2up?q=median https://archive.org/details/introductiontost0000peck_o4j9/pa... "If there were an even number of observations (say, six instead of five), the calculation would be more complex. We would need to find the middle pair of numbers (the third and fourth observations), and then find the value that’s halfway between them, by adding the values together and dividing by two." - https://archive.org/details/introductiontost0000haan/page/n67/mode/2up?q=median https://archive.org/details/introductiontost0000haan/page/n6... "If there is an even number of values, the median is equal to the mean of the two middle values of the ordered set." - https://archive.org/details/introductiontoec0000merr/page/18/mode/2up?q=median https://archive.org/details/introductiontoec0000merr/page/18...

hhmc 4y ago

Also, the median can only be 1stddev away from the mean in any distribution, so this can further winnow the search.

mjburgess 4y ago
...this isn't true: 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,10^30
- zeristor 4y ago
  Reminds me of the distribution in salaries. I read …, 1, 10, 30 but …, 1, 10^30 is far more dramatic.
- Chirono 4y ago
  I’d double check your working there because it certainly is true. In your example the standard deviation (2e29) is far bigger than the difference between the median and mean (5e28).
- hhmc 4y ago
  Someone has already explained why your example is wrong, but for a general proof, for dist X, let u = mean, m = median | u - m | = |E(X - m)| <= E(|X - m|) <= E(|X - u|) <= E(sqrt((X - u)^2)) <= stddev mostly steps via Jensen's inequality
  - mjburgess 4y ago
    oops... I interpreted the claim, admittedly bizarrely, as something like: fit N(u, s) to {(x, y)...} then m will lie u+-s rather than, def. s^2 = E[X^2] - E[X]^2