4 ms·

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point fr

> So rather than comparing mean performance, we'll compare minimum performance. If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have far too much variance and noise for this to be a useful test. The minimum performer could be someone who had a sudden personal crisis. Or who had 10 competitors suddenly pop up. Or any number of other circumstances outside their control. The minimum performer is, almost by definition, an outlier. It doesn't seem rational to suppose that an outlier is representative of the group. I can understand that statistically this test may be more rigorous. In practice I would expect it to be less rigorous. Because the assumption it makes (that a single outlier is representative of the group) seems even more dubious than the assumptions required for Paul's original idea.

angelbob 11y ago
Yup. The author mentions that noise is a serious problem for this method in the article, and talks about some ways he looked at trying to reduce it, but didn't come up with a good one.
JDDunn9 11y ago
You could use the mean of the bottom decile.
- notahacker 11y ago
  That would probably be a sensible stat for measuring marginal applications to a school. But when it comes to measuring portfolio startup companies, none of the bottom decile are expected to be worth anything in the medium run, and all the results that actually matter are in the top quartile...
rcthompson 11y ago
The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was used by the Allies in World War 2 to estimate the total number of German tanks by sampling the serial numbers from destroyed tanks[2].) Of course, a real thresholding process would not be perfect, so the lower bound of the distribution of accepted candidates would not be a perfect vertical cutoff as in the examples. Just like any process that adds additional variation to the data, this would reduce the statistical power. You could accept more bias in return for lower variance in your test by taking, say, the 5th percentile instead of the sample minimum as your test statistic. (You can think of the sample minimum as the zeroth percentile.) [1] https://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29#Estimation_of_maximum https://en.wikipedia.org/wiki/Uniform_distribution_%28contin... [2] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem
- nazka 11y ago
  It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?
  - chestervonwinch 11y ago
    If you are estimating a parameter with a sample statistic, two properties it should have are unbiasedness and low variance: https://en.wikipedia.org/wiki/Minimum-variance_unbiased_estimator https://en.wikipedia.org/wiki/Minimum-variance_unbiased_esti...
  - rcthompson 11y ago
    The robustness of a sample min or max for a continuous distribution is basically proportional to the density of the distribution at that extremum. A steep or vertical drop-off at the edge of the distribution is the ideal case. The article gives a formula for the statistical power of the hypothesis test derived from the sample min. It depends on the function h(x), whose purpose is to establish a lower bound on the density of the distribution at the min, and hence a lower bound on the robustness of the sample min as an estimator.
nicolapede 11y ago
> If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean) Both the mean and the minimum are a scalar-valued function of the whole sample.
- deleted 11y ago
  [deleted]
makomk 11y ago
If I'm following his argument correctly, he assumes that there is a hard cut-off at some unknown point in the performance metric being measured, below which no-one is selected and above which all or most people are, and tests whether that hard cut-off point is the same for the two groups. It's going to be an accurate test in that scenario, but that's not a good model of reality. This is especially true when the performance metric being used is noisy and measured after the people are selected - which is exactly the scenario we're actually interested in!
learnstats2 11y ago
Worse, it's about the easiest statistic to manipulate, if you know that is going to be a Key Performance Indicator. (Having said that, the mean is not difficult to manipulate) There's an important game theory issue to KPIs: - if everyone is behaving perfectly, then why bother checking the stat? - if someone is behaving maliciously, and wants to game the stat, will they be forced to actually improve the situation? A malicious actor barely has to break stride here.

angelbob 11y ago

Yup. The author mentions that noise is a serious problem for this method in the article, and talks about some ways he looked at trying to reduce it, but didn't come up with a good one.

JDDunn9 11y ago

You could use the mean of the bottom decile.

notahacker 11y ago
That would probably be a sensible stat for measuring marginal applications to a school. But when it comes to measuring portfolio startup companies, none of the bottom decile are expected to be worth anything in the medium run, and all the results that actually matter are in the top quartile...

rcthompson 11y ago

The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was used by the Allies in World War 2 to estimate the total number of German tanks by sampling the serial numbers from destroyed tanks[2].) Of course, a real thresholding process would not be perfect, so the lower bound of the distribution of accepted candidates would not be a perfect vertical cutoff as in the examples. Just like any process that adds additional variation to the data, this would reduce the statistical power. You could accept more bias in return for lower variance in your test by taking, say, the 5th percentile instead of the sample minimum as your test statistic. (You can think of the sample minimum as the zeroth percentile.) [1] https://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29#Estimation_of_maximum https://en.wikipedia.org/wiki/Uniform_distribution_%28contin... [2] https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem

nazka 11y ago
It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?
- chestervonwinch 11y ago
  If you are estimating a parameter with a sample statistic, two properties it should have are unbiasedness and low variance: https://en.wikipedia.org/wiki/Minimum-variance_unbiased_estimator https://en.wikipedia.org/wiki/Minimum-variance_unbiased_esti...
- rcthompson 11y ago
  The robustness of a sample min or max for a continuous distribution is basically proportional to the density of the distribution at that extremum. A steep or vertical drop-off at the edge of the distribution is the ideal case. The article gives a formula for the statistical power of the hypothesis test derived from the sample min. It depends on the function h(x), whose purpose is to establish a lower bound on the density of the distribution at the min, and hence a lower bound on the robustness of the sample min as an estimator.

nicolapede 11y ago

> If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean) Both the mean and the minimum are a scalar-valued function of the whole sample.

deleted 11y ago
[deleted]

makomk 11y ago

If I'm following his argument correctly, he assumes that there is a hard cut-off at some unknown point in the performance metric being measured, below which no-one is selected and above which all or most people are, and tests whether that hard cut-off point is the same for the two groups. It's going to be an accurate test in that scenario, but that's not a good model of reality. This is especially true when the performance metric being used is noisy and measured after the people are selected - which is exactly the scenario we're actually interested in!

learnstats2 11y ago

Worse, it's about the easiest statistic to manipulate, if you know that is going to be a Key Performance Indicator. (Having said that, the mean is not difficult to manipulate) There's an important game theory issue to KPIs: - if everyone is behaving perfectly, then why bother checking the stat? - if someone is behaving maliciously, and wants to game the stat, will they be forced to actually improve the situation? A malicious actor barely has to break stride here.