3 ms·
One point about this that I think is overlooked is that if you don't include some sort of randomness, you'll end up over-optimizing for whatever you're measurin
by JoshuaDavid 6y ago
One point about this that I think is overlooked is that if you don't include some sort of randomness, you'll end up over-optimizing for whatever you're measuring, and under-optimizing for the thing you actually care about. More concretely, if we say that there are some underlying factors which correlate to both student "quality" and to admissions scores, and some random factors, we'll find that score and quality are correlated, but the highest quality students and the ones with the highest scores aren't the same ones. This is easier to explain using a graph: if we assume that "score" and "quality" are two gaussian variables, we get a graph that looks like
16 ^ Quality
15 | #####BBBB
14 | ########BBBBA
13 | ###########BBBBAA
12 | #############BBBBAA
11 | ###############CCCCAA
10 | #################CCCCAA
9 | ##################CCCCA
8 | ###################CCCC
7 | ####################CCC
6 | #####################CC
5 | #####################
4 | ###################
3 | #################
2 | #############
1 | #########
+----+----+----+----+----+----+>
0 5 10 15 20 25 30
Score
If you do the naive thing, and grab the very highest-scoring students, we set the cutoff at 27, and get all 10 of the students labeled by A in the above diagram, thus have a quality distribution of 10 - 14, with the bulk being around 10 - 13, and the average score being 11.5. If we instead set the cutoff at 23, we get 47 students (labelled B for students above the average of the originally selected ones, and C for the ones below that average) -- if we then pick among them randomly, we'll get a flatter distribution ranging from a low score of 6 to a high score of 15, with an average of 10.98. The distribution of scores looks like
Score | Strict | Cutoff/
| Cutoff | Random
------+--------+--------
15 | 0.00% | 8.51%
14 | 10.00% | 10.64%
13 | 20.00% | 12.77%
12 | 20.00% | 12.77%
11 | 20.00% | 12.77%
10 | 20.00% | 12.77%
9 | 10.00% | 10.64%
8 | 0.00% | 8.51%
7 | 0.00% | 6.38%
6 | 0.00% | 4.26%
So if you care a lot about simply not getting bad students, and not very much about getting the very best ones, you should do a strict cutoff, since that will give you the highest possible average. If you care more about getting the very best students, and are ok if you let in a few relative duds, you'll do better to lower your cutoff and select randomly from everyone who passed your cutoff.
Elite colleges get their reputation from having students who go on to do great things, so I'd argue that at least for them, the "cutoff plus random selection" makes sense.