4 ms·
From the answers to the questions I am inclined to believe in the "made it all up" theory. But at a 99% confidence level I can reject the possibility that it w
by btilly 17y ago
From the answers to the questions I am inclined to believe in the "made it all up" theory. But at a 99% confidence level I can reject the possibility that it was made up with a simple simulation of the type described.
How am I so sure? Well we're given all of the probabilities for all of the answers. We can therefore calculate what (assuming independence) the likely odds are of different numbers of correct answers. We're also given the number of students who got each number correct. Here is the result (I'll give the program that I used at the end):
0 right: 46 vs 32.56
1 right: 158 vs 147.61
2 right: 246 vs 273.73
3 right: 265 vs 278.71
4 right: 177 vs 174.46
5 right: 80 vs 70.58
6 right: 22 vs 18.74
7 right: 6 vs 3.23
8 right: 0 vs 0.35
9 right: 0 vs 0.02
10 right: 0 vs 0.00
The argument is that we should be suspicious because those two sets of numbers are too similar. The tails are not heavy enough. Now I admit that graphically it does look close. But if you look, the tails _are_ heavy. Is that difference significant?
We can use a confidence testing to answer that. The null hypothesis is that the distribution comes from a simple simulation and should be close to the theoretical numbers that I just gave. The alternate hypothesis is that the tails should be heavier and the middle lower.
Let's look at the middle because it is easier to calculate. Theoretically 0.55244 of the test population should get one of those two numbers correct. It is claimed that 511/1000 did. How likely is that?
Well in the null hypothesis each student is an independent observation. Each observation is a 1 or 0 with average 0.55244. The variance of an observation is readily calculated and turns out to be 0.2472500464.
The number of people getting one of those two answers after 1000 observations is therefore approximately normal with average 552.44 and variance 1000 * 0.2472500464 = 247.2500464. The standard deviation is the square root of that, which is 15.724. That puts the observed value more than 2.63 standard deviations out. at a 99% confidence we can conclude that the numbers were NOT generated by a simple simulation of the type described.
For the curious, here is how I calculated the theoretical distribution of answers:
#! /usr/bin/perl -w
use strict;
use Math::Polynomial;
my $ans = 1;
# Number from the % of students who got each question right.
for my $x (qw(.28 .26 .27 .1 .14 .61 .43 .11 .23 .29)) {
$ans *= Math::Polynomial->new(1-$x, $x);
}
print "$ans\n";
__END__
(5.295902595984e-07 x^10 + 2.07154949414e-05 x^9 +
0.000345419801607472 x^8 + 0.00322944050567802 x^7 +
0.0187445906666514 x^6 + 0.0705826597827996 x^5 +
0.174464060957845 x^4 + 0.278709766012616 x^3 +
0.273731283943492 x^2 + 0.147609870020204 x +
0.0325616632239042)
- dagw 17y agoAssuming the people faking these sort of numbers aren't complete morons they'll know that this sort of naive statistical test is the first thing people will try. Thus they'll probably run their simulation and then manually poke a few of the numbers to make sure that this sort of test fails. Also If they're slightly smarter still they'll realize that the chance of getting a question right or wrong is not independent. You're chance of someone getting questions 5-10 right is probably quite closely correlated to whether or not they got questions 1-4 right. Anyway my basic point is that assuming the person in charge of writing a simulation didn't sleep through all of statistics 101 it is quite trivial to write a simple simulation that won't be detected by simple statistical analysis.
- btilly 17y agoYou're overestimating them. The final digit analysis that broke this whole story is something that only catches numbers being made up by hand. If they rolled a die to randomize things, they wouldn't have left an obvious pattern like that. Given that, it was reasonable to investigate the obvious possibility first for this case. Particularly since there was a suggestion that they had made up the set of numbers with such a simulation.
- dagw 17y agoMaybe I'm guilty of overestimating them, but I always find it hard to believe that people who would go through the trouble of faking these sort of things can be so monumentally stupid. These people work with polling and statistics, they must know about all of these tests. It cannot be that hard to use the knowledge to write a randomizer which produces reasonable results...maybe that's a good idea for a startup.
- camccann 17y agoThese people work with polling and statistics Well, current evidence seems to suggest that maybe they don't, actually, work with polling and statistics...