4 ms·

> I ran a binomial analysis and found the likelihood that someone whose answers matched on 19 out of the 45 planted questions had about a 1:100 chance of doing

> I ran a binomial analysis and found the likelihood that someone whose answers matched on 19 out of the 45 planted questions had about a 1:100 chance of doing so This is just bad math. Take a question (A), there are 5 possible answers, using his analysis the probability of a student picking the same answer as the cheating answer is 1 in 5. But let's say the question is hard, and of the 5 possible answers, 2 are highly plausible, so plausible in fact that the students always go for 1 of the 2. Now, if the cheating answer is one of the plausible answers the probability is low, but if it's one of the plausible answers, then it's high. And more specifically, if the cheat answer is correct - what's the probability the student got it right? Well what you should be doing is take the other 55 non-cheating answers, calculate the probaility of correctness and then use that as the probability. The "1/100" threshold is overwhelmingly determine by how he selected the answers on the leaked answer sheet, and you can't say it just average out with such a small sample. Modelling the whole thing as random choosing is just sloppy maths.

luma 3y ago
> Most of these answers were not just wrong, but obviously wrong to anyone who had paid attention in class. Already accounted for per the article.
- SilverBirch 3y ago
  No it's not, he says some of the answers were obviously wrong, but he did the binomial analysis on 45 questions. If you plant 45 answers in a 80 answer test and they're mostly obviously wrong then the students who cheat are going to fail either way, so it can't really be true that they were wrong. And if the answers were very obviously wrong then the probability of choosing the wrong answer for each would be much higher than assuming a 20% chance of matching the fake answer. So the whole 1in 100 is highly questionable. To put it another way, let's say he planted only wrong answers, and we have an idiot student that always gets the questions wrong. Now the probability of matching the planted answer has clearly jumped, right? On the flip side, if the planted answers are right, and in general you expect your class to pick correct answers (ie, they're going to pass) then the probability they pick the same as the cheat answer could be as high as 90%
tgv 3y ago
It isn't. "Most of these answers were not just wrong, but obviously wrong", according to the teacher. The chances of picking the worst possible answer is lower than 0.2. If the .01 threshold is fair? It isn't if you assume that someone tried to pass the test by filling out the whole thing at random. If the students tried to pass by choosing the correct answer, >= 19 obviously wrong answers out of 45 seems fair enough.
- SilverBirch 3y ago
  I'm not saying the conclusion they're cheating is wrong - in fact the threshold is way to generous, but fundamentally the statistical model he's picked just isn't a good fit.
  - tgv 3y ago
    What model would you propose? You don't know the likelihood in advance, do you? The only thing that could have been added (that I can think of!) is that the likelihood of cheating is not memoryless, as the binomial process assumes, but rather that a cheater would not copy a few answers, but very many. So the chance that a cheater picks the bad answer is > 0.2, the chance that someone who doesn't know the answer = 0.2, and the chance that someone who does < 0.2. But given the fact that we don't know the correct probabilities, nor the scale of cheating, and not even how often people take a random guess, makes the model a decent starting point, IMO, for getting a first estimate. After all, the teacher is not researching stochastic models of cheating, but just grading papers.
    - SilverBirch 3y ago
      I'm not sure there's a completely thorough way of proving cheating on a single measure of a relatively short test. If I wanted something more thorough I'd probably compare the mean score on the non-cheat questions vs the score on the cheat questions though. Let's say you provide the correct answers on the "leaked" test. If the score on the leaked questions is significantly higher than the score on the unseen questions then you can confidently say there was some cheating. At that point you'd probably interview the candidates on their tests rather than throw accusations.

luma 3y ago

> Most of these answers were not just wrong, but obviously wrong to anyone who had paid attention in class. Already accounted for per the article.

SilverBirch 3y ago
No it's not, he says some of the answers were obviously wrong, but he did the binomial analysis on 45 questions. If you plant 45 answers in a 80 answer test and they're mostly obviously wrong then the students who cheat are going to fail either way, so it can't really be true that they were wrong. And if the answers were very obviously wrong then the probability of choosing the wrong answer for each would be much higher than assuming a 20% chance of matching the fake answer. So the whole 1in 100 is highly questionable. To put it another way, let's say he planted only wrong answers, and we have an idiot student that always gets the questions wrong. Now the probability of matching the planted answer has clearly jumped, right? On the flip side, if the planted answers are right, and in general you expect your class to pick correct answers (ie, they're going to pass) then the probability they pick the same as the cheat answer could be as high as 90%

tgv 3y ago

It isn't. "Most of these answers were not just wrong, but obviously wrong", according to the teacher. The chances of picking the worst possible answer is lower than 0.2. If the .01 threshold is fair? It isn't if you assume that someone tried to pass the test by filling out the whole thing at random. If the students tried to pass by choosing the correct answer, >= 19 obviously wrong answers out of 45 seems fair enough.

SilverBirch 3y ago
I'm not saying the conclusion they're cheating is wrong - in fact the threshold is way to generous, but fundamentally the statistical model he's picked just isn't a good fit.
- tgv 3y ago
  What model would you propose? You don't know the likelihood in advance, do you? The only thing that could have been added (that I can think of!) is that the likelihood of cheating is not memoryless, as the binomial process assumes, but rather that a cheater would not copy a few answers, but very many. So the chance that a cheater picks the bad answer is > 0.2, the chance that someone who doesn't know the answer = 0.2, and the chance that someone who does < 0.2. But given the fact that we don't know the correct probabilities, nor the scale of cheating, and not even how often people take a random guess, makes the model a decent starting point, IMO, for getting a first estimate. After all, the teacher is not researching stochastic models of cheating, but just grading papers.
  - SilverBirch 3y ago
    I'm not sure there's a completely thorough way of proving cheating on a single measure of a relatively short test. If I wanted something more thorough I'd probably compare the mean score on the non-cheat questions vs the score on the cheat questions though. Let's say you provide the correct answers on the "leaked" test. If the score on the leaked questions is significantly higher than the score on the unseen questions then you can confidently say there was some cheating. At that point you'd probably interview the candidates on their tests rather than throw accusations.