3 ms·

> Also, suspicion was by only a 1:100 coincidence probability standard of evidence, and by some imperfect metric. ...only? In a sample size of 96, what's the

by ajross 3y ago

> Also, suspicion was by only a 1:100 coincidence probability standard of evidence, and by some imperfect metric. ...only? In a sample size of 96, what's the value in analysis beyond that level? No, this seems pretty solid to me. The math to do this is routine freshman statistics stuff, something every practicing scientist knows. The assumptions just require that wrong answers be roughly evenly distributed across the choices (e.g. you could construct a test where everyone who was wrong would be led to choice C and rarely B or D, but that's a little pathological; and regardless it's something that would be evident in the data set and seems not to have been). I mean, standards vary but I'll bet in most jurisdictions 100:1 odds count as "beyond a reasonable doubt" for jury instructions in criminal trials. At the very least you'd bring the trial, which is what happened here. If I know there's only a 1% chance I'm wrong, it's absolutely valid for me to accuse you of cheating.

sokoloff 3y ago
> I'll bet in most jurisdictions 100:1 odds count as "beyond a reasonable doubt" for jury instructions in criminal trials. IANAL, but I sure hope that 100:1 chance, standing alone, doesn't meet the bar for "beyond a reasonable doubt". To me, a 1% chance of an accusation being false is definitionally "reasonable doubt".
- Wowfunhappy 3y ago
  ...I have no idea where I heard this, but for some reason I was under the impression "beyond a reasonable doubt" meant ~95% confident. (Whereas "preponderance of evidence" meant >50% confident.)
- ajross 3y ago
  Per Wikipedia, the original framing is Blackstone's ratio from the enlightenment, where he used 10:1. Real courts don't put numbers on this, of course, but that's the thinking that produced the philosophy. Basically: no, given the level of innaccuracy we already know to be present, I think it's very clear that a sincerely-believed 99% confidence will send someone to jail pretty much anywhere in the US. https://en.wikipedia.org/wiki/Reasonable_doubt https://en.wikipedia.org/wiki/Reasonable_doubt https://en.wikipedia.org/wiki/Blackstone%27s_ratio https://en.wikipedia.org/wiki/Blackstone%27s_ratio
neilv 3y ago
The quoted sentence I objected to was the sloppy writing, conflating of "suspected" with "caught". I think "caught" would be read by most readers as implying guilt. Suspected at 1:100, certainly. Quietly confronted and questioned at 1:100, sure. Convicted at 1:100, absolutely not. I'm not sure this is the best analogy, but imagine that, for every 100 people in society, 1 of them is wrongly convicted, at random. Further imagine that this lottery happens repeatedly, so that even more than 1 in 100 end up getting hit. That sounds like a miserable society in which to live.
- sokoloff 3y ago
  Headline: "80 million people worldwide caught cheating" just in the first round.
robochat 3y ago
Actually, I can't seem to get the maths to work. Isn't it just the Binomial distribution? Each question has 5 options so the probability of 'success' is 1/5 and so to get 19 questions 'specifically wrong' out of the 45 planted questions by chance is just (1/5)*19 * (1-1/5)*26 * 45!/(19!26!) = 1 in 2588 but in the article it is 1:100. What am I doing wrong ?
- probably_wrong 3y ago
  Here's my best guess: 19 is the cutoff point for a binomial test [1] where the probability of at least that many answers to match those in the honeypot test goes below 0.01. But this holds only if you assume p=0.25. Why would you use 0.25 instead of 0.2? I guess it would make sense if you only looked at wrong answers - that is, you wouldn't be asking "what's the probability of your answers matching those on the fake test" but rather "what's the probability of your wrong answers being wrong because you used the fake test". Since you are only looking at wrong answers, your probability is 1 in 4 instead of 1 in 5. [1] At least, according to this calculator: https://www.socscistatistics.com/tests/binomial/default2.aspx https://www.socscistatistics.com/tests/binomial/default2.asp...
kmod 3y ago
"1% false positive rate => 99% confidence about a positive signal" is a really intuitive step to take, but you should be careful because it turns out to be horribly wrong in general: https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy (Perhaps not in this case but in general this can get you into trouble) Though personally I'd say that it's quite likely that the p=0.01 threshold falsely flagged a non-cheater, which to me says it is too lax.

by ajross 3y ago

sokoloff 3y ago

> I'll bet in most jurisdictions 100:1 odds count as "beyond a reasonable doubt" for jury instructions in criminal trials. IANAL, but I sure hope that 100:1 chance, standing alone, doesn't meet the bar for "beyond a reasonable doubt". To me, a 1% chance of an accusation being false is definitionally "reasonable doubt".

Wowfunhappy 3y ago
...I have no idea where I heard this, but for some reason I was under the impression "beyond a reasonable doubt" meant ~95% confident. (Whereas "preponderance of evidence" meant >50% confident.)
ajross 3y ago
Per Wikipedia, the original framing is Blackstone's ratio from the enlightenment, where he used 10:1. Real courts don't put numbers on this, of course, but that's the thinking that produced the philosophy. Basically: no, given the level of innaccuracy we already know to be present, I think it's very clear that a sincerely-believed 99% confidence will send someone to jail pretty much anywhere in the US. https://en.wikipedia.org/wiki/Reasonable_doubt https://en.wikipedia.org/wiki/Reasonable_doubt https://en.wikipedia.org/wiki/Blackstone%27s_ratio https://en.wikipedia.org/wiki/Blackstone%27s_ratio

neilv 3y ago

The quoted sentence I objected to was the sloppy writing, conflating of "suspected" with "caught". I think "caught" would be read by most readers as implying guilt. Suspected at 1:100, certainly. Quietly confronted and questioned at 1:100, sure. Convicted at 1:100, absolutely not. I'm not sure this is the best analogy, but imagine that, for every 100 people in society, 1 of them is wrongly convicted, at random. Further imagine that this lottery happens repeatedly, so that even more than 1 in 100 end up getting hit. That sounds like a miserable society in which to live.

sokoloff 3y ago
Headline: "80 million people worldwide caught cheating" just in the first round.

robochat 3y ago

Actually, I can't seem to get the maths to work. Isn't it just the Binomial distribution? Each question has 5 options so the probability of 'success' is 1/5 and so to get 19 questions 'specifically wrong' out of the 45 planted questions by chance is just (1/5)*19 * (1-1/5)*26 * 45!/(19!26!) = 1 in 2588 but in the article it is 1:100. What am I doing wrong ?

probably_wrong 3y ago
Here's my best guess: 19 is the cutoff point for a binomial test [1] where the probability of at least that many answers to match those in the honeypot test goes below 0.01. But this holds only if you assume p=0.25. Why would you use 0.25 instead of 0.2? I guess it would make sense if you only looked at wrong answers - that is, you wouldn't be asking "what's the probability of your answers matching those on the fake test" but rather "what's the probability of your wrong answers being wrong because you used the fake test". Since you are only looking at wrong answers, your probability is 1 in 4 instead of 1 in 5. [1] At least, according to this calculator: https://www.socscistatistics.com/tests/binomial/default2.aspx https://www.socscistatistics.com/tests/binomial/default2.asp...

kmod 3y ago

"1% false positive rate => 99% confidence about a positive signal" is a really intuitive step to take, but you should be careful because it turns out to be horribly wrong in general: https://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy (Perhaps not in this case but in general this can get you into trouble) Though personally I'd say that it's quite likely that the p=0.01 threshold falsely flagged a non-cheater, which to me says it is too lax.