4 ms·
I guessed GPT2 each time, 200 times in a row and only found that GPT2 was correct 89/200 times, so about 45% was GPT2 for me.
by lelandbatey 6y ago
I guessed GPT2 each time, 200 times in a row and only found that GPT2 was correct 89/200 times, so about 45% was GPT2 for me.
- wnoise 6y agoIn [2]: scipy.stats.binom_test(89, 200, 0.5) Out[2]: 0.13736665086863936 Unusual to be this lopsided (1-in-7), but not crazy.
- zaik 6y agoReminder that the p-value of a test is NOT the probability of H0 being true, see [0]. It only shows that, if we assume a significance of 0.05 we cannot reject the hypothesis (in our case that 89/200 is the result of a binomial distribution with p=.5). [0] https://en.wikipedia.org/wiki/Misuse_of_p-values#Clarifications_about_p-values https://en.wikipedia.org/wiki/Misuse_of_p-values#Clarificati...
- wnoise 6y agoYes. That is what I said and how I interpreted it. If the split is even (H0 true), getting a result that lopsided is a 1-out-7 deal. It's rare that a p-value is what you want, but for answering "how unusual is this case", it's the exact right tool for the job.
- hntrader 6y agoIt only shows the probability of observing either that statistic, or something even more extreme, under the null distribution. The implication that we can then "reject the null hypothesis" is more parlance and heuristic than anything.