4 ms·
Very nice hack and analysis thereof. I do take issue with this statement, though, in particular the part highlighted (by me): > "it’s impossible for your oppo
by FabHK 9y ago
Very nice hack and analysis thereof.
I do take issue with this statement, though, in particular the part highlighted (by me):
> "it’s impossible for your opponents to scientifically prove that you’re cheating. This impossibility is due to methodological defects in the current state of scientific practice, and we’ll highlight some ongoing work to fix these defects."
The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically.
Of course, there are problems with the current state of scientific practice and the mindless application of statistics (particularly the 5% significance level). The replicability crisis in psychology comes to mind.
However, I think statistics is unfairly maligned here. Statistics mostly delivers on its promise, it just promises less than what some people think.
The world is messy, and statistics can help us to make sense of it, but it can only do so much. In particular, it's not magic. You need many observations to make statistical statements, and more observations to make more precise statements (or statements at a higher significance level). That's intrinsic to how the world is. It's not a problem with "methodological defects in the current state of scientific practice". And there aren't simple solutions "to fix these defects".
(Having said that, the article is great. Many proposals (some of them mentioned in the article), such as a standard 0.5% significance level, more Bayes, awareness of power in addition to significance, avoiding p-hacking, etc., are important and useful steps. But they still won't allow your opponent to prove that you were cheating :-)
- jrochkind1 9y agoI think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data to determine validity yet". Among other things that different tests would reveal. p-value is not the only way to analyze your results statistically. p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice. I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed. (But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!) There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association. http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503 http://www.nature.com/news/statisticians-issue-warning-over-... https://www.sciencedirect.com/science/article/pii/S1063458412007789 https://www.sciencedirect.com/science/article/pii/S106345841... http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomerror/EP713_RandomError7.html http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer... I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing. It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.
- chimeracoder 9y ago> perhaps saying in some cases "you don't have enough data to determine validity yet Contrary to popular belief, that's actually exactly what a p-value is. A p-value determines whether your sample size is large enough to reject the null hypothesis, given the observed variance in the sample data. Introductory statistics classes don't really explain this properly, because it would require knowledge of continuity theory that most don't have at that stage, but that's really what a p-value is already telling you.
- jrochkind1 9y agoI admit I'm not a statistician (or someone that regularly does quantitative research) and don't really understand what a p-value says (without refreshing myself on it for reading for 30 minutes each time heh). So thanks for correction. But I think many of the researchers applying it don't either, heh. The real point is just that there are other statistical tests that might be appropriate here, and perhaps would be more reliable at catching this. And that there's actually widespread agreement among experts that p-value testing is widely wrongly used. > large enough to reject the null hypothesis Or at any rate, that there's a calculated max 5% chance of the null hypothesis being true, if the assumptions/pre-requisites of the p-value test were actually met. Which is still actually kinda large, 5%, depending. And as I try to think about and confuse myself, may actually be entirely the wrong test as applied here, not sure.
- chimeracoder 9y ago> But I think many of the researchers applying it don't either, heh. As a statistician myself, I'd agree with that statement. I've actually said previously - and only half-joking - that p-values should be banned from research journals. The problem with p-values is that they don't mean what people generally assume they do, and because they look so close to what people want them to mean ("the probability that my conclusions are wrong, given my assumptions and data"), it's very easy to project spurious meaning onto them. In reality, p-values actually provide very little information, and the information they do provide is generally not of relevance. But they're so commonly-used that it's very hard to convince people to use more sophisticated techniques for reporting and modeling information.
- godelski 9y ago> The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically. Not if you take enough samples. Just have to do it enough to reduce the noise. But to your main point, yeah, in a game of Catan you can just get unlucky. And I think a fair amount of people claim that Catan dice aren't fair in the first place. Also, did they test the unweighted dice? To understand if there was an inherent and unknown bias in their dice to start with? If they did, I missed that part, and they just assumed their dice were fair to begin with.
- cortesoft 9y agoI don’t think you are disagreeing with the person you are responding to. They are trying to say the problem isn’t necessarily the ‘scientific process’, it is that if you cheat rarely enough you might not give your opponent enough samples to prove it. Your point that if you had enough samples you could prove it is actually agreeing with them.
- godelski 9y agoOh, I'm definitely agreeing with the person. I was just being a little snarky at that one statement. I am also trying to ask a question that is extremely relevant to the author's "study". "Are the original dice fair?" That was never asked, it was assumed. It is a pretty hefty assumption too. I think everyone that has played a game of Catan has, at some point, questioned the fairness of the dice. I'm not saying they are unfair, but because of manufacturing methods, it is quite possible to get a pair of dice with an ever so slight bias. ALWAYS ALWAYS ALWAYS find the bias in the instruments you are using to perform an experiment. THEY ALL HAVE THEM. NEVER NEVER NEVER assume your instruments are accurate without first verifying.
- LeifCarrotson 9y ago> I think everyone that has played a game of Catan has, at some point, questioned the fairness of the dice. This is why a lot of people play with a set of 36 cards (one 2, two 3's, three 4's...one 12). There really aren't that many rolls in a game, and there's a lot of variation that may result in a 5 never coming up, which adds more chance to the game than I like. Cards mean that while the order is random, you get each number in the deck eventually.
- cortesoft 9y agoYeah, the idea that being able to adjust the priors for a Bayesian analysis based on suspicious behavior, and then saying that proves cheating, is certainly not the case. The cheater is not going to agree with your assessment of their ‘suspicious’ behavior. You might as well just skip the analysis and say you proved they cheated because of their behavior.
- colechristensen 9y agoStatistics is fine, people who are bad at statistics are the problem. When it comes down to it, many "scientists" are rather poorly educated.
- verylongname 9y agoIndeed, statistics is fine, but I think the problem is larger than scientists being poorly educated. It looks to me like an awful lot of scientists aren't acting in good faith. I don't think that their conduct necessarily rises to the level of dishonesty, more like they aren't taking obvious steps to mitigate the potential for erroneous results.
- xapata 9y ago> aren't taking obvious steps Because none of their peers think those steps are obvious, and their peers determine their funding, status, etc.
- deleted 9y ago[deleted]
- savanaly 9y ago>The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically. I don't understand what you're saying or maybe I just want to go further. But I would say it's an ill-posed question whether someone's cheating in that sense. They could win a one in a million chance repeatedly over and over on their first try and it wouldn't be proof that they're cheating. It's not even necessarily probable. The probability that they're cheating depends not just on how good their results seem to be but also other things, primarily: on your priors about how likely it is that they would cheat. If a known cheater gets a string of mild good luck it is more indicative of cheating than if a known honest person does. Bayesian reasoning has to be used for this right?
- ineedasername 9y agoWell said. It doesn't reveal problems with current methodology, it reveals problems with its application, and understanding. Larger sample sizes would fix quite a bit. Studies, even "pilot" or "priming" studies, are not reliable at a .05 significance. Especially when most of the false positives will get published, but nearly none of the false negatives will see the light of day.
- philipwhiuk 9y agoMore practically, all dice are slightly loaded.