12 ms·

Using loaded dice to cheat at Settlers of Catan, and p-values to avoid suspicion

266 points by jackpirate 9y ago

cdancette 9y ago
This post explains in a very simple way the scientific method (p-values) to prove that a result is significant or not, and the problems with it. It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false.
- 9mit3t2m9h9a 9y ago
  5% false discovery rate is true if the apriori probability of each result is 50%. If a journal wants to publish only surprising results, and accepts p=0.05 with a good methodology as true, it will have higher rate of false claims, because the most interesting things are more surprising than a coin toss. And if you slice the data from a single experiment in 40 independent ways, your chances to get something with purely random significance p<0.05 are better than 50% for a single study…
  - cdancette 9y ago
    It doesn't "mean" that the a priori is 50%, I guess what you want to say is that if we consider the a priori probabilities are equal between the two hypothesis, then we can use the 5% p-value. But if the probability of the hypothesis is much less likely, we might need a p-value much lower to be sure. Edit: my comment doesn't mean much since you edited your first sentence.
- mehrdadn 9y ago
  > It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false. That would only be the case if scientists were robots who immediately published anything with a p-value up to 0.05. They're not, though. If they get clearly nonsensical results, they will obviously re-evaluate it. In other words, the p-value doesn't incorporate the fact that the experiment passed sanity checks in your own head (and the reviewers') before it was published. (And yes, there are bad actors in every field who game the system, but my point still stands.)
  - capnrefsmmat 9y ago
    I suspect this factor is balanced, or completely overruled, g the scientists who get p values greater than 0.05, decide that result doesn’t pass their sanity check (it clearly should be significant!) and collect more data or tweak the methods until it’s significant.
  - golergka 9y ago
    I think that if scientists were robots, results would be much better than we have today - robots don't care about careers and grants.
  - adrianN 9y ago
    From what I've heard on HN, scientists are actually robots who massage their data until they get a p-value <0.05 and then immediately publish.
    - nonbel 9y ago
      No, the publishing process takes a long time. Sometimes it could be years.
      - adrianN 9y ago
        Yeah, but in Russia they use nine women to produce a baby in only one month.
        nonbel 9y ago
        This sounds like some kind of comment about divisibility of the work to publish something. I don't get it though. After the bulk of the paper is written, I can easily proofread, typeset, etc everything myself in less than a week. Now get someone else to double check that. Lets say that is another week. After that the only thing is to get someone worthwhile to spend some time on your paper and point out anything confusing or erroneous. Granted, this could take a month or so of study. However, I never really saw that happen in practice. In reality you would be lucky to get people to glance over it one evening. So what is taking so long?
- nonbel 9y ago
  >"It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false." How so? I don't see any connection between significance level and % of false scientific results at all. If you assume the "null hypothesis" is always true, then 5% of the results should falsely say otherwise. Of course, this is if all the assumptions behind the math hold, no p-hacking, etc. However, that is like saying it is extremely rare for there to be a correlation between any two phenomena. We don't live in that universe. In our universe, correlations are extremely common: >"These armchair considerations are borne out by the finding that in psychological and sociological investigations involving very large numbers of subjects, it is regularly found that almost all correlations or differences between means are statistically significant. See, for example, the papers by Bakan [1] and Nunnally [8]. Data currently being analyzed by Dr. David Lykken and myself, derived from a huge sample of over 55,000 Minnesota high school seniors, reveal statistically significant relationships in 91% of pairwise associations among a congeries of 45 miscellaneous variables such as sex, birth order, religious preference, number of siblings, vocational choice, club membership, college choice, mother's education, dancing, interest in woodworking, liking for school, and the like. The 9% of non-significant associations are heavily concentrated among a small minority of variables having dubious reliability, or involving arbitrary groupings of non-homogeneous or nonmonotonic sub-categories. The majority of variables exhibited significant relationships with all but three of the others, often at a very high confidence level" -Theory testing in psychology and physics: A methodological paradox. http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
- emiliobumachar 9y ago
  Obligatory xkcd: https://www.xkcd.com/882/ https://www.xkcd.com/882/
- capnrefsmmat 9y ago
  This is not true — it’s actually worse. The 5% figure means that, when there is no signal to detect, we have a 5% chance of falsely claiming there is one. It does not say anything about the case when there is a signal and we do not detect it, which is known as the type II error rate. With reasonable assumptions about sample size and the fraction of times there really is a signal, you can find that the majority of published results are false: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... I have written an intuitive explanation of this, and a bunch more, in my book: https://www.statisticsdonewrong.com/p-value.html https://www.statisticsdonewrong.com/p-value.html
  - fnl 9y ago
    Great book, thanks for writing it!
9mit3t2m9h9a 9y ago
What does the calculation of advantage mean there? The advantage per one nice and one naughty roll (a pair of rolls) is multiplied by the total number of single rolls in a 4-player game. And I guess an honest player would pick the nice locations as often as naughty ones (there is no reason to actually prefer them). I don't know the rules, so I don't know if there are enough naughty locations to roll only on them without _that_ being suspicious.
frozenport 9y ago
Thats why we need other groups to replicate.
kd5bjo 9y ago
By this analysis, you can average between 5 and 15 more resources over the course of the game vs unloaded dice, and then goes on to show that there aren't enough dice rolls in a game to prove the dice are loaded. The problem here is that you don't get the expected number of resources in every game, and there's no analysis of the variance. I suspect that the result of insignificance is correct, in that this "cheat" provides such a slight advantage that it won't materially affect a single game. By the time you've played enough games to reliably use the advantage, your opponents will have seen enough die rolls to show their bias.
- pc86 9y ago
  > By the time you've played enough games to reliably use the advantage, your opponents will have seen enough die rolls to show their bias. I have a hard time believing that after 100 games someone would say "you know what, it appears to me that sixes have been rolled slightly more over the previous 100 games than I would expect" and an even harder time believing the next logical would be "you must have altered the dice!"
  - adrianN 9y ago
    Looks like you only play boardgames with casuals.
    - pc86 9y ago
      That must be it :)
    - logfromblammo 9y ago
      If you're OP, you keep a running mental tally of other player's resource cards, so it is immediately obvious which rolls have been coming up more often than chance, and which ones have been coming up less. There are 21 different ways to load 2 dice with one weighted face each. Here's a table showing which rolls are advantaged or disadvantaged, according to which faces are weighted. 02 03 04 05 06 07 08 09 10 11 12 1,1 < < < < < < > > > > > 1,2 < < < < < = > > > > > 1,3 < < < < < = > > > > > 1,4 < < < < < = > > < > > 1,5 < < = = < = > = = < > 1,6 < = = = = > = = = = < 2,2 = < < < = < = > > > = 2,3 = < < < < = < > > > = <<< GOOD 2,4 = < = < > = > > = > = 2,5 = = < = = > = = < = = 3,3 = = < = > < > = > = = 3,4 = = = = < > < = = = = <<< BEST 4,4 = = > = > < > = < = = 5,2* = = < = = > = = < = = 5,3 = > = > > = > < = < = 5,4 = > > > < = < < < < = <<< GOOD 5,5 = > > > = < = < < < < 6,1* < = = = = > = = = = < 6,2 > < = = > = < = = < < 6,3 > > < > > = < < < < < 6,4 > > > > > = < < < < < 6,5 > > > > > = < < < < < 6,6 > > > > > < < < < < < * duplicated to show symmetry So for this game, I'd probably weight 3 and 4, to advantage 7s over 6s and 8s, and build more on the 5s and 9s. Most players that have no knowledge of the load of the dice will prefer to have at least one settlement on a 6 or 8. The 2,3 and 5,4 pairs also disadvantage 6 and 8, with bias for higher or lower numbers.
rando444 9y ago
I really enjoy these fun thought experiments. The only thing I didn't see clarified though is whether the die was loaded using the water only once, or whether they re-loaded the die every night during the week they were testing.
- jackpirate 9y ago
  Hi, author here. I loaded the dice only once. The weighed 0.65 ounces before hand, and 0.75 ounces afterward. They kept that weight the whole week.
  - nonbel 9y ago
    So a suspicious player could detect the loaded dice simply by weighing them? That actually makes a lot of sense. If you look into actual dice/coin manufacturing they do not run tests like "roll a sample of dice a million times each, then check p-value". Instead they have a clever manufacturing process with stringent tolerances. If they run such tests at all it would be to find the upper bound on deviation from fairness over the lifetime of the dice (eg 100k rolls).
IncRnd 9y ago
A hypothesis was presented: using higher numbers will give 5-15 more resource cards over the course of a game. Then, conclusions were drawn that standard statistical techniques are deficient for not being able to detect the bias. Where are the results of the actual experiment, say of 10 games played using the biased strategy, against a control opponent? Perhaps, standard statistical techniques are correct. Without empirical tests of the hypothesis, where is the science? Perhaps, the dice are only loaded for a short time, in which time all the skew occurred. Without testing of the hypothesis, it is impossible to know.
deleted 9y ago
[deleted]
nmg 9y ago
Einstein: "God does not play dice." Hawking: "God definitely plays dice, but He sometimes confuses us by throwing them where they can't be seen" Izbicki: "And if the dice are loaded, we can't prove it." There is some plain & profound truth in this article underlying some pretty cool math.
FabHK 9y ago
Very nice hack and analysis thereof. I do take issue with this statement, though, in particular the part highlighted (by me): > "it’s impossible for your opponents to scientifically prove that you’re cheating. This impossibility is due to methodological defects in the current state of scientific practice, and we’ll highlight some ongoing work to fix these defects." The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically. Of course, there are problems with the current state of scientific practice and the mindless application of statistics (particularly the 5% significance level). The replicability crisis in psychology comes to mind. However, I think statistics is unfairly maligned here. Statistics mostly delivers on its promise, it just promises less than what some people think. The world is messy, and statistics can help us to make sense of it, but it can only do so much. In particular, it's not magic. You need many observations to make statistical statements, and more observations to make more precise statements (or statements at a higher significance level). That's intrinsic to how the world is. It's not a problem with "methodological defects in the current state of scientific practice". And there aren't simple solutions "to fix these defects". (Having said that, the article is great. Many proposals (some of them mentioned in the article), such as a standard 0.5% significance level, more Bayes, awareness of power in addition to significance, avoiding p-hacking, etc., are important and useful steps. But they still won't allow your opponent to prove that you were cheating :-)
- jrochkind1 9y ago
  I think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data to determine validity yet". Among other things that different tests would reveal. p-value is not the only way to analyze your results statistically. p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice. I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed. (But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!) There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association. http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503 http://www.nature.com/news/statisticians-issue-warning-over-... https://www.sciencedirect.com/science/article/pii/S1063458412007789 https://www.sciencedirect.com/science/article/pii/S106345841... http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomerror/EP713_RandomError7.html http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer... I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing. It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.
TimMurnaghan 9y ago
The other clue players can use to detect cheating is your behaviour. They can notice that you are favouring the higher numbers. This is also what can catch people in gambling games. The betting patterns of card counters are noticeable - and the eudaemonic folks who found how to predict tilted roulette tables bet on sectors of the wheel - which again is a distinctive pattern. So it's pretty much always those behavioural cues that other people will use to suspect cheating rather than the p-value.
- toblender 9y ago
  If I'm playing with someone that is making AK47's into utensils, I would think twice about raising my concerns regarding the loaded dice. Hmm the dice are behaving funny, but that's fine, everything is just fine... https://izbicki.me/blog/turning-an-ak-47-into-a-serving-ladle.html https://izbicki.me/blog/turning-an-ak-47-into-a-serving-ladl...
  - logfromblammo 9y ago
    Ah yes, the "let the Wookiee win" strategy. You lose all the games, but you do get to keep playing with both your arms. I tried strategically throwing games a long time ago. It backfired after I conceded defeat, and rather than bask in victory, the "winning" player gave me a narrowed-eyes suspicious look and flipped over my hidden cards. They made it obvious that I would have won already if I had made the optimal play on my previous turn. She was pissed, because even though she won the game, she didn't beat me, because I wasn't even playing the same game as everyone else. We have some seriously competitive people in my family, and we had a good argument about whether one was required to make a guaranteed winning move if it was possible to do so.
- xapata 9y ago
  > betting patterns of card counters Large changes in bet-size are an obvious indicator, so card counters must work in teams. One is the counter, the other gambler is the whale who bets large when the count is favorable. Unfortunately, the casino has a simple defense: table limits. The whale must sit at the high-roller table, the counter must sit elsewhere. Count cards as much as you like, but you'll never make money at a modern casino ("you" being the aggregate of all card counters, not an individual).
tzahola 9y ago
Our set came with plastic dice. Crap!
- andresgottlieb 9y ago
  FYI: https://www.wikihow.com/Load-Dice https://www.wikihow.com/Load-Dice
SurrealSoul 9y ago
Now I have something else to blame besides luck when I lose
empath75 9y ago
> To measure this bias, my wife and I spent the next 7 days rolling dice while eating dinner. Some people have very different relationships with their spouses than I do.
- colemannugent 9y ago
  I thought that was weird as well. They only had to do like 100 rolls to have enough data to draw conclusions from.
  - cdancette 9y ago
    100 rolls is not enough. If you keep the same ratio, you can compute the p-value and I'm sure it will be way over 0.05.
zejn 9y ago
> posted on 2018-12-14 OP cheats at time too, posting this a full year ahead of schedule. :)
dep_b 9y ago
The dice should have been tested before changing them. They might haven't been perfect dice to begin with. Also the lack of automatization is disturbing. I find it shocking that no robot arm or pattern recognition was used to do the tests. Poor wife indeed! "If only my husband was a bigger geek" she must've thought.
- Simon_says 9y ago
  He got his wife to do it. That's the original automation.
pougetj 9y ago
Great article! One nitpick however in the informal definition of a p-value: the p-value is the probability of getting results similar or more extreme to the results we observed if the dice are not biased.
tzs 9y ago
I wonder if there are any good, simple procedures that could be applied during a game to mitigate against biased dice (without giving up on physical dice)? For games that only use one die, perhaps something like generating a random number in 1 <= r <= 6n[1], using some procedure that all the players have input to so they can all agree to trust the number, and then shifting all die roll results by that number? So if the number were 3 and the die came up 2, it would count as 5. That would not remove the bias from the die, but it would move it to a different number. For games where some numbers are consistently good for you and some are consistently bad for you that would be enough to make it so that biasing the die does not work in the long run--some games it would end up in your favor and some it would end up against you. For games with two dice, that might not work as well. If a single random number were picked and used to shift both die results, it would shift the bias just like with the single die game. However, the dice will still be biased to come up matching, and in some games matching numbers on the two is significant. One might try to address this by generating two random numbers at the start of the game, one for each die, for the shifts. That would have bias against getting a match on the two dice, so would provide an advantage in games where a match is bad. [1] Generalizing to n-sided dice is left as an exercise for the reader.
- dkarl 9y ago
  Presumably the person who supplies the dice knows how they're loaded. With a shift, they might still be able to use that information to their advantage. Instead, you could use the agreed-upon randomness at the beginning of the game to apply numeric labels to the faces of the dice. As long as the labels themselves aren't loaded... and the person who supplied the dice would still have information about the distribution.
- maksimum 9y ago
  I think you raise a very interesting question, and I like your initial solution. I feel like a game is fair if all players know that a die is biased, but no player has an advantage in predicting the outcome. To achieve that: for each die, for each roll have the players agree on a random permutation of the observed faces to actual faces for that roll. As a result even if all players know that 6 will land with probability 1 in the original throw, after permuting each side has probability 1/6. The agreement algorithm: have the parties each propose a permutation, and compose the permutations together in order to get a final "unbiased" permutation. I think if there is at least one honest party, this will work. Another approach is to let everyone sample the dies before playing until they are satisfied, and let everyone adjust their strategy accordingly.
simonebrunozzi 9y ago
I really like Settlers of Catan, but I have never found a nicely designed online site to play it. Does any of you know of anything? Also surprised that there's no "free" + "open source" version of it, like FreeCiv or FreeCol or FreeOrion.
- bbojan 9y ago
  What about Pioneers (http://pio.sourceforge.net/ http://pio.sourceforge.net/)?
- prvnsmpth 9y ago
  http://www.playcatan.com http://www.playcatan.com had a pretty good client... too bad it's not open for new registrations any more. They have Catan Universe now instead - not sure how good it is, though.
- ovrdrv3 9y ago
  Tabletop Simulator is good, once you get used to it and learn a few keyboard shortcuts! Only downside is it's usually $30 and I am not 100 percent sure if the catan ports are safe to forever remain? If anyone has insight on the legalities of Tabletop Simulator I am curious.
  - bllguo 9y ago
    In the US it's $20, with discounts to $10 (look for the upcoming Steam sale). I am not qualified to speak on legality but a cursory search suggests that the catan ports are technically not safe... still, there are many versions in the workshop as catan is such a popular game. It's hard to imagine that they all get pulled without replacement.
    - ovrdrv3 9y ago
      Thanks for the reply. I hope they continue on there! >In the US it's $20, with discounts to $10 (look for the upcoming Steam sale). Thanks, looks like it will finally be time to gift it to a friend!
- 9y ago
maherbeg 9y ago
A nice mitigation technique is for everyone to bring their own dice, put them into a hat and require random selection of two of them per turn. Of course if you suspect your friends are cheating you have bigger problems.
- maksimum 9y ago
  This solves the problem by reducing it to a trusted random number generator (the hat). At that point you may as well skip a step and draw individual outcomes from the hat.
  - deleted 9y ago
    [deleted]
jps359 9y ago
Cool analysis. Doesn't every player use the same die for a game like Catan?
- chalence 9y ago
  Yes, but presumably only you would know the bias of the dice and would exploit it by preferentially building next to higher numbered spaces.
andrepd 9y ago
I don't get this. >It’s impossible for your opponents to scientifically prove that you’re cheating. Then two paragraphs later you scientifically prove you were cheating (simply roll dice thousands of times).
- burkaman 9y ago
  > while playing a game. Your opponents can't prove cheating based on the game alone.
  - godelski 9y ago
    Which is kind of a dumb thing to say. They even note that a typical Catan game only has 60 rolls. You could have many suspicious looking rolls that would result from fair dice. The fact is that 60 rolls isn't going to be enough even if you were cheating a lot more than what was shown in this post.
- cortesoft 9y ago
  Because you don’t roll dice a thousand times during the game.
darepublic 9y ago
I can finally beat my in-laws at this game this holiday season!
- logfromblammo 9y ago
  Skilled players can beat unskilled players regardless of the bias of the dice. It doesn't matter if you get 5-15 more brick cards if it's an ore game. And how do you keep the other players from building on the bias-favored numbers? If you're going to cheat, you're better off sleeving your most critical resource cards, to protect them from 7-rolls, soldiers, and monopoly cards taking them from your hand. You unsleeve and swap when you get a favorable resource roll, and hope nobody is OP enough to keep a running total of all opponent resources in their head.
  - rockostrich 9y ago
    The problem is, how do you know those numbers are bias-favored if your sample size is a few dice rolls? And if there is a bias in those few dice rolls, there's no guarantee it will continue. Sure, a more skilled player will always beat a much less skilled player, but the randomness of the dice certainly closes the skill gap. It's much different than a game like Agricola or Dominion where you're playing 100% against what the other people do and not relying on the outcomes of 4 dice rolls to determine what you want to do on your turn.
    - logfromblammo 9y ago
      The randomness of the board set up partially cancels the bias you introduce into the dice. There are 20 different possible numeric token layouts, and each of those has trillions of different possible tile combinations. You really have no practical way of knowing in advance which resource tiles will be under the number tokens you have biased for by loading the dice. So cheating in this fashion doesn't really provide that much of an advantage over the other players. Since the intentional cheating is scientifically indistinguishable from randomness, all good players will already be able to compensate for unfavorable variations in die rolls.
logfromblammo 9y ago
I'd be interested to see the results by loading the 1-2 edge of the die, or even the 1-2-3 corner. The former should result in more 5s and 6s, and the latter more 4s, 5s, and 6s. And since the weight would be not centered on a face, would make the die more likely to tumble over to a preferred face if the die had to shed its sideways momentum. The unbalanced die would prefer to tumble or skip when a non-preferred face shows, and slide or rock when a preferred face shows. The problem, of course, is that there is no way to prevent your opponent from accidentally benefiting from your cheating. If you have biased the dice to favor 8s over 6s, they might just plop down on an 8 before you can place your settlements. You would have to use two sets of loaded dice, one biased high, and the other biased low, and swap them out after first settlements are placed. Also, in my experience, the other players will still accuse you of cheating whether they can prove it scientifically or not, because they prefer to believe that you are better than they are at being sneaky and underhanded than better at honest strategy. For instance, Clue (aka Cluedo) is a game where one skilled player can repeatedly curbstomp lesser skilled players. If you do, they then grab your player sheet to look at all the strange and indecipherable symbols you have on it that are not simple Xs, and they accuse you of cheating. You can either explain the mathematical advantage of the additional information you record, lose all future advantage against those players, and get accused of "breaking the game", or you can remain silent and still not be able to play it again because you're a "cheater". Besides that, what self-respecting board gamer doesn't just leave the crappy wooden dice in the box, and use the good dice from their dice bag? I like this analysis, though. It reminds me of the guy who designed a one-sided die.
jimrandomh 9y ago
This is a neat project! The statistical test used could be a lot more sensitive, by taking advantage of information about the physics of biased dice. They test whether 6 is rolled more often than it should be relative to the other 5 numbers, but they should instead look at whether it's rolled more often relative to 1. Since these are on opposite sides of the die, and unfairness in 6-sided dice comes from uneven distribution of weight, these go together; you get to combine the information from too-many 6s and too-few 1s and get a test that needs only about half as many rolls to reach a given confidence level.
- jackpirate 9y ago
  I'm the author, and I totally agree! Originally, I was going to point out other null hypotheses that we could try to reject. This has the advantage that some of them (like the one you propose) model the physics. And it also has the disadvantage that if we propose too many hypotheses to test, then we will be more likely to get false positives. But the article was already too long, and this is a HUGE can of worms to open.
punnerud 9y ago
Could it be that your dice was biased to begin with? Or have you already rolled them ~5000 times before you placed them in water?
- askvictor 9y ago
  I was wondering this; is a control unnecessary here as we assume an equal distribution, or is that assumption false?
  - punnerud 9y ago
    I think it’s false because every mechanical device have some variation, but is it so small it can don’t “affect” the result?
smallnamespace 9y ago
Note that if your only goal was to test whether the dice was loaded, using a test against only one outcome (in this case, only 6s) is not the best way. Use either a Kolmogorov-Smirnov test[1] or the Anderson-Darling test[2]. The intuition is that these tests are more powerful because they take the difference between the entire empirical distribution minus the expected probability mass distribution. You're using 'all the numbers' simultaneously to check for cheating. Funnily enough, I first learned about these tests a nearly a decade ago, precisely because I wanted to know whether Settlers dice were loaded. [1] https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_tes... [2] https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test
- FabHK 9y ago
  I think for this case (with 6 discrete outcomes) you could just use Pearson's Chi-Square test. What you describe is general enough for continuous distributions. https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test
hinkley 9y ago
My old boss was into Settlers, and they stayed late to play some games a few times. I got invited once and weird stuff happened. Somehow I ended up next to both 6's on the map. As near as I can tell, one of the dice we used would not land on 4. Whatever was going on, the number of 6's and 9's rolled that night were abnormally high. By the time people figured out we were rolling as many 6's as we had 7's and 8's combined, I already had a town on the coast and pretty much steamrolled everybody with sheep and wheat. Really the game wasn't that fun at that point, so I just tried to end it as fast as possible. When you're playing an RPG, everybody seems to gravitate toward their 'lucky dice' which are most likely defective in the right way. But if you're playing a game with others you probably want the fairest dice you can find. I remember years ago seeing a sales video from some retired aerospace engineer that was making geometrically perfect dice. He'd worked out the resins so they cured uniformly. He'd stack his dice next to some random set and point out how the other guy's stack curved to one side while his was perfectly straight. Probably didn't sell a lot of those to DND players, except perhaps GMs.
- vanderZwan 9y ago
  > Probably didn't sell a lot of those to DND players, except perhaps GMs. GM's don't have to show their dice rolls and can overrule the outcome if it fits their storyline better, so the rolling is mostly ceremonial anyway.

266 points by jackpirate 9y ago

This post explains in a very simple way the scientific method (p-values) to prove that a result is significant or not, and the problems with it. It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false.

9mit3t2m9h9a 9y ago
5% false discovery rate is true if the apriori probability of each result is 50%. If a journal wants to publish only surprising results, and accepts p=0.05 with a good methodology as true, it will have higher rate of false claims, because the most interesting things are more surprising than a coin toss. And if you slice the data from a single experiment in 40 independent ways, your chances to get something with purely random significance p<0.05 are better than 50% for a single study…
- cdancette 9y ago
  It doesn't "mean" that the a priori is 50%, I guess what you want to say is that if we consider the a priori probabilities are equal between the two hypothesis, then we can use the 5% p-value. But if the probability of the hypothesis is much less likely, we might need a p-value much lower to be sure. Edit: my comment doesn't mean much since you edited your first sentence.
mehrdadn 9y ago
> It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false. That would only be the case if scientists were robots who immediately published anything with a p-value up to 0.05. They're not, though. If they get clearly nonsensical results, they will obviously re-evaluate it. In other words, the p-value doesn't incorporate the fact that the experiment passed sanity checks in your own head (and the reviewers') before it was published. (And yes, there are bad actors in every field who game the system, but my point still stands.)
- capnrefsmmat 9y ago
  I suspect this factor is balanced, or completely overruled, g the scientists who get p values greater than 0.05, decide that result doesn’t pass their sanity check (it clearly should be significant!) and collect more data or tweak the methods until it’s significant.
- golergka 9y ago
  I think that if scientists were robots, results would be much better than we have today - robots don't care about careers and grants.
- adrianN 9y ago
  From what I've heard on HN, scientists are actually robots who massage their data until they get a p-value <0.05 and then immediately publish.
  - nonbel 9y ago
    No, the publishing process takes a long time. Sometimes it could be years.
    - adrianN 9y ago
      Yeah, but in Russia they use nine women to produce a baby in only one month.
      - nonbel 9y ago
        This sounds like some kind of comment about divisibility of the work to publish something. I don't get it though. After the bulk of the paper is written, I can easily proofread, typeset, etc everything myself in less than a week. Now get someone else to double check that. Lets say that is another week. After that the only thing is to get someone worthwhile to spend some time on your paper and point out anything confusing or erroneous. Granted, this could take a month or so of study. However, I never really saw that happen in practice. In reality you would be lucky to get people to glance over it one evening. So what is taking so long?
nonbel 9y ago
>"It's crazy that by using a p-value of 0.05, it means that 5% of all scientific results might be false." How so? I don't see any connection between significance level and % of false scientific results at all. If you assume the "null hypothesis" is always true, then 5% of the results should falsely say otherwise. Of course, this is if all the assumptions behind the math hold, no p-hacking, etc. However, that is like saying it is extremely rare for there to be a correlation between any two phenomena. We don't live in that universe. In our universe, correlations are extremely common: >"These armchair considerations are borne out by the finding that in psychological and sociological investigations involving very large numbers of subjects, it is regularly found that almost all correlations or differences between means are statistically significant. See, for example, the papers by Bakan [1] and Nunnally [8]. Data currently being analyzed by Dr. David Lykken and myself, derived from a huge sample of over 55,000 Minnesota high school seniors, reveal statistically significant relationships in 91% of pairwise associations among a congeries of 45 miscellaneous variables such as sex, birth order, religious preference, number of siblings, vocational choice, club membership, college choice, mother's education, dancing, interest in woodworking, liking for school, and the like. The 9% of non-significant associations are heavily concentrated among a small minority of variables having dubious reliability, or involving arbitrary groupings of non-homogeneous or nonmonotonic sub-categories. The majority of variables exhibited significant relationships with all but three of the others, often at a very high confidence level" -Theory testing in psychology and physics: A methodological paradox. http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
emiliobumachar 9y ago
Obligatory xkcd: https://www.xkcd.com/882/ https://www.xkcd.com/882/
capnrefsmmat 9y ago
This is not true — it’s actually worse. The 5% figure means that, when there is no signal to detect, we have a 5% chance of falsely claiming there is one. It does not say anything about the case when there is a signal and we do not detect it, which is known as the type II error rate. With reasonable assumptions about sample size and the fraction of times there really is a signal, you can find that the majority of published results are false: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... I have written an intuitive explanation of this, and a bunch more, in my book: https://www.statisticsdonewrong.com/p-value.html https://www.statisticsdonewrong.com/p-value.html
- fnl 9y ago
  Great book, thanks for writing it!

9mit3t2m9h9a 9y ago

What does the calculation of advantage mean there? The advantage per one nice and one naughty roll (a pair of rolls) is multiplied by the total number of single rolls in a 4-player game. And I guess an honest player would pick the nice locations as often as naughty ones (there is no reason to actually prefer them). I don't know the rules, so I don't know if there are enough naughty locations to roll only on them without _that_ being suspicious.

frozenport 9y ago

Thats why we need other groups to replicate.

kd5bjo 9y ago

By this analysis, you can average between 5 and 15 more resources over the course of the game vs unloaded dice, and then goes on to show that there aren't enough dice rolls in a game to prove the dice are loaded. The problem here is that you don't get the expected number of resources in every game, and there's no analysis of the variance. I suspect that the result of insignificance is correct, in that this "cheat" provides such a slight advantage that it won't materially affect a single game. By the time you've played enough games to reliably use the advantage, your opponents will have seen enough die rolls to show their bias.

pc86 9y ago
> By the time you've played enough games to reliably use the advantage, your opponents will have seen enough die rolls to show their bias. I have a hard time believing that after 100 games someone would say "you know what, it appears to me that sixes have been rolled slightly more over the previous 100 games than I would expect" and an even harder time believing the next logical would be "you must have altered the dice!"
- adrianN 9y ago
  Looks like you only play boardgames with casuals.
  - pc86 9y ago
    That must be it :)
  - logfromblammo 9y ago
    If you're OP, you keep a running mental tally of other player's resource cards, so it is immediately obvious which rolls have been coming up more often than chance, and which ones have been coming up less. There are 21 different ways to load 2 dice with one weighted face each. Here's a table showing which rolls are advantaged or disadvantaged, according to which faces are weighted. 02 03 04 05 06 07 08 09 10 11 12 1,1 < < < < < < > > > > > 1,2 < < < < < = > > > > > 1,3 < < < < < = > > > > > 1,4 < < < < < = > > < > > 1,5 < < = = < = > = = < > 1,6 < = = = = > = = = = < 2,2 = < < < = < = > > > = 2,3 = < < < < = < > > > = <<< GOOD 2,4 = < = < > = > > = > = 2,5 = = < = = > = = < = = 3,3 = = < = > < > = > = = 3,4 = = = = < > < = = = = <<< BEST 4,4 = = > = > < > = < = = 5,2* = = < = = > = = < = = 5,3 = > = > > = > < = < = 5,4 = > > > < = < < < < = <<< GOOD 5,5 = > > > = < = < < < < 6,1* < = = = = > = = = = < 6,2 > < = = > = < = = < < 6,3 > > < > > = < < < < < 6,4 > > > > > = < < < < < 6,5 > > > > > = < < < < < 6,6 > > > > > < < < < < < * duplicated to show symmetry So for this game, I'd probably weight 3 and 4, to advantage 7s over 6s and 8s, and build more on the 5s and 9s. Most players that have no knowledge of the load of the dice will prefer to have at least one settlement on a 6 or 8. The 2,3 and 5,4 pairs also disadvantage 6 and 8, with bias for higher or lower numbers.

rando444 9y ago

I really enjoy these fun thought experiments. The only thing I didn't see clarified though is whether the die was loaded using the water only once, or whether they re-loaded the die every night during the week they were testing.

jackpirate 9y ago
Hi, author here. I loaded the dice only once. The weighed 0.65 ounces before hand, and 0.75 ounces afterward. They kept that weight the whole week.
- nonbel 9y ago
  So a suspicious player could detect the loaded dice simply by weighing them? That actually makes a lot of sense. If you look into actual dice/coin manufacturing they do not run tests like "roll a sample of dice a million times each, then check p-value". Instead they have a clever manufacturing process with stringent tolerances. If they run such tests at all it would be to find the upper bound on deviation from fairness over the lifetime of the dice (eg 100k rolls).

IncRnd 9y ago

A hypothesis was presented: using higher numbers will give 5-15 more resource cards over the course of a game. Then, conclusions were drawn that standard statistical techniques are deficient for not being able to detect the bias. Where are the results of the actual experiment, say of 10 games played using the biased strategy, against a control opponent? Perhaps, standard statistical techniques are correct. Without empirical tests of the hypothesis, where is the science? Perhaps, the dice are only loaded for a short time, in which time all the skew occurred. Without testing of the hypothesis, it is impossible to know.

deleted 9y ago

[deleted]

nmg 9y ago

Einstein: "God does not play dice." Hawking: "God definitely plays dice, but He sometimes confuses us by throwing them where they can't be seen" Izbicki: "And if the dice are loaded, we can't prove it." There is some plain & profound truth in this article underlying some pretty cool math.

FabHK 9y ago

Very nice hack and analysis thereof. I do take issue with this statement, though, in particular the part highlighted (by me): > "it’s impossible for your opponents to scientifically prove that you’re cheating. This impossibility is due to methodological defects in the current state of scientific practice, and we’ll highlight some ongoing work to fix these defects." The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically. Of course, there are problems with the current state of scientific practice and the mindless application of statistics (particularly the 5% significance level). The replicability crisis in psychology comes to mind. However, I think statistics is unfairly maligned here. Statistics mostly delivers on its promise, it just promises less than what some people think. The world is messy, and statistics can help us to make sense of it, but it can only do so much. In particular, it's not magic. You need many observations to make statistical statements, and more observations to make more precise statements (or statements at a higher significance level). That's intrinsic to how the world is. It's not a problem with "methodological defects in the current state of scientific practice". And there aren't simple solutions "to fix these defects". (Having said that, the article is great. Many proposals (some of them mentioned in the article), such as a standard 0.5% significance level, more Bayes, awareness of power in addition to significance, avoiding p-hacking, etc., are important and useful steps. But they still won't allow your opponent to prove that you were cheating :-)

jrochkind1 9y ago
I think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data to determine validity yet". Among other things that different tests would reveal. p-value is not the only way to analyze your results statistically. p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice. I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed. (But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!) There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association. http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503 http://www.nature.com/news/statisticians-issue-warning-over-... https://www.sciencedirect.com/science/article/pii/S1063458412007789 https://www.sciencedirect.com/science/article/pii/S106345841... http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://blogs.plos.org/publichealth/2015/06/24/p-values/ http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://www.ejves.com/article/S1078-5884(15)00585-7/pdf http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomerror/EP713_RandomError7.html http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer... I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing. It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.

TimMurnaghan 9y ago

The other clue players can use to detect cheating is your behaviour. They can notice that you are favouring the higher numbers. This is also what can catch people in gambling games. The betting patterns of card counters are noticeable - and the eudaemonic folks who found how to predict tilted roulette tables bet on sectors of the wheel - which again is a distinctive pattern. So it's pretty much always those behavioural cues that other people will use to suspect cheating rather than the p-value.

toblender 9y ago
If I'm playing with someone that is making AK47's into utensils, I would think twice about raising my concerns regarding the loaded dice. Hmm the dice are behaving funny, but that's fine, everything is just fine... https://izbicki.me/blog/turning-an-ak-47-into-a-serving-ladle.html https://izbicki.me/blog/turning-an-ak-47-into-a-serving-ladl...
- logfromblammo 9y ago
  Ah yes, the "let the Wookiee win" strategy. You lose all the games, but you do get to keep playing with both your arms. I tried strategically throwing games a long time ago. It backfired after I conceded defeat, and rather than bask in victory, the "winning" player gave me a narrowed-eyes suspicious look and flipped over my hidden cards. They made it obvious that I would have won already if I had made the optimal play on my previous turn. She was pissed, because even though she won the game, she didn't beat me, because I wasn't even playing the same game as everyone else. We have some seriously competitive people in my family, and we had a good argument about whether one was required to make a guaranteed winning move if it was possible to do so.
xapata 9y ago
> betting patterns of card counters Large changes in bet-size are an obvious indicator, so card counters must work in teams. One is the counter, the other gambler is the whale who bets large when the count is favorable. Unfortunately, the casino has a simple defense: table limits. The whale must sit at the high-roller table, the counter must sit elsewhere. Count cards as much as you like, but you'll never make money at a modern casino ("you" being the aggregate of all card counters, not an individual).

tzahola 9y ago

Our set came with plastic dice. Crap!

andresgottlieb 9y ago
FYI: https://www.wikihow.com/Load-Dice https://www.wikihow.com/Load-Dice

SurrealSoul 9y ago

Now I have something else to blame besides luck when I lose

empath75 9y ago

> To measure this bias, my wife and I spent the next 7 days rolling dice while eating dinner. Some people have very different relationships with their spouses than I do.

colemannugent 9y ago
I thought that was weird as well. They only had to do like 100 rolls to have enough data to draw conclusions from.
- cdancette 9y ago
  100 rolls is not enough. If you keep the same ratio, you can compute the p-value and I'm sure it will be way over 0.05.

zejn 9y ago

> posted on 2018-12-14 OP cheats at time too, posting this a full year ahead of schedule. :)

dep_b 9y ago

The dice should have been tested before changing them. They might haven't been perfect dice to begin with. Also the lack of automatization is disturbing. I find it shocking that no robot arm or pattern recognition was used to do the tests. Poor wife indeed! "If only my husband was a bigger geek" she must've thought.

Simon_says 9y ago
He got his wife to do it. That's the original automation.

pougetj 9y ago

Great article! One nitpick however in the informal definition of a p-value: the p-value is the probability of getting results similar or more extreme to the results we observed if the dice are not biased.

tzs 9y ago

I wonder if there are any good, simple procedures that could be applied during a game to mitigate against biased dice (without giving up on physical dice)? For games that only use one die, perhaps something like generating a random number in 1 <= r <= 6n[1], using some procedure that all the players have input to so they can all agree to trust the number, and then shifting all die roll results by that number? So if the number were 3 and the die came up 2, it would count as 5. That would not remove the bias from the die, but it would move it to a different number. For games where some numbers are consistently good for you and some are consistently bad for you that would be enough to make it so that biasing the die does not work in the long run--some games it would end up in your favor and some it would end up against you. For games with two dice, that might not work as well. If a single random number were picked and used to shift both die results, it would shift the bias just like with the single die game. However, the dice will still be biased to come up matching, and in some games matching numbers on the two is significant. One might try to address this by generating two random numbers at the start of the game, one for each die, for the shifts. That would have bias against getting a match on the two dice, so would provide an advantage in games where a match is bad. [1] Generalizing to n-sided dice is left as an exercise for the reader.

dkarl 9y ago
Presumably the person who supplies the dice knows how they're loaded. With a shift, they might still be able to use that information to their advantage. Instead, you could use the agreed-upon randomness at the beginning of the game to apply numeric labels to the faces of the dice. As long as the labels themselves aren't loaded... and the person who supplied the dice would still have information about the distribution.
maksimum 9y ago
I think you raise a very interesting question, and I like your initial solution. I feel like a game is fair if all players know that a die is biased, but no player has an advantage in predicting the outcome. To achieve that: for each die, for each roll have the players agree on a random permutation of the observed faces to actual faces for that roll. As a result even if all players know that 6 will land with probability 1 in the original throw, after permuting each side has probability 1/6. The agreement algorithm: have the parties each propose a permutation, and compose the permutations together in order to get a final "unbiased" permutation. I think if there is at least one honest party, this will work. Another approach is to let everyone sample the dies before playing until they are satisfied, and let everyone adjust their strategy accordingly.

simonebrunozzi 9y ago

I really like Settlers of Catan, but I have never found a nicely designed online site to play it. Does any of you know of anything? Also surprised that there's no "free" + "open source" version of it, like FreeCiv or FreeCol or FreeOrion.

bbojan 9y ago
What about Pioneers (http://pio.sourceforge.net/ http://pio.sourceforge.net/)?
prvnsmpth 9y ago
http://www.playcatan.com http://www.playcatan.com had a pretty good client... too bad it's not open for new registrations any more. They have Catan Universe now instead - not sure how good it is, though.
ovrdrv3 9y ago
Tabletop Simulator is good, once you get used to it and learn a few keyboard shortcuts! Only downside is it's usually $30 and I am not 100 percent sure if the catan ports are safe to forever remain? If anyone has insight on the legalities of Tabletop Simulator I am curious.
- bllguo 9y ago
  In the US it's $20, with discounts to $10 (look for the upcoming Steam sale). I am not qualified to speak on legality but a cursory search suggests that the catan ports are technically not safe... still, there are many versions in the workshop as catan is such a popular game. It's hard to imagine that they all get pulled without replacement.
  - ovrdrv3 9y ago
    Thanks for the reply. I hope they continue on there! >In the US it's $20, with discounts to $10 (look for the upcoming Steam sale). Thanks, looks like it will finally be time to gift it to a friend!
9y ago

maherbeg 9y ago

A nice mitigation technique is for everyone to bring their own dice, put them into a hat and require random selection of two of them per turn. Of course if you suspect your friends are cheating you have bigger problems.

maksimum 9y ago
This solves the problem by reducing it to a trusted random number generator (the hat). At that point you may as well skip a step and draw individual outcomes from the hat.
- deleted 9y ago
  [deleted]

jps359 9y ago

Cool analysis. Doesn't every player use the same die for a game like Catan?

chalence 9y ago
Yes, but presumably only you would know the bias of the dice and would exploit it by preferentially building next to higher numbered spaces.

andrepd 9y ago

I don't get this. >It’s impossible for your opponents to scientifically prove that you’re cheating. Then two paragraphs later you scientifically prove you were cheating (simply roll dice thousands of times).

burkaman 9y ago
> while playing a game. Your opponents can't prove cheating based on the game alone.
- godelski 9y ago
  Which is kind of a dumb thing to say. They even note that a typical Catan game only has 60 rolls. You could have many suspicious looking rolls that would result from fair dice. The fact is that 60 rolls isn't going to be enough even if you were cheating a lot more than what was shown in this post.
cortesoft 9y ago
Because you don’t roll dice a thousand times during the game.

darepublic 9y ago

I can finally beat my in-laws at this game this holiday season!

logfromblammo 9y ago
Skilled players can beat unskilled players regardless of the bias of the dice. It doesn't matter if you get 5-15 more brick cards if it's an ore game. And how do you keep the other players from building on the bias-favored numbers? If you're going to cheat, you're better off sleeving your most critical resource cards, to protect them from 7-rolls, soldiers, and monopoly cards taking them from your hand. You unsleeve and swap when you get a favorable resource roll, and hope nobody is OP enough to keep a running total of all opponent resources in their head.
- rockostrich 9y ago
  The problem is, how do you know those numbers are bias-favored if your sample size is a few dice rolls? And if there is a bias in those few dice rolls, there's no guarantee it will continue. Sure, a more skilled player will always beat a much less skilled player, but the randomness of the dice certainly closes the skill gap. It's much different than a game like Agricola or Dominion where you're playing 100% against what the other people do and not relying on the outcomes of 4 dice rolls to determine what you want to do on your turn.
  - logfromblammo 9y ago
    The randomness of the board set up partially cancels the bias you introduce into the dice. There are 20 different possible numeric token layouts, and each of those has trillions of different possible tile combinations. You really have no practical way of knowing in advance which resource tiles will be under the number tokens you have biased for by loading the dice. So cheating in this fashion doesn't really provide that much of an advantage over the other players. Since the intentional cheating is scientifically indistinguishable from randomness, all good players will already be able to compensate for unfavorable variations in die rolls.

logfromblammo 9y ago

I'd be interested to see the results by loading the 1-2 edge of the die, or even the 1-2-3 corner. The former should result in more 5s and 6s, and the latter more 4s, 5s, and 6s. And since the weight would be not centered on a face, would make the die more likely to tumble over to a preferred face if the die had to shed its sideways momentum. The unbalanced die would prefer to tumble or skip when a non-preferred face shows, and slide or rock when a preferred face shows. The problem, of course, is that there is no way to prevent your opponent from accidentally benefiting from your cheating. If you have biased the dice to favor 8s over 6s, they might just plop down on an 8 before you can place your settlements. You would have to use two sets of loaded dice, one biased high, and the other biased low, and swap them out after first settlements are placed. Also, in my experience, the other players will still accuse you of cheating whether they can prove it scientifically or not, because they prefer to believe that you are better than they are at being sneaky and underhanded than better at honest strategy. For instance, Clue (aka Cluedo) is a game where one skilled player can repeatedly curbstomp lesser skilled players. If you do, they then grab your player sheet to look at all the strange and indecipherable symbols you have on it that are not simple Xs, and they accuse you of cheating. You can either explain the mathematical advantage of the additional information you record, lose all future advantage against those players, and get accused of "breaking the game", or you can remain silent and still not be able to play it again because you're a "cheater". Besides that, what self-respecting board gamer doesn't just leave the crappy wooden dice in the box, and use the good dice from their dice bag? I like this analysis, though. It reminds me of the guy who designed a one-sided die.

jimrandomh 9y ago

This is a neat project! The statistical test used could be a lot more sensitive, by taking advantage of information about the physics of biased dice. They test whether 6 is rolled more often than it should be relative to the other 5 numbers, but they should instead look at whether it's rolled more often relative to 1. Since these are on opposite sides of the die, and unfairness in 6-sided dice comes from uneven distribution of weight, these go together; you get to combine the information from too-many 6s and too-few 1s and get a test that needs only about half as many rolls to reach a given confidence level.

jackpirate 9y ago
I'm the author, and I totally agree! Originally, I was going to point out other null hypotheses that we could try to reject. This has the advantage that some of them (like the one you propose) model the physics. And it also has the disadvantage that if we propose too many hypotheses to test, then we will be more likely to get false positives. But the article was already too long, and this is a HUGE can of worms to open.

punnerud 9y ago

Could it be that your dice was biased to begin with? Or have you already rolled them ~5000 times before you placed them in water?

askvictor 9y ago
I was wondering this; is a control unnecessary here as we assume an equal distribution, or is that assumption false?
- punnerud 9y ago
  I think it’s false because every mechanical device have some variation, but is it so small it can don’t “affect” the result?

smallnamespace 9y ago

Note that if your only goal was to test whether the dice was loaded, using a test against only one outcome (in this case, only 6s) is not the best way. Use either a Kolmogorov-Smirnov test[1] or the Anderson-Darling test[2]. The intuition is that these tests are more powerful because they take the difference between the entire empirical distribution minus the expected probability mass distribution. You're using 'all the numbers' simultaneously to check for cheating. Funnily enough, I first learned about these tests a nearly a decade ago, precisely because I wanted to know whether Settlers dice were loaded. [1] https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_tes... [2] https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test

FabHK 9y ago
I think for this case (with 6 discrete outcomes) you could just use Pearson's Chi-Square test. What you describe is general enough for continuous distributions. https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test

hinkley 9y ago

My old boss was into Settlers, and they stayed late to play some games a few times. I got invited once and weird stuff happened. Somehow I ended up next to both 6's on the map. As near as I can tell, one of the dice we used would not land on 4. Whatever was going on, the number of 6's and 9's rolled that night were abnormally high. By the time people figured out we were rolling as many 6's as we had 7's and 8's combined, I already had a town on the coast and pretty much steamrolled everybody with sheep and wheat. Really the game wasn't that fun at that point, so I just tried to end it as fast as possible. When you're playing an RPG, everybody seems to gravitate toward their 'lucky dice' which are most likely defective in the right way. But if you're playing a game with others you probably want the fairest dice you can find. I remember years ago seeing a sales video from some retired aerospace engineer that was making geometrically perfect dice. He'd worked out the resins so they cured uniformly. He'd stack his dice next to some random set and point out how the other guy's stack curved to one side while his was perfectly straight. Probably didn't sell a lot of those to DND players, except perhaps GMs.

vanderZwan 9y ago
> Probably didn't sell a lot of those to DND players, except perhaps GMs. GM's don't have to show their dice rolls and can overrule the outcome if it fits their storyline better, so the rolling is mostly ceremonial anyway.