6 ms·
The candy weighing demonstration, or, the unwisdom of crowds
- calinet6 13y agoA wonderful demonstration of statistics. On similar ground, this reminds me of Deming's red bead demonstration, relating statistics to corporations and management practices. Best explained dynamically: https://www.youtube.com/watch?v=JeWTD-0BRS4 https://www.youtube.com/watch?v=JeWTD-0BRS4 (delightfully, this is posted by the Mayo Clinic).
- Patient0 13y agoThanks for sharing this. Initially I found the video to be a bit dull but on reflection I think it's a brilliant demonstration on how companies tend to be managed in practise.
- calinet6 13y agoExactly. Look into more of Deming if you liked that; he was brilliant all around and accurately evaluated and solved many management problems on a grand scale.
- secondForty 13y agoNeat example. So nobody in the class just waits until you're not looking and weighs the whole bag with the scale?
- bittercynic 13y agoYou could make that strategy difficult by using a big, floppy bag and a scale with a small platform. Or simply a scale incapable of measuring a weight much larger than the heaviest piece of candy.
- mturmon 13y agoMaking the bag itself heavy would also help.
- chime 13y agoWhen you shake the bag, smaller candies settle down to the bottom, the larger ones get to the top. So even if someone tries to shake the bag to get a 'random' sample, they will be getting a biased sample if they pick all 5 candies from the same layer (usually the top).
- jckt 13y agoThere's a name for that. http://en.wikipedia.org/wiki/Granular_convection http://en.wikipedia.org/wiki/Granular_convection
- huhtenberg 13y agoYup. That's how you increase fruit content in your müsli - shake the bag, scoop from the top and leave oats to others :)
- Robadob 13y agoI would really have expected atleast a small number of people to have carefully looked to workout the distribution and whether there were a higher density of one type (e.g. small) at one position. I imagine the use of a bag rather than a Jar as per the usual school fair game could make spotting these harder though (unless they're allowed to pick up the bag).
- lotsofcows 13y agoPeople forget that Galton's original example was with a group of people with good domain knowledge. It would be interesting to try the experiment on a group of old-school sweet sellers.
- mappum 13y agoThe real reason for the bias is that everyone thought they could sneak a piece.
- mappum 13y agoThe real reason for the bias is that everyone thought they could sneak a piece.
- ChuckMcM 13y agoIn this day and age nobody is going to hand the scale to the next group with the bag sitting on top of it? Its a bit cynical but I would expect that some pairs estimate to be really really close if not spot on as an effect of they just weighed the bag directly.
- rcthompson 13y agoYou could keep the bag on your desk and have students come up and sample from it and weigh them at their own desks, thus ensuring that the bag never comes close to the scale.
- abcd_f 13y agoNot even that. As you pass the 1.5 kg bag to the next group, it's easy to see that that it's not 3-4 kilos as you have just guessed.
- pgeorgi 13y agoPeople routinely overestimate weights. Speaking from experience, it's possible to routinely win bets with an unmarked 2kg dumbbell (which weights "at least 5!")
- teraflop 13y agoAll you have to do is use a scale which is small enough that it can't measure the weight of the complete bag. I guess that still doesn't prevent someone from weighing all of the candies, one small batch at a time, and adding up the results. But hopefully in a classroom it would be obvious if someone was trying to cheat that way. I suspect that asking the students to work in pairs helps to discourage those kinds of shenanigans, too.
- ZoF 13y agoKind of surprised that kids don't realize this.... What grade is it in? The first reaction I think any class I was in would have to this demonstration would be to figure out how we're being cheated. Given a bag with a random sampling of candy and being told to 'pick 5 pieces' I doubt I would choose 5 of the same large candy bars. It seems highly surprising that 100% of the time this is done you don't have a single pair of students reaching just a bit further into that bag. It doesn't even seem to be a particularly impressive demonstration.
- phreeza 13y agoBy that logic, shouldn't they be guessing too low? I think the point is they realise there are different sizes, but consistently overcompensate for this effect.
- baddox 13y ago> Now call out to the students who are sitting near where you hid the envelope: “Um, uh, what’s that over there . . . is it an envelope??? Really? What’s inside? Could you open it up?” A student opens it and reads out what’s written on the sheet inside: “Your guesses are all too high!” Maybe it's because I recently did some reading about magicians, but if I were one of the students I would be thinking that he could have any number of hidden envelopes with different predictions, and he just chose the one that ended up being correct. Of course, I'm deliberately missing the point of the story.
- sp332 13y agoThe point is that they all guessed low. The weight of the bag never changes, and there's no pretense that the teacher didn't know how much the bag weighed all along. You don't even need the envelope - you could just weigh the bag at the end. Same effect.
- rahimnathwani 13y ago"he just chose the one that ended up being correct" Yes, but it wouldn't be so impressive if he were to choose the envelope which said "your guesses were all wrong, and normally distributed about the actual value".
- the_cat_kittles 13y agothis isnt the unwisdom of crowds, its the difficulty of noticing sampling bias
- gwern 13y agoCorrelations between participants (such as due to a systematic bias such as a selection bias) is one of the most common ways for 'wisdom of crowds' to fail. So I'm not sure what your point is.
- the_cat_kittles 13y agoMy point is this accidental selection bias doesn't depend on the crowd- you could just as easily make this mistake if you are 1 person in a room by yourself. Some examples of what I would call unwisdom of crowds might be the biasing feedback effect a prediction market's current price has on people's judgement, or how those with the most money (or the with the smallest marginal value of a dollar) exert a larger influence on a prediction market's predictions, if they are implemented that way.
- bentcorner 13y agoI wonder if he just asked the students how much the bag weighed without giving a scale if you'd get a better answer. http://phenomena.nationalgeographic.com/2013/01/31/the-real-wisdom-of-the-crowds/ http://phenomena.nationalgeographic.com/2013/01/31/the-real-... This post is more "ask a bad question, get a bad answer".
- npsimons 13y agoThis post is also from a teacher of statistics. Sure, you can weigh the whole bag to get the total (or average) weight; how do you propose to measure an aspect of an entire population, say one as large as a country? Sampling has its uses, when done properly. Besides, as others have pointed out, most people are notoriously bad at estimating weight.
- ska 13y agoThe "wisdom of crowds" part of the title is a bit unfortunate. While there can be systemic problems with that type of approach, this does not really demonstrate them. What is demonstrated is when you give the students an algorithm for a biased estimator, the estimate they get is biased. This is good; empirical demonstration is useful... but it isn't the wisdom/unwisdom of crowds, really. edit: good responses! Unfortunately I don't have enough time right now to properly clarify how/why I'm looking at it this way.
- mrow84 13y agoI think the point is that you are taking the weights of the candies as proxies for the guesses of individuals - the (sum * 20) procedure is effectively an average-based estimate, like with the 'traditional' wisdom of crowds type approach. He is then simply showing that for certain populations such approaches won't necessarily give good expected results, and so one should always exercise caution - whether the population is sweets or people's guesses at the weight of a cow is essentially immaterial.
- ska 13y agoBut that misses the whole point of why the "wisdom of crowds" actually works when it does. To wit; the individual biases and estimation errors can tend to cancel out. This is exactly the opposite of the demonstration, which is a systemic, biased estimator.
- jimhefferon 13y agoPerhaps you missed that the title says unwisdom? That is, not-the wisdom.
- ska 13y agoNo, I didn't miss that. It the lack of "crowd" that is the problem. There is no crowd, really, there are a bunch of people performing a sampling algorithm. But crowds don't work that way. This is a perfectly good demonstration of bias in statistical estimators. It just doesn't have anything to do with crowds, their wisdom, or their unwisdom.
- stuartd 13y agoI thought 'aftermath' was a great pun (HN must be Americanising me because of course it should be 'aftermaths')
- Houshalter 13y agoMaybe it just weighs less because some of the students are secretly eating the candy.