5 ms·

A random sample of 20,000 is massive. A non-random sample of even a million can easily be worthless. Political polls are very careful about getting a sample re

A random sample of 20,000 is massive. A non-random sample of even a million can easily be worthless. Political polls are very careful about getting a sample representative of the country which is why they can have some value. They also have very detailed models, which a lot of thought have gone into, about what the general population looks like and how to get accurate results. The weighting from this facebook poll is not based a bunch of well-developed knowledge but just guesses at what to do.

slantyyz 16y ago
The problem even with political polls is unless an expected result is obviously hard to fudge, you still need a lot of healthy skepticism, because the money behind a poll tends to subtly skew the results. That's why GOP poll results tend to conflict with DEM results.
mytype 16y ago
Our sampling rigor cannot be compared to political polls, it's true. However we do weight the sample not based on guesses but on official data about the gender, age, and personality distribution in the US. We did not use geographic weights because our respondents were more or less properly distributed. Any poll can be attacked. Even political polls are subject to bias in who picks up the phone, who responds to the surveyor, etc. This bias is partially driven by personality type, which we are able to nearly eliminate through weighting by personality traits. How many polls can do that? Most polling is much less random than political polls, and yet the results are still treated as worthwhile. Some firms put up surveys on sites (site audience bias), or have carefully selected paid volunteers (bias to incorporate people who are willing to do that), etc. The fact that our results are corroborated by Forrestor and Sybase, to the extent that we can be compared, is very satisfying to me personally. Sure, we may be a few percentage points off on some things, but are the overall results invalid? No. I challenge anyone to point to any result in the study and give a reasonable explanation why that result is likely way off. The only things that are suspect to me are the major outliers, subsets like native americans for which we did not have much data. I'll give you that. Anything else? Let me know.
- flatulent1 16y ago
  There is an element of non-randomness in drawing from Facebook users although I'm not sure who it favors. (I'm one of the minority not being on there. I don't text people either) Looks like you did a pretty good job with this. It's a thought provoking read for sure. It occurred to me that there may be an issue with the "lust" category. It lumps those who lust for an Escalade in with those who lust for a human - very different things. I suspect that geeks feeling plenty of lust for humans may be less likely to regard it as a sin and might be less likely to respond to that choice. It's interesting that the iPad demographic seems to lean away from slender and no children (which probably includes most gays?). If I had to guess, I'd say that group leans more than average towards the Mac. It would have been interesting to see sexual orientation and OSes used/avoided or primary OS in this. I suppose it might have generated a number of emotional responses though LOL
- bandushrew 16y ago
  "I challenge anyone to point to any result in the study and give a reasonable explanation why that result is likely way off." plan to buy: forrester 3.8% plan to buy: you 2.0% so your results differ markedly from the forrester report itself? Your results are different (and likely differ from reality) because you did not use a random sample, you used a self selected sample of people who belong to facebook and wanted to respond to a poll. That is simply a fact. Having taken a non-random sample you then projected your own opinions into all your conclusions. Its an amusing post, but not really an interesting one.
  - mytype 16y ago
    There's two explanations for that difference: 1) Forrestor's 3.8% was collected in June 2010, whereas our data was collected from March through May. Clearly the iPad is gaining momentum, as time goes on more people are planning to buy one (at least for now). 2) More importantly, Forrestor said that "no time frame was specified in the survey question". So they asked people who don't own an iPad if they intend to buy one. That's fairly different from our question, which allowed the respondent to choose from: 1) plan to buy one 2) want to play with one first 3) will wait for later versions 4) waiting for the consensus opinion Clearly, with the variety of options, some people who might have simply answered "yes" to the "intend to buy" Forrestor question would pick 2, 3 or 4 in our question. Does that settle that? And by the way, these are people who wanted to take a personality quiz, not answer a poll. Their motivation had nothing to do with the iPad, that question was randomly inserted into the personality quiz. While you seem to be "amused" by a blogger who in your mind completely makes stuff up, I'm amused by a commenter who after 5 minutes of review feels confident enough to slam a study that took multiple people dozens of hours to complete. We're not publishing this in a science journal. That's not what we're going for. It's not complete BS either though. We do normalization, we have lots of data, and our psychological measures are based on the best contemporary research. The results are worthwhile. I'll say again: I challenge anyone to point out a major flaw in our data (not my interpretation of it).
moultano 16y ago
All data is flawed. What matters is whether the data is too flawed to be useful.
walkon 16y ago
The weighting from this facebook poll is not based a bunch of well-developed knowledge but just guesses at what to do. As mytype clarified: we do weight the sample not based on guesses but on official data about the gender, age, and personality distribution in the US. We did not use geographic weights because our respondents were more or less properly distributed. You criticized MyType for guessing while you guessed that they guessed. That's a big high on the hypocritical scale.

slantyyz 16y ago

The problem even with political polls is unless an expected result is obviously hard to fudge, you still need a lot of healthy skepticism, because the money behind a poll tends to subtly skew the results. That's why GOP poll results tend to conflict with DEM results.

mytype 16y ago

Our sampling rigor cannot be compared to political polls, it's true. However we do weight the sample not based on guesses but on official data about the gender, age, and personality distribution in the US. We did not use geographic weights because our respondents were more or less properly distributed. Any poll can be attacked. Even political polls are subject to bias in who picks up the phone, who responds to the surveyor, etc. This bias is partially driven by personality type, which we are able to nearly eliminate through weighting by personality traits. How many polls can do that? Most polling is much less random than political polls, and yet the results are still treated as worthwhile. Some firms put up surveys on sites (site audience bias), or have carefully selected paid volunteers (bias to incorporate people who are willing to do that), etc. The fact that our results are corroborated by Forrestor and Sybase, to the extent that we can be compared, is very satisfying to me personally. Sure, we may be a few percentage points off on some things, but are the overall results invalid? No. I challenge anyone to point to any result in the study and give a reasonable explanation why that result is likely way off. The only things that are suspect to me are the major outliers, subsets like native americans for which we did not have much data. I'll give you that. Anything else? Let me know.

flatulent1 16y ago
There is an element of non-randomness in drawing from Facebook users although I'm not sure who it favors. (I'm one of the minority not being on there. I don't text people either) Looks like you did a pretty good job with this. It's a thought provoking read for sure. It occurred to me that there may be an issue with the "lust" category. It lumps those who lust for an Escalade in with those who lust for a human - very different things. I suspect that geeks feeling plenty of lust for humans may be less likely to regard it as a sin and might be less likely to respond to that choice. It's interesting that the iPad demographic seems to lean away from slender and no children (which probably includes most gays?). If I had to guess, I'd say that group leans more than average towards the Mac. It would have been interesting to see sexual orientation and OSes used/avoided or primary OS in this. I suppose it might have generated a number of emotional responses though LOL
bandushrew 16y ago
"I challenge anyone to point to any result in the study and give a reasonable explanation why that result is likely way off." plan to buy: forrester 3.8% plan to buy: you 2.0% so your results differ markedly from the forrester report itself? Your results are different (and likely differ from reality) because you did not use a random sample, you used a self selected sample of people who belong to facebook and wanted to respond to a poll. That is simply a fact. Having taken a non-random sample you then projected your own opinions into all your conclusions. Its an amusing post, but not really an interesting one.
- mytype 16y ago
  There's two explanations for that difference: 1) Forrestor's 3.8% was collected in June 2010, whereas our data was collected from March through May. Clearly the iPad is gaining momentum, as time goes on more people are planning to buy one (at least for now). 2) More importantly, Forrestor said that "no time frame was specified in the survey question". So they asked people who don't own an iPad if they intend to buy one. That's fairly different from our question, which allowed the respondent to choose from: 1) plan to buy one 2) want to play with one first 3) will wait for later versions 4) waiting for the consensus opinion Clearly, with the variety of options, some people who might have simply answered "yes" to the "intend to buy" Forrestor question would pick 2, 3 or 4 in our question. Does that settle that? And by the way, these are people who wanted to take a personality quiz, not answer a poll. Their motivation had nothing to do with the iPad, that question was randomly inserted into the personality quiz. While you seem to be "amused" by a blogger who in your mind completely makes stuff up, I'm amused by a commenter who after 5 minutes of review feels confident enough to slam a study that took multiple people dozens of hours to complete. We're not publishing this in a science journal. That's not what we're going for. It's not complete BS either though. We do normalization, we have lots of data, and our psychological measures are based on the best contemporary research. The results are worthwhile. I'll say again: I challenge anyone to point out a major flaw in our data (not my interpretation of it).

moultano 16y ago

All data is flawed. What matters is whether the data is too flawed to be useful.

walkon 16y ago

The weighting from this facebook poll is not based a bunch of well-developed knowledge but just guesses at what to do. As mytype clarified: we do weight the sample not based on guesses but on official data about the gender, age, and personality distribution in the US. We did not use geographic weights because our respondents were more or less properly distributed. You criticized MyType for guessing while you guessed that they guessed. That's a big high on the hypocritical scale.