4 ms·
Not to be pedantic, but this headline is not accurate. There’s nothing that makes me confident that respondents to the game dev conference survey are representa
by prepend 8y ago
Not to be pedantic, but this headline is not accurate. There’s nothing that makes me confident that respondents to the game dev conference survey are representative enough to say “nearly half of game developers.”
This is a really important point as trying to represent a bunch of people’s opinion based on a single, limited survey is not productive.
- criddell 8y agoI would think that conference attendees would be fairly representative of commercial game developers. I don't know enough about statistics to say how big the error bars would be here, but I'm constantly astonished by how well statistical methods can work with even seemingly small samples.
- MCrekt 8y agoBut considering how many developers don’t live where the conference takes place and/or can’t get there I have to agree with the top here and say this survey is severely limited.
- walshemj 8y agoit would indicate the better paid ones who can afford to g to industry conferences
- gizmo686 8y agoThe issue isn't with sample size. It is with sample method. The population of gamedevs who go to a conference is probably not a random selection of gamedevs in general.
- walshemj 8y agoIts a sample size of 4000 and electoral polls are smaller than that 1k or 3k
- gdix 8y ago> The issue isn't with sample size.
- naniwaduni 8y agoThe issue isn't with sample size. The sample is almost certainly biased, we just don't necessarily know how much and in what direction.
- frobozz 8y agosample size is irrelevant if the selection criteria are poor.
- gamblor956 8y agoIf the sample size is large enough the selection criteria doesn't matter.
- jonas21 8y agoI guess that's technically true, but only if your sample size is 100%. :-)
- kolinko 8y agoThere were examples of very large polls that failed spectacularly because of the badly chosen samples https://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html https://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.h...
- alasdair_ 8y agoTake a sample of 5000 random people (an enormous sample size) attending the Republican National Convention and ask them which party they will vote for.
- jcoffland 8y agoI would expect smaller game companies to be less well represented.
- skybrian 8y agoWell, you can do a lot with a small sample provided that it's a random sample. It seems like you'd have to know how conference attendees differ from the population of game developers you're interested in.
- tridentboy 8y agoStatistically speaking 4000 is a good enough number. I mean, election polls in the US usually use less than that for a population of 300 million and they're usually close to the real number.
- maccio92 8y agoExcept in 2016. And that 4000 is a biased population of only game devs that attended a certain conference. Maybe if it was a totally random sample of 4000 of all game devs it would be statistically significant.
- learc83 8y ago>Except in 2016 In 2016 the national polls were very close to the final result--within a couple points. Remember that Clinton won the popular vote. State polls were less accurate, but the general consensus is they were wrong due to undecided voters breaking more for Trump than is normal in the final few days--after the final state polls were conducted. That being said, your point stands. Sample method is generally more important than sample size, and 4,000 game devs who self selected to attend a conference is very unlikely to be a representative sample.
- phkahler 8y ago>> Statistically speaking 4000 is a good enough number. It seems to me the GP comment meant that because it was a survey among people who attended GDC it's not representative. If that is why they questioned it, I'd say it's an even more relevant survey, as those are the people who care more about the industry and have more influence. No mater how you slice it, I'd say it's an indicator not to be ignored.
- prepend 8y agoIt’s an indicator, but just unlikely to be an indicator for the population of game developers.
- frobozz 8y agoWhen they do these election polls, do they stand in the street randomly choosing people coming out of the conference centre where one of the major parties is having their AGM?