5 ms·
It's a bit hard to assess the significance of this. Sample size isn't even given. 40 people went through this test, or 4 million?
by TimH 15y ago
It's a bit hard to assess the significance of this. Sample size isn't even given. 40 people went through this test, or 4 million?
- westiseast 15y agosorry, yes you're right. The sample size was not huge (@500 people viewed each variant) although the result was statistically significant. I just added a note to the post about this. My aim is to repeat the test at a later date to see if the results repeat themselves.
- marekmroz 15y agoCorrect me if I am wrong, but isn't statistical significance a function of sample size?
- cmurphycode 15y agoYes-- sample size, and effect size. He's saying that the results were strong enough that given the sample size, they were statistically significant. Simple example: if I have a study of 20,000 people and I find that 51% of people prefer foos to bars, that may be similarly significant to your study of 20 people, all of which prefer foos to bars.
- westiseast 15y agoIf I've understood it correctly, the smaller your sample size, the bigger the difference you need between response rates. So if the difference is 60%, then a sample of 30 people can be statistically significant. But if you're looking for a 5% improvement on something, you'll need to sample a large number of people. I just used this: http://www.prconline.com/education/tools/statsignificance/index.asp http://www.prconline.com/education/tools/statsignificance/in...
- VMG 15y agoYou are right. The author should plug the numbers in here: http://www.usereffect.com/split-test-calculator http://www.usereffect.com/split-test-calculator I'd go as far as to consider the data to be worthless without the confidence value