4 ms·

Would be nice to see actual the numbers to judge whether the results are statistically significant. At first blush all but the last Dell treatment are absolute

Would be nice to see actual the numbers to judge whether the results are statistically significant. At first blush all but the last Dell treatment are absolutely horrendous to my eye, so I just don't know how to even judge these.

robomartin 14y ago
How would you calculate what would constitute statistically significant results for each of these sites? How about any other site?
- saurik 14y ago
  It doesn't really have anything to do with the site... it has to do with what the difference is vs. the total size of the sample. If you see a 50% increase but only had 3 people in your sample, that is not statistically significant because the probability of that happening by chance is so high. To make this concrete, if the actual proportion was 4:1 against that figure (as in, you "should" have measured a whopping decrease to 25% the original conversion), you'd still measure the totally incorrect 50% increase in one out of every ten attempts at that measurement.
  - robomartin 14y ago
    I failed to make the point of my question clear enough. Nickpotier critiqued the results without offering up any numbers. I wanted to see what he thought the numbers should have been for each example given.
- jmount 14y ago
  Roughly a good sample size is around -log(d/2)/(2 e a^2) where d is the significance you want and a is the difference in conversion rates you hope to measure. See http://www.win-vector.com/blog/2013/03/a-bit-more-on-sample-size/ http://www.win-vector.com/blog/2013/03/a-bit-more-on-sample-...
  - Sprint 14y ago
    I read that and I have no idea what those values are. Is there an example of this with actual real-life data?
    - russell 14y ago
      He gave you a reference. Google "statistical sample size". There are several calculators in the results that you can play around. Everybody should take a college level course in statistics, even poets and painters.

robomartin 14y ago

How would you calculate what would constitute statistically significant results for each of these sites? How about any other site?

saurik 14y ago
It doesn't really have anything to do with the site... it has to do with what the difference is vs. the total size of the sample. If you see a 50% increase but only had 3 people in your sample, that is not statistically significant because the probability of that happening by chance is so high. To make this concrete, if the actual proportion was 4:1 against that figure (as in, you "should" have measured a whopping decrease to 25% the original conversion), you'd still measure the totally incorrect 50% increase in one out of every ten attempts at that measurement.
- robomartin 14y ago
  I failed to make the point of my question clear enough. Nickpotier critiqued the results without offering up any numbers. I wanted to see what he thought the numbers should have been for each example given.
jmount 14y ago
Roughly a good sample size is around -log(d/2)/(2 e a^2) where d is the significance you want and a is the difference in conversion rates you hope to measure. See http://www.win-vector.com/blog/2013/03/a-bit-more-on-sample-size/ http://www.win-vector.com/blog/2013/03/a-bit-more-on-sample-...
- Sprint 14y ago
  I read that and I have no idea what those values are. Is there an example of this with actual real-life data?
  - russell 14y ago
    He gave you a reference. Google "statistical sample size". There are several calculators in the results that you can play around. Everybody should take a college level course in statistics, even poets and painters.