3 ms·
Let's skip straight to the actual data: https://www.nature.com/articles/s41586-022-04643-y/tables/2 https://www.nature.com/articles/s41586-022-04643-y/tables/2
by incrudible 4y ago
Let's skip straight to the actual data:
https://www.nature.com/articles/s41586-022-04643-y/tables/2 https://www.nature.com/articles/s41586-022-04643-y/tables/2
The way I see it, even if this data isn't just noise, the difference is negligible. Moreover, generating slightly more ideas isn't necessarily valuable. Most ideas are worthless distractions. Curiously, some of the data suggests that the virtual team picked the better ideas, but again, the difference is negligible.
- chewbacha 4y agoyea, that's how I read it too. Those standard deviations are pretty wide.
- Mathnerd314 4y agoIt seems they have sufficient data points that the difference in the means is significant even with high variance. For example, take "number of creative ideas" because it has the largest Cohen's d besides the gaze measurements. Doing Welch's t-test to compare and assuming the virtual/physical were evenly split (not reported), the statistic is (14.74-16.77) / sqrt(6.23^2/150 + 7.27^2/150)=2.6 and at 75 degrees of freedom that's something like p=0.03, still pretty significant. In the paper they use a negative binomial distribution because the distributions are significantly non-normal and get a lower p=0.002. Then in the field studies with 745 more pairs they get pretty much the same results. So overall the statistics seem fine and they probably found a replicable effect. The more likely issue is garbage in, garbage out - their study design is really not that great, e.g. the time taken to switch between the screen and the tablet could be the cause of the difference.