4 ms·
In the comments, the authors give the sample size as about 20 in each group. Here's a more sensible graph (error bars to create a 95% CI on the estimated group
by john_horton 15y ago
In the comments, the authors give the sample size as about 20 in each group. Here's a more sensible graph (error bars to create a 95% CI on the estimated group means):
https://skitch.com/johnjosephhorton/fih62/quartz-2 https://skitch.com/johnjosephhorton/fih62/quartz-2
Obviously, there are some statistical power issues. It's unfortunate that they didn't get someone to help them with a power analysis up front---given how many very tech savvy people are interested in Kahn Academy and want to see it succeed, I think they could have gotten this sort of assistance pretty easily.
- JunkDNA 15y agoThanks for doing that, I found myself wishing they had put error bars on when I saw that graph but was too lazy to do it myself. With respect to the statistical power issue, it may be they just didn't have enough students. Unlike your typical clinical trial or scientific study where you're recruiting people, I got the impression this was from an existing pool of students for a specific school. Once you go beyond this, the design gets tricky since there are practical issues to how big a class can be. Then you need more teachers, and you need to normalize for per-teacher variability. Not saying a study like that wouldn't be great, but it would be complex to get right.
- john_horton 15y agono problem & agreed about what probably happened - my point was more that knowing they had that sample size (n=40), someone could have told them that even if Kahn academy was extraordinarily effective, there was almost no chance they'd detect statistically significant effect. If they knew that, they might have pursued a more qualitative approach rather than have people see the take-away as "no effect."