4 ms·
Yes this is an awful design with a completely predictable outcome, and I hope for ethical reasons the control students will get chromebooks and access to the bl
by edtechdev 15y ago
Yes this is an awful design with a completely predictable outcome, and I hope for ethical reasons the control students will get chromebooks and access to the blended learning stuff afterward at least. And using the same teacher in both conditions doesn't solve problems - the teacher expects students in the experimental group to do better, and will influence the result even if not actively trying to.
But unfortunately A) this is the kind of studies the department of education wants and funds (randomized experimental designs - see the 'what works' database, where they ignore anything not using that design). And this is the case even when the control group is obviously in an intentionally deprived learning environment. And B) the opposite design, tightly equalizing every aspect of the control and experimental groups, isn't good either, because it wouldn't take advantage of the new things blended learning / software afford (like animations, interactive feedback) that traditional paper-based instruction doesn't. See page 2 of this article from the Concord Consortium about the 'hobbled horse race' and so forth: https://docs.google.com/viewer?url=http%3A%2F%2Fwww.concord.org%2Fsites%2Fdefault%2Ffiles%2Fnewsletters%2F2007%2Fspring%2Fat-concord-spring-2007.pdf https://docs.google.com/viewer?url=http%3A%2F%2Fwww.concord....
There are 2 alternative research designs I would have done in this case:
1) Design-based research: http://edutechwiki.unige.ch/en/Design-based_research http://edutechwiki.unige.ch/en/Design-based_research
Do the best you can with the Khan Academy curriculum in a class. Pre-test and post-test, interview and observe and so forth. Measure what they learned, their engagement, etc. THEN you try to figure out things that didn't work out so well, things you missed...try to apply a theory or framework to explain what happened. THEN try it out a second time making revisions, and see if you can increase the learning gains and engagement even more the next time. Use of pre-test / baseline measures makes it possible to do comparisons.
2) Do short studies comparing different designs of the Khan Academy software and curriculum. This is traditional, individual lab-based studies. Can be useful if there is a valid issue where the designers / researchers don't know the best path, and want to test/compare different options.
- Wilduck 15y agoMethod (1) seems like it could provide some valid insight. To me, however, collecting information from a number of schools seems like the best approach. I'm not terribly familiar with how studies are done in schools. Is it incredibly difficult to collect cross-section which is large enough to control for differences in teachers/students? Maybe my inner economics student is showing through, but this seems like a prime target for good old-fashioned regression analysis.
- wisty 15y agoEducation is not so special that it can't be subjected to the scientific method. Medicine is similarly complex, and benefited immensely from randomised trials. There may be challenges (the placebo effect of new methodologies, and the impact which observation has on behavior), but it's not so special that big gains can't be made by studying what works in an empirical manner. When education catches up to where medicine was in the 1937 (with the introduction of placebo trials), or even the 1950s (with double blind trials) they can start looking into methodological improvements. Until then, there's lots of low-hanging apples. I guess the best way forward is for the education departments to stop trying to control everything. They send out instructions, telling teachers what they are meant to do. Why not just AB test the instructions they are sending principals? That way, they can learn which of their decisions actually work.
- edtechdev 15y agoStudents aren't rats. Which makes things a little more complex for educational research: http://www.elearning-reviews.org/topics/pedagogy/educational-principles/2002-berliner-educational-research-hardest-science/ http://www.elearning-reviews.org/topics/pedagogy/educational... Like, little Johnny's Mom isn't going to like it when he gets a 'placebo' math class one year.