3 ms·

I'd really appreciate knowing your specific objections to A/B testing in digital education. For instance, I have an adaptive quiz system that delivers interven

I'd really appreciate knowing your specific objections to A/B testing in digital education. For instance, I have an adaptive quiz system that delivers interventions when students struggle. An intervention might be, for instance, a video for teaching fraction addition. In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? It feels unethical to NOT use A/B tests to improve digital education. Why only apply that technology to the improvement of advertising and YouTube recommendations, vs curricular recommendations?

raxxorrax 7y ago
For math there certainly is some leniency to use such tests for demonstration purposes and I am not opposed to using digital tools to enhance learning. I think it is even mandatory to provide the best support and that it can vastly improve classical learning. My reservations are primarily that it mechanizes learning and experience tells me that pedagogical measures will quickly be reduced to just test pupils that leads to the opposite of the goal, that test become more individualized for the learning requirements of the kids in question. Another problem is that the pupils might be smart. So I wouldn't be surprised when the tests show that the best form of test is the one they just like the most or can be done with the least amount of effort, because they take being tested for granted. And while teaching math, you might also teach them to apply A/B tests on other people, including teachers of course. > In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? I wouldn't have any issues here. Most of the criticism is void as long as pupils get access to both resources and if there is an analysis after the A/B test and the students know about it. Would you still think it ethical, if you made it a competition between the tested groups? Perhaps I am to cynical if I believe such measures will always be used as an excuse to delay employing more teachers and that the result of the tests will be treated as gospel, even if the failure rate is known and most classes don't provide random samples. But aside from that, I don't think this form of test is a "technology" and there can be a lot of criticism leveled against youtube and the ad industry too for that matter.
- dr_dshiv 7y ago
  I love this thoughtful response! >Would you still think it ethical, if you made it a competition between the tested groups? I do -- but it's all about framing, transparency and perceived intent. We are planning a participatory design session between teachers, product people and edu researchers -- to design the a/b tests and generate generalizable scientific knowledge. Might you have any further advice for us?

raxxorrax 7y ago

For math there certainly is some leniency to use such tests for demonstration purposes and I am not opposed to using digital tools to enhance learning. I think it is even mandatory to provide the best support and that it can vastly improve classical learning. My reservations are primarily that it mechanizes learning and experience tells me that pedagogical measures will quickly be reduced to just test pupils that leads to the opposite of the goal, that test become more individualized for the learning requirements of the kids in question. Another problem is that the pupils might be smart. So I wouldn't be surprised when the tests show that the best form of test is the one they just like the most or can be done with the least amount of effort, because they take being tested for granted. And while teaching math, you might also teach them to apply A/B tests on other people, including teachers of course. > In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? I wouldn't have any issues here. Most of the criticism is void as long as pupils get access to both resources and if there is an analysis after the A/B test and the students know about it. Would you still think it ethical, if you made it a competition between the tested groups? Perhaps I am to cynical if I believe such measures will always be used as an excuse to delay employing more teachers and that the result of the tests will be treated as gospel, even if the failure rate is known and most classes don't provide random samples. But aside from that, I don't think this form of test is a "technology" and there can be a lot of criticism leveled against youtube and the ad industry too for that matter.

dr_dshiv 7y ago
I love this thoughtful response! >Would you still think it ethical, if you made it a competition between the tested groups? I do -- but it's all about framing, transparency and perceived intent. We are planning a participatory design session between teachers, product people and edu researchers -- to design the a/b tests and generate generalizable scientific knowledge. Might you have any further advice for us?