4 ms·
With big enough cohorts we can AB test by quantifying outcomes? Treatment group A has AI-generated texts and Treatment group B has the originals. We can have
by willvarfar 2y ago
With big enough cohorts we can AB test by quantifying outcomes? Treatment group A has AI-generated texts and Treatment group B has the originals. We can have a questionnaire for how the group felt about the material etc, but we can also perhaps measure life outcomes over a bigger period e.g. performance at school or in the market place?
As I write this it feels naive and reminds me of a thousand ill-thought-out AB website tests etc. But still :D
- bryanrasmussen 2y agoexactly how long are you planning on running these tests for - sounds like minimum 20 years?
- willvarfar 2y agoYeap :) There are plenty of very long-term studies going on where groups are identified and then followed over years or decades. Another approach is today you might go looking for people with different levels of airborne lead exposure as a child, and then compare the average income today to see if their is correlation etc. In these kinds of studies the treatment groups weren't wittingly part of the experiment, but you can look backwards. Either way, in 20 years we'll probably be able to identify schools where AI texts predominated earlier than other schools and, adjusting for other factors, try and tease out the actual impact of that choice.
- bryanrasmussen 2y agoI mean the thing is, ethically speaking, there is a strong supposition that the AI predominating instructional format will underperform the other formats, therefore you are essentially putting large groups of people, on purpose, in learning situations the theory is will underperform for the rest of their lives. That seems like a bad thing. I suppose that it will happen however, and the backwards looking thing will be what is done to determine the result.