4 ms·

Why don't policies go through double blind trials or more A/B tests? And attach automatic rollbacks to policies based on metrics. I'm all for trying new solutio

Why don't policies go through double blind trials or more A/B tests? And attach automatic rollbacks to policies based on metrics. I'm all for trying new solutions but they should be iterated on quickly.

jkingsbery 3y ago
The facetious answer, of course, is that because the people designing these policies learned math in the California public school system. Less facetiously: A/B tests are great at lots of things. But these results really shouldn't surprise anyone. The idea that you will improve things for one group of students by making things worse for another group is really not only silly, it cost real kids a year of potential progress. An A/B test would cost some percentage of kids that year, which is better than hurting the whole group, but A/B testing on something that obviously would hurt their future does not seem fair to those hurt by the proposed changes.
alex_sf 3y ago
Education is not a javascript app. Even ignoring the consent and ethical issues, doing something double-blind in a government ran system impacting statistically significant number of children simply can't be done. You can't maintain the blinding. 'Automatic rollbacks' are, frankly, absurd. How do you 'roll back' a child in 8th grade that failed algebra? Your quick iteration turns into dozens of cohorts to be tracked and managed.
- michael_vo 3y ago
  I meant rollback of a policy that performed poorly. It is an interesting point you bring up about rolling back a child that I haven't considered. That might be a good policy to have. A question to ask is it better for a kid to score 90% in algebra and it takes 2 more years or is it better that they score 50% and then move on to calculus. That would be a good A/B test to compare their lifetime earnings. My hypothesis is that calculus is a pretty useless skill and basic maths is more important in life. There could be ways of doing A/B tests if the standardized tests are the same between states that have different curriculums.

jkingsbery 3y ago

The facetious answer, of course, is that because the people designing these policies learned math in the California public school system. Less facetiously: A/B tests are great at lots of things. But these results really shouldn't surprise anyone. The idea that you will improve things for one group of students by making things worse for another group is really not only silly, it cost real kids a year of potential progress. An A/B test would cost some percentage of kids that year, which is better than hurting the whole group, but A/B testing on something that obviously would hurt their future does not seem fair to those hurt by the proposed changes.

alex_sf 3y ago

Education is not a javascript app. Even ignoring the consent and ethical issues, doing something double-blind in a government ran system impacting statistically significant number of children simply can't be done. You can't maintain the blinding. 'Automatic rollbacks' are, frankly, absurd. How do you 'roll back' a child in 8th grade that failed algebra? Your quick iteration turns into dozens of cohorts to be tracked and managed.

michael_vo 3y ago
I meant rollback of a policy that performed poorly. It is an interesting point you bring up about rolling back a child that I haven't considered. That might be a good policy to have. A question to ask is it better for a kid to score 90% in algebra and it takes 2 more years or is it better that they score 50% and then move on to calculus. That would be a good A/B test to compare their lifetime earnings. My hypothesis is that calculus is a pretty useless skill and basic maths is more important in life. There could be ways of doing A/B tests if the standardized tests are the same between states that have different curriculums.