5 ms·

You can bet that there are a lot of social science researchers within Facebook (I know at least two), but they've learned to never ever publish the results of t

You can bet that there are a lot of social science researchers within Facebook (I know at least two), but they've learned to never ever publish the results of that research and only use it to increase revenue.

dr_dshiv 7y ago
This a million times. All the public hysteria does is punish publication. The work goes on.
- kohtatsu 7y ago
  It also leads people to leave the platform.
- dr_dshiv 7y ago
  Hysteria around product experiments has had a big effect in education. As we all know, most online software companies conduct A/B tests to improve outcomes. IMO, education companies should do a/b testing in order to optimize student outcomes. It is unethical to not run tests. But, this can be described as "psychological manipulation" and "turning children into guinea pigs" -- and people freak. After all, there is not informed consent in A/B tests. Since educational software optimization does not actually lead to greater profit (the edu market is weird) the optimization work stops and no science is published. I find this disturbing. https://gizmodo.com/pearson-embedded-a-social-psychological-experiment-in-s-1825367784 https://gizmodo.com/pearson-embedded-a-social-psychological-...
  - gbear605 7y ago
    The problem is that when you run a/b tests, the changed case is often worse. If the thing being tested is a website, that might not be so bad. If the thing being tested is your education, that might be horrible.
    - mygo 7y ago
      No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. For example, FSU recently went up in rankings from a ~#50 public university to a #18 public university year over year over the past decade. Whatever changes they put into place are now being studied by other universities and if there’s anything novel that they’re doing, it will likely become more widespread as other universities seeking higher rankings implement new things. Or, it could be possible that the university studied the success of other universities and implemented what worked best and stopped what wasn’t. You run an a/b test when there aren’t already tens of thousands of other colleges and universities like yourself out there that you can study and learn from. Of course, rankings DNE educational quality, but it’s one example. Another example, that may have a better correlation with educational quality (although part of the equation is the constitution of the cohort) may be bar exam pass rates for law school grads listed by law school.
      - dr_dshiv 7y ago
        > No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. Why would it be any different in webtech or advertising? Why wouldn't everyone just copy the best vs gathering empirical data?
        mygo 7y ago
        Are universities not gathering empirical data when they study the results of others’ efforts at scale? Study being the key word in that scenario, it’s more than just copying what they see, but understanding why it’s effective, and if it would be for them. And it may be quicker and more reliable with less risk than A/B testing in their setting. The most highly trafficked web tech companies can gather statistically significant feedback data about a change in moments. A/B testing a curriculum or educational practice could take a semester or more, and then the risks are higher — it would be a two-sided hypothesis test where the B group could not only do better, but could also do much worse, and it would reflect poorly on the institution if so. People are paying tens of thousands of dollars per seat per year for the best education they can get. Seeing how many items are in the shopping cart right on the “checkout” button doesn’t really reflect poorly on Amazon, but it could help Amazon increase conversion rates by .03%, which could mean millions of dollars at their scale, and they could also complete the test fairly quickly given their volume (in a day or so?) at a 99.7% confidence interval. With that being said, I’m sure that smaller scale or faster turnaround time A/B tests are being ran at Uni’s.
  - raxxorrax 7y ago
    In my opinion, A/B tests are a flawed application to optimize the curriculum before even touching ethical concerns. The mechanism is far too reductive and would only increase current problems in education. We know that a mentor for every child isn't logistically feasible but conducting these tests would just make sure we continue to work past the problem. > It is unethical to not run tests Certainly not.
    - dr_dshiv 7y ago
      I'd really appreciate knowing your specific objections to A/B testing in digital education. For instance, I have an adaptive quiz system that delivers interventions when students struggle. An intervention might be, for instance, a video for teaching fraction addition. In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? It feels unethical to NOT use A/B tests to improve digital education. Why only apply that technology to the improvement of advertising and YouTube recommendations, vs curricular recommendations?
      - raxxorrax 7y ago
        For math there certainly is some leniency to use such tests for demonstration purposes and I am not opposed to using digital tools to enhance learning. I think it is even mandatory to provide the best support and that it can vastly improve classical learning. My reservations are primarily that it mechanizes learning and experience tells me that pedagogical measures will quickly be reduced to just test pupils that leads to the opposite of the goal, that test become more individualized for the learning requirements of the kids in question. Another problem is that the pupils might be smart. So I wouldn't be surprised when the tests show that the best form of test is the one they just like the most or can be done with the least amount of effort, because they take being tested for granted. And while teaching math, you might also teach them to apply A/B tests on other people, including teachers of course. > In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? I wouldn't have any issues here. Most of the criticism is void as long as pupils get access to both resources and if there is an analysis after the A/B test and the students know about it. Would you still think it ethical, if you made it a competition between the tested groups? Perhaps I am to cynical if I believe such measures will always be used as an excuse to delay employing more teachers and that the result of the tests will be treated as gospel, even if the failure rate is known and most classes don't provide random samples. But aside from that, I don't think this form of test is a "technology" and there can be a lot of criticism leveled against youtube and the ad industry too for that matter.

dr_dshiv 7y ago

This a million times. All the public hysteria does is punish publication. The work goes on.

kohtatsu 7y ago
It also leads people to leave the platform.
dr_dshiv 7y ago
Hysteria around product experiments has had a big effect in education. As we all know, most online software companies conduct A/B tests to improve outcomes. IMO, education companies should do a/b testing in order to optimize student outcomes. It is unethical to not run tests. But, this can be described as "psychological manipulation" and "turning children into guinea pigs" -- and people freak. After all, there is not informed consent in A/B tests. Since educational software optimization does not actually lead to greater profit (the edu market is weird) the optimization work stops and no science is published. I find this disturbing. https://gizmodo.com/pearson-embedded-a-social-psychological-experiment-in-s-1825367784 https://gizmodo.com/pearson-embedded-a-social-psychological-...
- gbear605 7y ago
  The problem is that when you run a/b tests, the changed case is often worse. If the thing being tested is a website, that might not be so bad. If the thing being tested is your education, that might be horrible.
  - mygo 7y ago
    No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. For example, FSU recently went up in rankings from a ~#50 public university to a #18 public university year over year over the past decade. Whatever changes they put into place are now being studied by other universities and if there’s anything novel that they’re doing, it will likely become more widespread as other universities seeking higher rankings implement new things. Or, it could be possible that the university studied the success of other universities and implemented what worked best and stopped what wasn’t. You run an a/b test when there aren’t already tens of thousands of other colleges and universities like yourself out there that you can study and learn from. Of course, rankings DNE educational quality, but it’s one example. Another example, that may have a better correlation with educational quality (although part of the equation is the constitution of the cohort) may be bar exam pass rates for law school grads listed by law school.
    - dr_dshiv 7y ago
      > No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. Why would it be any different in webtech or advertising? Why wouldn't everyone just copy the best vs gathering empirical data?
      - mygo 7y ago
        Are universities not gathering empirical data when they study the results of others’ efforts at scale? Study being the key word in that scenario, it’s more than just copying what they see, but understanding why it’s effective, and if it would be for them. And it may be quicker and more reliable with less risk than A/B testing in their setting. The most highly trafficked web tech companies can gather statistically significant feedback data about a change in moments. A/B testing a curriculum or educational practice could take a semester or more, and then the risks are higher — it would be a two-sided hypothesis test where the B group could not only do better, but could also do much worse, and it would reflect poorly on the institution if so. People are paying tens of thousands of dollars per seat per year for the best education they can get. Seeing how many items are in the shopping cart right on the “checkout” button doesn’t really reflect poorly on Amazon, but it could help Amazon increase conversion rates by .03%, which could mean millions of dollars at their scale, and they could also complete the test fairly quickly given their volume (in a day or so?) at a 99.7% confidence interval. With that being said, I’m sure that smaller scale or faster turnaround time A/B tests are being ran at Uni’s.
- raxxorrax 7y ago
  In my opinion, A/B tests are a flawed application to optimize the curriculum before even touching ethical concerns. The mechanism is far too reductive and would only increase current problems in education. We know that a mentor for every child isn't logistically feasible but conducting these tests would just make sure we continue to work past the problem. > It is unethical to not run tests Certainly not.
  - dr_dshiv 7y ago
    I'd really appreciate knowing your specific objections to A/B testing in digital education. For instance, I have an adaptive quiz system that delivers interventions when students struggle. An intervention might be, for instance, a video for teaching fraction addition. In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? It feels unethical to NOT use A/B tests to improve digital education. Why only apply that technology to the improvement of advertising and YouTube recommendations, vs curricular recommendations?
    - raxxorrax 7y ago
      For math there certainly is some leniency to use such tests for demonstration purposes and I am not opposed to using digital tools to enhance learning. I think it is even mandatory to provide the best support and that it can vastly improve classical learning. My reservations are primarily that it mechanizes learning and experience tells me that pedagogical measures will quickly be reduced to just test pupils that leads to the opposite of the goal, that test become more individualized for the learning requirements of the kids in question. Another problem is that the pupils might be smart. So I wouldn't be surprised when the tests show that the best form of test is the one they just like the most or can be done with the least amount of effort, because they take being tested for granted. And while teaching math, you might also teach them to apply A/B tests on other people, including teachers of course. > In that case, why not compare our video with a Khan academy video teaching the same topic, and see the results? I wouldn't have any issues here. Most of the criticism is void as long as pupils get access to both resources and if there is an analysis after the A/B test and the students know about it. Would you still think it ethical, if you made it a competition between the tested groups? Perhaps I am to cynical if I believe such measures will always be used as an excuse to delay employing more teachers and that the result of the tests will be treated as gospel, even if the failure rate is known and most classes don't provide random samples. But aside from that, I don't think this form of test is a "technology" and there can be a lot of criticism leveled against youtube and the ad industry too for that matter.