5 ms·

After getting burned in 2014 and then again in 2016, I'm sure Facebook has learned its lesson: never let social science researchers anywhere near the company. T

by knzhou 7y ago

After getting burned in 2014 and then again in 2016, I'm sure Facebook has learned its lesson: never let social science researchers anywhere near the company. They don't actually do anything (effect size in this study is 0.1%, even though, as I personally checked in 2014, the filtering it used was the absolute crudest, strongest possible), but they can generate an infinite amount of bad press.

c3534l 7y ago
Ha! I can safely say that companies do not, in general, learn their lesson until it becomes an existential threat.
- jordanbeiber 7y ago
  Companies are often filled with humans. It’s a human thing.
nostrademons 7y ago
You can bet that there are a lot of social science researchers within Facebook (I know at least two), but they've learned to never ever publish the results of that research and only use it to increase revenue.
- dr_dshiv 7y ago
  This a million times. All the public hysteria does is punish publication. The work goes on.
  - kohtatsu 7y ago
    It also leads people to leave the platform.
  - dr_dshiv 7y ago
    Hysteria around product experiments has had a big effect in education. As we all know, most online software companies conduct A/B tests to improve outcomes. IMO, education companies should do a/b testing in order to optimize student outcomes. It is unethical to not run tests. But, this can be described as "psychological manipulation" and "turning children into guinea pigs" -- and people freak. After all, there is not informed consent in A/B tests. Since educational software optimization does not actually lead to greater profit (the edu market is weird) the optimization work stops and no science is published. I find this disturbing. https://gizmodo.com/pearson-embedded-a-social-psychological-experiment-in-s-1825367784 https://gizmodo.com/pearson-embedded-a-social-psychological-...
    - gbear605 7y ago
      The problem is that when you run a/b tests, the changed case is often worse. If the thing being tested is a website, that might not be so bad. If the thing being tested is your education, that might be horrible.
      - mygo 7y ago
        No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. For example, FSU recently went up in rankings from a ~#50 public university to a #18 public university year over year over the past decade. Whatever changes they put into place are now being studied by other universities and if there’s anything novel that they’re doing, it will likely become more widespread as other universities seeking higher rankings implement new things. Or, it could be possible that the university studied the success of other universities and implemented what worked best and stopped what wasn’t. You run an a/b test when there aren’t already tens of thousands of other colleges and universities like yourself out there that you can study and learn from. Of course, rankings DNE educational quality, but it’s one example. Another example, that may have a better correlation with educational quality (although part of the equation is the constitution of the cohort) may be bar exam pass rates for law school grads listed by law school.
        dr_dshiv 7y ago
        > No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. Why would it be any different in webtech or advertising? Why wouldn't everyone just copy the best vs gathering empirical data?
        mygo 7y ago
        Are universities not gathering empirical data when they study the results of others’ efforts at scale? Study being the key word in that scenario, it’s more than just copying what they see, but understanding why it’s effective, and if it would be for them. And it may be quicker and more reliable with less risk than A/B testing in their setting. The most highly trafficked web tech companies can gather statistically significant feedback data about a change in moments. A/B testing a curriculum or educational practice could take a semester or more, and then the risks are higher — it would be a two-sided hypothesis test where the B group could not only do better, but could also do much worse, and it would reflect poorly on the institution if so. People are paying tens of thousands of dollars per seat per year for the best education they can get. Seeing how many items are in the shopping cart right on the “checkout” button doesn’t really reflect poorly on Amazon, but it could help Amazon increase conversion rates by .03%, which could mean millions of dollars at their scale, and they could also complete the test fairly quickly given their volume (in a day or so?) at a 99.7% confidence interval. With that being said, I’m sure that smaller scale or faster turnaround time A/B tests are being ran at Uni’s.

by knzhou 7y ago

c3534l 7y ago

Ha! I can safely say that companies do not, in general, learn their lesson until it becomes an existential threat.

jordanbeiber 7y ago
Companies are often filled with humans. It’s a human thing.

nostrademons 7y ago

You can bet that there are a lot of social science researchers within Facebook (I know at least two), but they've learned to never ever publish the results of that research and only use it to increase revenue.

dr_dshiv 7y ago
This a million times. All the public hysteria does is punish publication. The work goes on.
- kohtatsu 7y ago
  It also leads people to leave the platform.
- dr_dshiv 7y ago
  Hysteria around product experiments has had a big effect in education. As we all know, most online software companies conduct A/B tests to improve outcomes. IMO, education companies should do a/b testing in order to optimize student outcomes. It is unethical to not run tests. But, this can be described as "psychological manipulation" and "turning children into guinea pigs" -- and people freak. After all, there is not informed consent in A/B tests. Since educational software optimization does not actually lead to greater profit (the edu market is weird) the optimization work stops and no science is published. I find this disturbing. https://gizmodo.com/pearson-embedded-a-social-psychological-experiment-in-s-1825367784 https://gizmodo.com/pearson-embedded-a-social-psychological-...
  - gbear605 7y ago
    The problem is that when you run a/b tests, the changed case is often worse. If the thing being tested is a website, that might not be so bad. If the thing being tested is your education, that might be horrible.
    - mygo 7y ago
      No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. For example, FSU recently went up in rankings from a ~#50 public university to a #18 public university year over year over the past decade. Whatever changes they put into place are now being studied by other universities and if there’s anything novel that they’re doing, it will likely become more widespread as other universities seeking higher rankings implement new things. Or, it could be possible that the university studied the success of other universities and implemented what worked best and stopped what wasn’t. You run an a/b test when there aren’t already tens of thousands of other colleges and universities like yourself out there that you can study and learn from. Of course, rankings DNE educational quality, but it’s one example. Another example, that may have a better correlation with educational quality (although part of the equation is the constitution of the cohort) may be bar exam pass rates for law school grads listed by law school.
      - dr_dshiv 7y ago
        > No need to a/b test when there are tens of thousands of universities in the world all doing slightly different things. The cream rises and then is copied. Why would it be any different in webtech or advertising? Why wouldn't everyone just copy the best vs gathering empirical data?
        mygo 7y ago
        Are universities not gathering empirical data when they study the results of others’ efforts at scale? Study being the key word in that scenario, it’s more than just copying what they see, but understanding why it’s effective, and if it would be for them. And it may be quicker and more reliable with less risk than A/B testing in their setting. The most highly trafficked web tech companies can gather statistically significant feedback data about a change in moments. A/B testing a curriculum or educational practice could take a semester or more, and then the risks are higher — it would be a two-sided hypothesis test where the B group could not only do better, but could also do much worse, and it would reflect poorly on the institution if so. People are paying tens of thousands of dollars per seat per year for the best education they can get. Seeing how many items are in the shopping cart right on the “checkout” button doesn’t really reflect poorly on Amazon, but it could help Amazon increase conversion rates by .03%, which could mean millions of dollars at their scale, and they could also complete the test fairly quickly given their volume (in a day or so?) at a 99.7% confidence interval. With that being said, I’m sure that smaller scale or faster turnaround time A/B tests are being ran at Uni’s.