3 ms·

Framing A/B testing as "social experiments" for shock value is disingenuous, but it is interesting to think about the real-world knock-on effects that recommend

by mjr00 4y ago

Framing A/B testing as "social experiments" for shock value is disingenuous, but it is interesting to think about the real-world knock-on effects that recommendation engines have. In this instance, recommendation engines helped people get jobs. If you're a relatively unknown musician on Spotify, being "blessed by the algorithm" and put into algorithmic playlists associated with more popular artist translates to real-life streams and income. If you're an unknown artist pushing work on Twitter or a wannabe "influencer" on Instagram/Tiktok/YT, showing up in the algofeed and getting a bunch of views, likes, and reposts could be what makes you "go viral" and start your career. There's a cottage industry of people making a career out of posting content on platforms, or trying to, and one of the biggest factors is whether or not these opaque, vaguely understood algorithms happen to select your content to be recommended. Crazy as it may seem, deploying a retrained recommendation ML model could impact whether someone makes rent this month.

SkyBelow 4y ago
>Framing A/B testing as "social experiments" for shock value is disingenuous, Is it? I've only done minor human experimentation, but had to go through the IRB process. What I did was on par with simple AB testing. The only difference here is that the AB testing is not being done to be published and not under the purview of the IRB. If it were, they would require the same sort of approve for human experimentation. Sure, the paperwork is nothing like a drug test, and I think the IRB is extra heavy weighted on the side of seeking too much approval, but there is a history of human experimentation to explain why that is the case. That new unmonitored experiments are happening under tech employees running loose instead of scientists running loose doesn't really indicate any significant difference and we should be watchful that similar abuses of power as had previously happened don't reoccur. I think we've seen enough with facebook and their influence on children to know it is a real concern and we might already be far enough along the path to be worth raising the alarm.
- BurningFrog 4y ago
  IRBs are a substantial break on scientific progress today, and need massive reform. Instituted to prevent repeats of the Tuskegee Syphilis Study, they now claim (slow) pre-approval rights to survey questions. To me, it shows that gate-keepers are going to gate-keep. Institutions always strive to expand their powers.
somenameforme 4y ago
Consider a slot machine company simply deploying an A/B test on the literal bells and whistles of their latest one armed bandit. They find that one setup increased any-word-except-addiction to a higher degree than the other, and so went with it. Would you consider this a social experiment? I ask because I am trying to understand where your objection is coming from. And if you would consider the above a social experiment, might it not simply be desensitization? A/B testing has become a somewhat regular part of 'engagement optimization strategies' at many companies which sounds perfectly vanilla, yet in many cases engagement is just the software word of choice in the above fill-in-the-blank.

by mjr00 4y ago

SkyBelow 4y ago

>Framing A/B testing as "social experiments" for shock value is disingenuous, Is it? I've only done minor human experimentation, but had to go through the IRB process. What I did was on par with simple AB testing. The only difference here is that the AB testing is not being done to be published and not under the purview of the IRB. If it were, they would require the same sort of approve for human experimentation. Sure, the paperwork is nothing like a drug test, and I think the IRB is extra heavy weighted on the side of seeking too much approval, but there is a history of human experimentation to explain why that is the case. That new unmonitored experiments are happening under tech employees running loose instead of scientists running loose doesn't really indicate any significant difference and we should be watchful that similar abuses of power as had previously happened don't reoccur. I think we've seen enough with facebook and their influence on children to know it is a real concern and we might already be far enough along the path to be worth raising the alarm.

BurningFrog 4y ago
IRBs are a substantial break on scientific progress today, and need massive reform. Instituted to prevent repeats of the Tuskegee Syphilis Study, they now claim (slow) pre-approval rights to survey questions. To me, it shows that gate-keepers are going to gate-keep. Institutions always strive to expand their powers.

somenameforme 4y ago

Consider a slot machine company simply deploying an A/B test on the literal bells and whistles of their latest one armed bandit. They find that one setup increased any-word-except-addiction to a higher degree than the other, and so went with it. Would you consider this a social experiment? I ask because I am trying to understand where your objection is coming from. And if you would consider the above a social experiment, might it not simply be desensitization? A/B testing has become a somewhat regular part of 'engagement optimization strategies' at many companies which sounds perfectly vanilla, yet in many cases engagement is just the software word of choice in the above fill-in-the-blank.