3 ms·

I hope everyone down voting this realizes that A/B testing has an "infinite" resolution. Most people when they hear A/B testing think of something like switchi

I hope everyone down voting this realizes that A/B testing has an "infinite" resolution. Most people when they hear A/B testing think of something like switching the color of a button, or what have you. But when you start A/B testing a complex series of permutations of components in aggregate, specifically designed to target certain psychological profiles, you can actually start to learn a lot about the group you're testing. To pretend that A/B testing is some simple "what image do you like more?" game is entirely disingenuous and is so characteristic of the attitude people in tech have when dealing with people or any kind of social side effects of systems they build. We absolutely do need to be very careful with this. Careful with how we test people, careful with how the information is recorded, and careful with what we do with the data. I'm going to start calling A/B testing "psychological side channel attacks" and maybe then places like HN will appreciate what's happening more.

Zak 8y ago
I think people on HN generally have a good understanding of what A/B testing is: "if I change X, will users do more of Y?" Of course Y is usually something that makes the site owner money. Sometimes A, B, both, or the entire business model of the site is unethical. In such cases, they would still be unethical if a comparison test was not run. In cases where none of them are unethical, I have trouble imagining a realistic scenario in which the act of running a comparison test makes it unethical.
- kadendogthing 8y ago
  >I think people on HN generally have a good understanding of what A/B testing is Not only do I not believe this, as most people on HN tend to have a very superficial understanding of whatever tech is being discussed, there is always a dismissal of any social consequences that may happen as a result of using any kind of technology. >I have trouble imagining a realistic scenario in which the act of running a comparison test makes it unethical. I can't tell if you're being serious or not. You don't need any kind of testing framework for a few famous examples to satisfy your "realistic" qualification. I see a lot of bland contrarian stuff on HN, but I'm kind of speechless right now.
  - Zak 8y ago
    I'm entirely serious. To cite one of the most famous examples, Facebook ran a sort of A/B test where its algorithm was adjusted to attempt to make users happy or sad. There's a fairly strong case to be made that intentionally making a large number of people sad just to see if you can is unethical. There's a somewhat weaker case to be made that manipulating the happy group was also an unethical distortion of their reality. I fail to see an ethical problem with the fact that it was an A/B comparison. Instead, A and probably B would be unethical to attempt under any conditions without consent.

Zak 8y ago

I think people on HN generally have a good understanding of what A/B testing is: "if I change X, will users do more of Y?" Of course Y is usually something that makes the site owner money. Sometimes A, B, both, or the entire business model of the site is unethical. In such cases, they would still be unethical if a comparison test was not run. In cases where none of them are unethical, I have trouble imagining a realistic scenario in which the act of running a comparison test makes it unethical.

kadendogthing 8y ago
>I think people on HN generally have a good understanding of what A/B testing is Not only do I not believe this, as most people on HN tend to have a very superficial understanding of whatever tech is being discussed, there is always a dismissal of any social consequences that may happen as a result of using any kind of technology. >I have trouble imagining a realistic scenario in which the act of running a comparison test makes it unethical. I can't tell if you're being serious or not. You don't need any kind of testing framework for a few famous examples to satisfy your "realistic" qualification. I see a lot of bland contrarian stuff on HN, but I'm kind of speechless right now.
- Zak 8y ago
  I'm entirely serious. To cite one of the most famous examples, Facebook ran a sort of A/B test where its algorithm was adjusted to attempt to make users happy or sad. There's a fairly strong case to be made that intentionally making a large number of people sad just to see if you can is unethical. There's a somewhat weaker case to be made that manipulating the happy group was also an unethical distortion of their reality. I fail to see an ethical problem with the fact that it was an A/B comparison. Instead, A and probably B would be unethical to attempt under any conditions without consent.