2 ms·

A/B tests are pretty problematic for social networks, since anything effective can also influence the control group. They are excellent for single-player activi

by sayrer 4y ago

A/B tests are pretty problematic for social networks, since anything effective can also influence the control group. They are excellent for single-player activities like shopping carts and signup flows. "Network Experimentation at Scale" from Facebook describes how difficult this problem is. Most A/B test frameworks don't reach this level of sophistication. It does make some sense to just ship things if you don't have time to build out something like that. (disclosure: I worked at Twitter long ago)

throwawaymaths 4y ago
They're also problematic because you'll likely optimize for whatever metric you pick with no regard to the ethics of the optimization. Sometimes a double blind test is not the right thing to do.
- sayrer 4y ago
  I've never really bought this argument. There might be unethical growth hackers in the company, but A/B tests are not the problem in that situation. In my experience, they mostly just catch bugs. Stuff like "hmm, our much better looking signup flow underperforms... oh, the form is broken on Safari." That kind of thing.
  - solarkraft 4y ago
    Think of the AI optimizing for a stated goal while ignoring implied constraints (e.g. eradicating humanity to stop wars). This can be the case for A/B testing. Sure, you can increase ad clicks by 30% ... if you trick the user into clicking it through a carefully timed layout jump. I think GP's argumentation may go in this direction. I'd probably not say A/B testing is the problem itself, it is a tool after all, but I could imagine it's sometimes not used very well. Another point: Spotify's core flow changes so much (feels like almost daily) that I've lost all confidence in using it.
  - P5fRxh5kUvp2th 4y ago
    To make it easier for everyone to understand the other posters point... a double blind test can help you determine the most effective way to cause pain to a monkey, but it will never answer the question of whether you should be doing so.

by sayrer 4y ago

throwawaymaths 4y ago

They're also problematic because you'll likely optimize for whatever metric you pick with no regard to the ethics of the optimization. Sometimes a double blind test is not the right thing to do.

sayrer 4y ago
I've never really bought this argument. There might be unethical growth hackers in the company, but A/B tests are not the problem in that situation. In my experience, they mostly just catch bugs. Stuff like "hmm, our much better looking signup flow underperforms... oh, the form is broken on Safari." That kind of thing.
- solarkraft 4y ago
  Think of the AI optimizing for a stated goal while ignoring implied constraints (e.g. eradicating humanity to stop wars). This can be the case for A/B testing. Sure, you can increase ad clicks by 30% ... if you trick the user into clicking it through a carefully timed layout jump. I think GP's argumentation may go in this direction. I'd probably not say A/B testing is the problem itself, it is a tool after all, but I could imagine it's sometimes not used very well. Another point: Spotify's core flow changes so much (feels like almost daily) that I've lost all confidence in using it.
- P5fRxh5kUvp2th 4y ago
  To make it easier for everyone to understand the other posters point... a double blind test can help you determine the most effective way to cause pain to a monkey, but it will never answer the question of whether you should be doing so.