3 ms·

> That's the whole shell game of A/B testing. You're testing your way to what works best for you and not your users. A/B testing is a tool. Any tool can be mis

by nkrisc 4y ago

> That's the whole shell game of A/B testing. You're testing your way to what works best for you and not your users. A/B testing is a tool. Any tool can be misused.

FridgeSeal 4y ago
On one hand: If a tool is being constantly and egregiously misused at large scale, might it not be reasonable to assign a degree of blame to the tool (e.g. tool has a “pit of success” that encourages/permits misuse. On the other hand: marketing and execs could misuse and abuse an empty room if they do desired, so I wonder why we bother giving them tools in the first place.
- nkrisc 4y ago
  > If a tool is being constantly and egregiously misused at large scale That’s quite the claim.
- hellweaver666 4y ago
  Cars kill people every day. Is the car to blame?
Moru 4y ago
A tool that tests a change that puts off a person permanently won't give a good result if that person never comes back. It's often better to think through and ask a few users before starting developement. If your change is made only to lure people in, not improve their life in any way, mabe you should not even start the A/B testing at all...
- nkrisc 4y ago
  > If your change is made only to lure people in, not improve their life in any way, mabe you should not even start the A/B testing at all... Yes. It is totally possible to misuse the tool and run bad tests. That’s why, at least where I work, we don’t run live A/B tests until we’ve done some initial validation that the idea isn’t just terrible, and we have some level of confidence it will work and not harm the user’s experience. For example, we could segment user feedback and customer support issues we received by test variant so we could look for unexpected issues we might have inadvertently caused with the challenger design. I can’t speak for how A/B testing is used elsewhere, but every A/B test I’ve run has been because we thought it would improve things for the user. You certainly could run an A/B test to see which banner ad is more deceitful, just like you could use React to build the website that sells the scam medicine that banner ad is advertising.

by nkrisc 4y ago

> That's the whole shell game of A/B testing. You're testing your way to what works best for you and not your users. A/B testing is a tool. Any tool can be misused.

FridgeSeal 4y ago

On one hand: If a tool is being constantly and egregiously misused at large scale, might it not be reasonable to assign a degree of blame to the tool (e.g. tool has a “pit of success” that encourages/permits misuse. On the other hand: marketing and execs could misuse and abuse an empty room if they do desired, so I wonder why we bother giving them tools in the first place.

nkrisc 4y ago
> If a tool is being constantly and egregiously misused at large scale That’s quite the claim.
hellweaver666 4y ago
Cars kill people every day. Is the car to blame?

Moru 4y ago

A tool that tests a change that puts off a person permanently won't give a good result if that person never comes back. It's often better to think through and ask a few users before starting developement. If your change is made only to lure people in, not improve their life in any way, mabe you should not even start the A/B testing at all...

nkrisc 4y ago
> If your change is made only to lure people in, not improve their life in any way, mabe you should not even start the A/B testing at all... Yes. It is totally possible to misuse the tool and run bad tests. That’s why, at least where I work, we don’t run live A/B tests until we’ve done some initial validation that the idea isn’t just terrible, and we have some level of confidence it will work and not harm the user’s experience. For example, we could segment user feedback and customer support issues we received by test variant so we could look for unexpected issues we might have inadvertently caused with the challenger design. I can’t speak for how A/B testing is used elsewhere, but every A/B test I’ve run has been because we thought it would improve things for the user. You certainly could run an A/B test to see which banner ad is more deceitful, just like you could use React to build the website that sells the scam medicine that banner ad is advertising.