4 ms·

I can see how you don't want to be in a situation where you are holding in your hand evidence that Shady Marketing Tactic X is proven to convert better. But I t

I can see how you don't want to be in a situation where you are holding in your hand evidence that Shady Marketing Tactic X is proven to convert better. But I think the solution is simple: Don't A/B test shady features. If something makes you uncomfortable or seems like a dark pattern, don't test it, because you would never want it on your site anyway. I don't think this is a failing of A/B testing. That someone can get numbers that show "evil" tactics can make money is not materially different from someone coding "evil" HTML in Notepad and getting results. As you said, it is just a tool.

GhotiFish 13y ago
The problem is you might not be aware you're making a shady feature. As an example, if you are making an ad for your app fooer. You A/B test some banners on an ad distribution network. You find that a simpler ad with a "download now" hyperlink graphic proves effective. No big deal right? People are going to be seeing these banners on places like news sites right? It's not tricking people. It's also pretty reasonable right? People are used to clicking on hyperlinks, it overcomes the resistance to click on images. A reasonable move for a banner ad to make. Except the ad distribution network serves content sites like softpedia as well. You saw the spike because you emulated the download button right under the mirror list, and people were confused. They downloaded your application under the pretext of getting something else. Or what about if I want a "Hey you should sign up for notifications for other job offerings in this field" message box? I'd love to have one of those in my application. I A/B test to see that this message box drastically increases signups. Except it was because people thought it was a paywall. The road to hell is paved with good intentions. Consider this case about the Weebly blogging service: http://minimaxir.com/2013/05/overly-attached-startup/ http://minimaxir.com/2013/05/overly-attached-startup/ They were A/B testing messaging for first week signups. This hyper aggressive and insane route was actually considered. Do you think they're black hat? I don't think so, but what if this had become standard practice?

GhotiFish 13y ago

The problem is you might not be aware you're making a shady feature. As an example, if you are making an ad for your app fooer. You A/B test some banners on an ad distribution network. You find that a simpler ad with a "download now" hyperlink graphic proves effective. No big deal right? People are going to be seeing these banners on places like news sites right? It's not tricking people. It's also pretty reasonable right? People are used to clicking on hyperlinks, it overcomes the resistance to click on images. A reasonable move for a banner ad to make. Except the ad distribution network serves content sites like softpedia as well. You saw the spike because you emulated the download button right under the mirror list, and people were confused. They downloaded your application under the pretext of getting something else. Or what about if I want a "Hey you should sign up for notifications for other job offerings in this field" message box? I'd love to have one of those in my application. I A/B test to see that this message box drastically increases signups. Except it was because people thought it was a paywall. The road to hell is paved with good intentions. Consider this case about the Weebly blogging service: http://minimaxir.com/2013/05/overly-attached-startup/ http://minimaxir.com/2013/05/overly-attached-startup/ They were A/B testing messaging for first week signups. This hyper aggressive and insane route was actually considered. Do you think they're black hat? I don't think so, but what if this had become standard practice?