4 ms·

I was the A/B test engineer for one of the most well known dot com brands about a decade ago. A/B testing media (advertising) is extremely misleading compared

I was the A/B test engineer for one of the most well known dot com brands about a decade ago. A/B testing media (advertising) is extremely misleading compared to testing against something transactional. The goal of a transactional sequence, called a conversion (payment in exchange for a product or defined service), is a supremely simple metric. Although the goal is simple to define and recognize the costs associated are wildly complex. Media is the inverse of this where the goals and consequences are not immediately clear or known but both the costs and revenue are immediately straight forward. The reason the financials of conversion are complex is because you need to account for the expenses to third parties to pull a user in, the purchasing horizon in time, potential for cannibalism against other purchasing decisions, and various other factors that cost the business money. The reason metrics associated with media are complex is because users HATE it. Increased advertising presence drives traffic away. Shifts in traffic occur over a wide duration due to a variety of factors so it is almost impossible to measure just how toxic advertising is to a given product oriented business in isolation in accordance with a variable time horizon. Its actually much worse than that for a variety of technical factors. In the past legitimate ad placements have been delivery vectors for content and logic of malicious criminal activity. Advertising logic is frequently poorly written. Many websites will isolate their advertisements into iframes in order to protect their website from accidental defacement of presentation and/or broken logic. Advertisements come into the page super slowly which completely pushes out the waterfall for what might have been a quick loading page. When I did A/B testing I would describe the media side of the business in terms of illegal job deals. I did this because of so many parallels in the transactions and risks in those markets, not because they are both filthy evil things.

badtension 4y ago
I can't say I understood everything you said (is all usage of word "media" == "advertising"? what are "illegal job deals"?) but just wanted to ask: is A/B testing ever not harmful to end users in the long run? Excluding some very minor UI testing like CTA button color which I presume doesn't really matter.
- throwaway0asd 4y ago
  From a revenue perspective the words media and advertising are interchangeable. The word media is preferred because it speaks to a business segment as opposed to something like an agency. A/B testing is great when you are helping eliminate confusion and dark patterns nobody can articulate more directly. Illegal job deals was an autocorrection of my phone. I meant illegal drug deals and didn’t catch the change.
  - badtension 4y ago
    Thank you, I have been reading about dark patterns recently and how damaging they are so you really got my attention now. I will try to do my own research on how it's done but if you can expand a bit on confusion and dark pattern elimination that would be very much appreciated. I feel like everything I ever stumbled upon regarding A/B testing was "sales funnel optimization" which seems to be the root of much evil.
    - throwaway0asd 4y ago
      The goal of any product oriented business is purchase through a point of sale. That is called a conversion. Dark patterns emerge when a seller attempts to influence the behavior of a purchaser beyond decisions most beneficial to the purchaser, which is not always intentional. One example of dark pattern elimination occurs when steps in a purchase process are dramatically reduced. That reduction eliminates opportunities for messaging but increases the potential for transaction quantity.
      - badtension 4y ago
        Ok I have a clearer picture now but it's still something that would probably need case by case analysis to get a full grasp what is possible and how to use it for the customer and not against him. Thank you.

badtension 4y ago

I can't say I understood everything you said (is all usage of word "media" == "advertising"? what are "illegal job deals"?) but just wanted to ask: is A/B testing ever not harmful to end users in the long run? Excluding some very minor UI testing like CTA button color which I presume doesn't really matter.

throwaway0asd 4y ago
From a revenue perspective the words media and advertising are interchangeable. The word media is preferred because it speaks to a business segment as opposed to something like an agency. A/B testing is great when you are helping eliminate confusion and dark patterns nobody can articulate more directly. Illegal job deals was an autocorrection of my phone. I meant illegal drug deals and didn’t catch the change.
- badtension 4y ago
  Thank you, I have been reading about dark patterns recently and how damaging they are so you really got my attention now. I will try to do my own research on how it's done but if you can expand a bit on confusion and dark pattern elimination that would be very much appreciated. I feel like everything I ever stumbled upon regarding A/B testing was "sales funnel optimization" which seems to be the root of much evil.
  - throwaway0asd 4y ago
    The goal of any product oriented business is purchase through a point of sale. That is called a conversion. Dark patterns emerge when a seller attempts to influence the behavior of a purchaser beyond decisions most beneficial to the purchaser, which is not always intentional. One example of dark pattern elimination occurs when steps in a purchase process are dramatically reduced. That reduction eliminates opportunities for messaging but increases the potential for transaction quantity.
    - badtension 4y ago
      Ok I have a clearer picture now but it's still something that would probably need case by case analysis to get a full grasp what is possible and how to use it for the customer and not against him. Thank you.