7 ms·

I genuinely believe that AB testing as practiced by every major tech company (and the incentives it creates) has caused much of this. AB testing is a really uns

I genuinely believe that AB testing as practiced by every major tech company (and the incentives it creates) has caused much of this. AB testing is a really unsophisticated way of measuring extremely short-term incremental gains in isolation. That's how we've ended up with products which have gradually become so hostile to users. Because individually the changes seem like small wins, but taken together and over the long term they are deeply destructive to the core product. AB testing doesn't capture long-term changes to user behavior from the accumulation of disparate features. The incentive structures within these companies are all set up to be able to prove these small wins in dollar values to get a raise or promotion which has led to a race to the bottom as teams create features nobody wants to justify their existence. Anecdotally I and many people I know have reached a kind of breaking point where these apps have become so demanding in terms of attention that we have to stop using them entirely. Whether it's YouTube's incessant advertisements, TikTok's infinite scroll or Facebook's insane jumble of a UI/feed these products are demonstrably inferior to their incarnations 5-7 years ago.

throwaway0asd 4y ago
I was the A/B test engineer for one of the most well known dot com brands about a decade ago. A/B testing media (advertising) is extremely misleading compared to testing against something transactional. The goal of a transactional sequence, called a conversion (payment in exchange for a product or defined service), is a supremely simple metric. Although the goal is simple to define and recognize the costs associated are wildly complex. Media is the inverse of this where the goals and consequences are not immediately clear or known but both the costs and revenue are immediately straight forward. The reason the financials of conversion are complex is because you need to account for the expenses to third parties to pull a user in, the purchasing horizon in time, potential for cannibalism against other purchasing decisions, and various other factors that cost the business money. The reason metrics associated with media are complex is because users HATE it. Increased advertising presence drives traffic away. Shifts in traffic occur over a wide duration due to a variety of factors so it is almost impossible to measure just how toxic advertising is to a given product oriented business in isolation in accordance with a variable time horizon. Its actually much worse than that for a variety of technical factors. In the past legitimate ad placements have been delivery vectors for content and logic of malicious criminal activity. Advertising logic is frequently poorly written. Many websites will isolate their advertisements into iframes in order to protect their website from accidental defacement of presentation and/or broken logic. Advertisements come into the page super slowly which completely pushes out the waterfall for what might have been a quick loading page. When I did A/B testing I would describe the media side of the business in terms of illegal job deals. I did this because of so many parallels in the transactions and risks in those markets, not because they are both filthy evil things.
- badtension 4y ago
  I can't say I understood everything you said (is all usage of word "media" == "advertising"? what are "illegal job deals"?) but just wanted to ask: is A/B testing ever not harmful to end users in the long run? Excluding some very minor UI testing like CTA button color which I presume doesn't really matter.
  - throwaway0asd 4y ago
    From a revenue perspective the words media and advertising are interchangeable. The word media is preferred because it speaks to a business segment as opposed to something like an agency. A/B testing is great when you are helping eliminate confusion and dark patterns nobody can articulate more directly. Illegal job deals was an autocorrection of my phone. I meant illegal drug deals and didn’t catch the change.
    - badtension 4y ago
      Thank you, I have been reading about dark patterns recently and how damaging they are so you really got my attention now. I will try to do my own research on how it's done but if you can expand a bit on confusion and dark pattern elimination that would be very much appreciated. I feel like everything I ever stumbled upon regarding A/B testing was "sales funnel optimization" which seems to be the root of much evil.
      - throwaway0asd 4y ago
        The goal of any product oriented business is purchase through a point of sale. That is called a conversion. Dark patterns emerge when a seller attempts to influence the behavior of a purchaser beyond decisions most beneficial to the purchaser, which is not always intentional. One example of dark pattern elimination occurs when steps in a purchase process are dramatically reduced. That reduction eliminates opportunities for messaging but increases the potential for transaction quantity.
        4y ago

throwaway0asd 4y ago

I was the A/B test engineer for one of the most well known dot com brands about a decade ago. A/B testing media (advertising) is extremely misleading compared to testing against something transactional. The goal of a transactional sequence, called a conversion (payment in exchange for a product or defined service), is a supremely simple metric. Although the goal is simple to define and recognize the costs associated are wildly complex. Media is the inverse of this where the goals and consequences are not immediately clear or known but both the costs and revenue are immediately straight forward. The reason the financials of conversion are complex is because you need to account for the expenses to third parties to pull a user in, the purchasing horizon in time, potential for cannibalism against other purchasing decisions, and various other factors that cost the business money. The reason metrics associated with media are complex is because users HATE it. Increased advertising presence drives traffic away. Shifts in traffic occur over a wide duration due to a variety of factors so it is almost impossible to measure just how toxic advertising is to a given product oriented business in isolation in accordance with a variable time horizon. Its actually much worse than that for a variety of technical factors. In the past legitimate ad placements have been delivery vectors for content and logic of malicious criminal activity. Advertising logic is frequently poorly written. Many websites will isolate their advertisements into iframes in order to protect their website from accidental defacement of presentation and/or broken logic. Advertisements come into the page super slowly which completely pushes out the waterfall for what might have been a quick loading page. When I did A/B testing I would describe the media side of the business in terms of illegal job deals. I did this because of so many parallels in the transactions and risks in those markets, not because they are both filthy evil things.

badtension 4y ago
I can't say I understood everything you said (is all usage of word "media" == "advertising"? what are "illegal job deals"?) but just wanted to ask: is A/B testing ever not harmful to end users in the long run? Excluding some very minor UI testing like CTA button color which I presume doesn't really matter.
- throwaway0asd 4y ago
  From a revenue perspective the words media and advertising are interchangeable. The word media is preferred because it speaks to a business segment as opposed to something like an agency. A/B testing is great when you are helping eliminate confusion and dark patterns nobody can articulate more directly. Illegal job deals was an autocorrection of my phone. I meant illegal drug deals and didn’t catch the change.
  - badtension 4y ago
    Thank you, I have been reading about dark patterns recently and how damaging they are so you really got my attention now. I will try to do my own research on how it's done but if you can expand a bit on confusion and dark pattern elimination that would be very much appreciated. I feel like everything I ever stumbled upon regarding A/B testing was "sales funnel optimization" which seems to be the root of much evil.
    - throwaway0asd 4y ago
      The goal of any product oriented business is purchase through a point of sale. That is called a conversion. Dark patterns emerge when a seller attempts to influence the behavior of a purchaser beyond decisions most beneficial to the purchaser, which is not always intentional. One example of dark pattern elimination occurs when steps in a purchase process are dramatically reduced. That reduction eliminates opportunities for messaging but increases the potential for transaction quantity.
      - 4y ago