4 ms·

That's pretty impressive! Having worked at Amazon much later, I do think that most copy changes do go through an A/B test. I remember an instance where we teste

by rp1 5y ago

That's pretty impressive! Having worked at Amazon much later, I do think that most copy changes do go through an A/B test. I remember an instance where we tested 20 different versions of a minor UI change. Interestingly, there was a wide spread in the performance of the different treatments.

vhgyu75e6u 5y ago
Current amzn employee working on the retail website. Unless a team is jumping/ignoring all standard operation precedures, everything must go through an A/B test to prove that it works at increasing the revenue or whatever the KPI target is. This includes text that might need to go through legal in some cases
- PaulDavisThe1st 5y ago
  Fair enough then. I'm wrong, for amzn anyway.
  - version_five 5y ago
    I think to some extent you and the original post you replied to are both probably right. There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant
    - stcredzero 5y ago
      the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences. I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.
      - GekkePrutser 5y ago
        > I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures. I think it'll be by just sitting back and waiting a few decades :)
        stcredzero 5y ago
        I think it'll be by just sitting back and waiting a few decades :) The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.
      - teddyh 5y ago
        “Bad men need nothing more to compass their ends, than that good men should look on and do nothing.” — John Stuart Mill, 1867
        stcredzero 5y ago
        I'm talking about the good people not even knowing what's happening, until they engage in data analysis.
    - kingcharles 5y ago
      Also, humans are weird, so A/B testing, if taken to its logical conclusion, will produce surreal results. What makes a human buy something is really not that well understood, IMO. For instance, I was in charge of design for a mortgage lender's web site. I begged my manager to let me try (for lulz) an A/B test replacing the photo of the woman call centre operator next to the quote form with a photo of a Labrador retriever wearing a headset (you knew he was a professional because he also wore glasses and a necktie). The dog form consistently outperformed the woman by 19% but we had to roll it back after a couple of weeks because.. well.. dog.
  - vhgyu75e6u 5y ago
    The other scenario is that that text is the first iteration and gave a positive KPI, if there are enough complains related to the structure of the text and is linked to increase in refunds, the team in charge would A/B another iteration targeting refund rate as KPI (although that might be difficult to correlate)
  - random314 5y ago
    It's TRUE for all large web tech companies. Everything is launched via A/B tests. Your performance evaluation as an engineer is dictated by how much your teams goal metrics moved thanks to your changes(impact measured by A/B tests). At any given time, a single engineer might have a couple of A/B tests running. That's the single biggest engineering change I have seen in the last 20 years. A/B tests are now crazy simple to set up; just click a few buttons and decide how much traffic to send. The major con is that it leaves a lot of dead code behind when the best of N A/B test arms are launched; the (N-1) unsuccessful arms stick around as dead code.

by rp1 5y ago

vhgyu75e6u 5y ago

Current amzn employee working on the retail website. Unless a team is jumping/ignoring all standard operation precedures, everything must go through an A/B test to prove that it works at increasing the revenue or whatever the KPI target is. This includes text that might need to go through legal in some cases

PaulDavisThe1st 5y ago
Fair enough then. I'm wrong, for amzn anyway.
- version_five 5y ago
  I think to some extent you and the original post you replied to are both probably right. There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant
  - stcredzero 5y ago
    the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences. I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.
    - GekkePrutser 5y ago
      > I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures. I think it'll be by just sitting back and waiting a few decades :)
      - stcredzero 5y ago
        I think it'll be by just sitting back and waiting a few decades :) The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.
    - teddyh 5y ago
      “Bad men need nothing more to compass their ends, than that good men should look on and do nothing.” — John Stuart Mill, 1867
      - stcredzero 5y ago
        I'm talking about the good people not even knowing what's happening, until they engage in data analysis.
  - kingcharles 5y ago
    Also, humans are weird, so A/B testing, if taken to its logical conclusion, will produce surreal results. What makes a human buy something is really not that well understood, IMO. For instance, I was in charge of design for a mortgage lender's web site. I begged my manager to let me try (for lulz) an A/B test replacing the photo of the woman call centre operator next to the quote form with a photo of a Labrador retriever wearing a headset (you knew he was a professional because he also wore glasses and a necktie). The dog form consistently outperformed the woman by 19% but we had to roll it back after a couple of weeks because.. well.. dog.
- vhgyu75e6u 5y ago
  The other scenario is that that text is the first iteration and gave a positive KPI, if there are enough complains related to the structure of the text and is linked to increase in refunds, the team in charge would A/B another iteration targeting refund rate as KPI (although that might be difficult to correlate)
- random314 5y ago
  It's TRUE for all large web tech companies. Everything is launched via A/B tests. Your performance evaluation as an engineer is dictated by how much your teams goal metrics moved thanks to your changes(impact measured by A/B tests). At any given time, a single engineer might have a couple of A/B tests running. That's the single biggest engineering change I have seen in the last 20 years. A/B tests are now crazy simple to set up; just click a few buttons and decide how much traffic to send. The major con is that it leaves a lot of dead code behind when the best of N A/B test arms are launched; the (N-1) unsuccessful arms stick around as dead code.