4 ms·

Have you ever worked at a large corporation like Facebook or Amazon? They have extensive internal A/B testing frameworks that record a wide variety of user metr

by rp1 5y ago

Have you ever worked at a large corporation like Facebook or Amazon? They have extensive internal A/B testing frameworks that record a wide variety of user metrics for arbitrary control and treatment exposures. It's very easy to set up an A/B test and pretty much necessary for almost all changes. Additionally, employees are expected to have concrete artifacts when they go up for promotion, and A/B test results are often a large part of that.

PaulDavisThe1st 5y ago
> Have you ever worked at a large corporation like Facebook or Amazon? Employee #2 at amzn. But sure, that doesn't count because I got out before the company hit 20 employees. I don't doubt that large corporations with heavy technology dependence engage in what you're describing to cover some aspects of their operations. I do doubt that is covers all aspects, and in particular, I very much doubt if amzn A/B tested the language described in the GP for the purposes claimed by the parent of my comment. EDIT: Given the comment @ https://news.ycombinator.com/item?id=29374358 https://news.ycombinator.com/item?id=29374358 I should probably just accept that for amzn at least, I'm just wrong.
- rp1 5y ago
  That's pretty impressive! Having worked at Amazon much later, I do think that most copy changes do go through an A/B test. I remember an instance where we tested 20 different versions of a minor UI change. Interestingly, there was a wide spread in the performance of the different treatments.
  - vhgyu75e6u 5y ago
    Current amzn employee working on the retail website. Unless a team is jumping/ignoring all standard operation precedures, everything must go through an A/B test to prove that it works at increasing the revenue or whatever the KPI target is. This includes text that might need to go through legal in some cases
    - PaulDavisThe1st 5y ago
      Fair enough then. I'm wrong, for amzn anyway.
      - version_five 5y ago
        I think to some extent you and the original post you replied to are both probably right. There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant
        stcredzero 5y ago
        the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences. I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.
        GekkePrutser 5y ago
        > I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures. I think it'll be by just sitting back and waiting a few decades :)
        stcredzero 5y ago
        I think it'll be by just sitting back and waiting a few decades :) The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.
        teddyh 5y ago

by rp1 5y ago

PaulDavisThe1st 5y ago

> Have you ever worked at a large corporation like Facebook or Amazon? Employee #2 at amzn. But sure, that doesn't count because I got out before the company hit 20 employees. I don't doubt that large corporations with heavy technology dependence engage in what you're describing to cover some aspects of their operations. I do doubt that is covers all aspects, and in particular, I very much doubt if amzn A/B tested the language described in the GP for the purposes claimed by the parent of my comment. EDIT: Given the comment @ https://news.ycombinator.com/item?id=29374358 https://news.ycombinator.com/item?id=29374358 I should probably just accept that for amzn at least, I'm just wrong.

rp1 5y ago
That's pretty impressive! Having worked at Amazon much later, I do think that most copy changes do go through an A/B test. I remember an instance where we tested 20 different versions of a minor UI change. Interestingly, there was a wide spread in the performance of the different treatments.
- vhgyu75e6u 5y ago
  Current amzn employee working on the retail website. Unless a team is jumping/ignoring all standard operation precedures, everything must go through an A/B test to prove that it works at increasing the revenue or whatever the KPI target is. This includes text that might need to go through legal in some cases
  - PaulDavisThe1st 5y ago
    Fair enough then. I'm wrong, for amzn anyway.
    - version_five 5y ago
      I think to some extent you and the original post you replied to are both probably right. There is not some painstaking language optimization designed to trick users. But there is not a filter for clarity either, so the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences, and gives the impression that Amazon et al are deliberately hostile when they are really just ignorant
      - stcredzero 5y ago
        the A/B testing naturally favors "dark patterns" - it's not malicious design, it's just unintended consequences Could there be such a thing as "meta-malicious design?" Bad consequences aren't executed directly, but conditions are put in place that cause the emergence of bad consequences. I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures.
        GekkePrutser 5y ago
        > I suspect, if AI ever does away with Homo sapiens it will be by convincing us not to breed by changing our incentive structures. I think it'll be by just sitting back and waiting a few decades :)
        stcredzero 5y ago
        I think it'll be by just sitting back and waiting a few decades :) The sci-fi tinfoil hat interpretation of this: It's already here, or at least its cronies and enablers, and the plan has already been put into effect.
        teddyh 5y ago