5 ms·

> The worst thing for me is how the Amazon search algorithm seems to want to show you everything but the item you searched for. Hey boss I made the site better

> The worst thing for me is how the Amazon search algorithm seems to want to show you everything but the item you searched for. Hey boss I made the site better! Through rigorous A/B testing I could figure out a way to tweak our search algorithm so people spend much more time on our site! It seems they now really enjoy browsing for products! Ok but seriously, I have witnessed A/B testing go wrong in the past so I'm biased to blame everything on it. I wouldn't think this particular thing happened though. :) What I could imagine is that they measure number of items bought or money spent, but even then if eg you don't also track how much of these people return stuff later you still might draw the wrong conclusions. Figuring out that a user is less likely to use your site six months down the line due to building frustration is even harder.

cle 3y ago
Amazon definitely tracks returns in their A/B tests, along with impact on long-term projections of customer value. What they also track is ad and sponsored products revenue. The sad truth with most Internet products is that advertisers are really good customers. They will pay you a lot of money with huge margins, and it's really hard for a business to say no to that.
matheusmoreira 3y ago
> rigorous A/B testing Also known as unethical, non-consensual human experimentation for profit maximization purposes.
- DiggyJohnson 3y ago
  I hate what advertising has done to the modern web just as much as anyone, but this strikes me as hyperbole. Does making this sort of claim not make you… tired? What’s the point of arguing like this? Nazi Germany and the Tuskegee Experiment are examples of “unethical, non-consensual human experimentation”. A/B testing features of software usually doesn’t make the same list.
  - asib 3y ago
    That's a bit of a false dichotomy. It doesn't have to be either Nazism or totally chill.
  - refulgentis 3y ago
    Godwin's Law.* If you're from a certain background it's exactly as described. In academia, frankly probably everywhere but tech, experiments as a term of art require consent when they involve humans. * n.b. you really should have left it out, it was a good post through "hyperbole", got close-minded in the next sentence, then just sort of blew the hatch doors off. Sometimes we just don't know something someone else knows. Not understanding someone else doesn't require they have a psychological condition, much less one worth noting.
    - mindcrime 3y ago
      Please stop. "Godwin's Law" is irrelevant bullshit. It's not a "law" and it doesn't prove anything, or do anything except add noise to the conversation.
      - basil-rash 3y ago
        Right? Parent is basically saying “wow it’s so unfortunate that you have forced me to end the conversation here, I’d have really liked to continue, but it’d be against the (entirely made up, by me) law”.
        refulgentis 3y ago
        Where? N.b. "godwin's Law" is the famous joke that as an Internet discussion approaches infinity length, hitler will be brought up Not a secret message saying discussion over lol
        basil-rash 3y ago
        The *
- whitexn--g28h 3y ago
  You are consenting to the experimentation by accessing the software. It’s covered by the terms and conditions.
  - esafak 3y ago
    Terms and conditions? It's hard to tell when people are joking. https://www.youtube.com/watch?v=AhsyZ5V1pok https://www.youtube.com/watch?v=AhsyZ5V1pok
  - matheusmoreira 3y ago
    > terms and conditions Nobody reads that stuff.
- stanleydrew 3y ago
  I don't expect you to back off of your take, but you should really consider how and why you came to this conclusion. If I put two different marketing messages on two different billboards to test whether one is more effective than the other, is that unethical non-consensual experimentation? If not, how is it different from A/B testing?
  - csydas 3y ago
    In strict terms yes, if you didn't get informed consent from your test subjects that would be unethical. Research has a lot of policies and systems set up to ensure that if your testing involves people, you must get informed consent from the persons before even trying to do the test, and it's really not hard to imagine why this is a stringent standard -- it's very easy to miss how "simple tests" can and often are adverse to those participating in the test or have unintended consequences that the researchers didn't accommodate for, regardless of the reason they did not. Ads are often portrayed as harmless but, like, there's a reason there are restrictions on advertising for certain highly addictive products and regulations against false or misleading advertising, or certain tactics aren't allowed.
    - jfim 3y ago
      I think that's the crux of the matter here. A/B testing can be anything from which page layout leads people to complete their shopping check out process to which ad campaign has the best ad click through rate. The former is pretty inoffensive, but the latter could be bad if it involves gambling/alcoholic beverage ads to people with gambling addiction or alcoholism, for example.
    - 13of40 3y ago
      > unethical If this is based on the possibility that one or more of the ads is harmful, how is it less ethical than the time-honored alternative, which is skipping the study and just running the ads?
      - matheusmoreira 3y ago
- erichanson 3y ago
  You know the Nazis had pieces of flair that they made the Jews wear.
- gopher_space 3y ago
  Narrowly true, but what's the difference between this and a diner trying out new blueberry pancake recipes?
  - mewpmewp2 3y ago
    It's unethical for a diner to try out new recipes. Per OpenAI policy you need to ask consent before trying out new recipes.
- concordDance 3y ago
  We do non-consensual human experimentation all the time. Whenever you try a new outfit you're doing it. It's an extremely broad category that contains good things (installing cycle lanes to see if they encourage cycling), neutral things (making both flower mugs and wave mugs and seeing which sells) and bad things (use your imagination).
- maxbond 3y ago
  I wouldn't make as strong a claim as the parent comment myself, but someone pointed out to me recently that A/B testing is really similar to cold reading. Is it morally equivalent to suggest to someone, in bad faith, that you're able to deliver messages from their dead loved ones, and to perform an A/B test of switching around menu items or change up some language to try and get fewer people to abandon their carts? I lean towards "no" but I have trouble either accepting or rejecting the proposition. It's hard for me to say that A/B testing is done in bad faith, but it's also hard for me to say it's entirely unmanipulative, either.
deleted 3y ago
[deleted]
xattt 3y ago
My best guess is the algorithm has been tweaked to return exact results maybe 1/10 or 1/20 times, like a slot machine with the psychological manipulation and “reward centre activation” that comes with it.

cle 3y ago

Amazon definitely tracks returns in their A/B tests, along with impact on long-term projections of customer value. What they also track is ad and sponsored products revenue. The sad truth with most Internet products is that advertisers are really good customers. They will pay you a lot of money with huge margins, and it's really hard for a business to say no to that.

matheusmoreira 3y ago

> rigorous A/B testing Also known as unethical, non-consensual human experimentation for profit maximization purposes.

DiggyJohnson 3y ago
I hate what advertising has done to the modern web just as much as anyone, but this strikes me as hyperbole. Does making this sort of claim not make you… tired? What’s the point of arguing like this? Nazi Germany and the Tuskegee Experiment are examples of “unethical, non-consensual human experimentation”. A/B testing features of software usually doesn’t make the same list.
- asib 3y ago
  That's a bit of a false dichotomy. It doesn't have to be either Nazism or totally chill.
- refulgentis 3y ago
  Godwin's Law.* If you're from a certain background it's exactly as described. In academia, frankly probably everywhere but tech, experiments as a term of art require consent when they involve humans. * n.b. you really should have left it out, it was a good post through "hyperbole", got close-minded in the next sentence, then just sort of blew the hatch doors off. Sometimes we just don't know something someone else knows. Not understanding someone else doesn't require they have a psychological condition, much less one worth noting.
  - mindcrime 3y ago
    Please stop. "Godwin's Law" is irrelevant bullshit. It's not a "law" and it doesn't prove anything, or do anything except add noise to the conversation.
    - basil-rash 3y ago
      Right? Parent is basically saying “wow it’s so unfortunate that you have forced me to end the conversation here, I’d have really liked to continue, but it’d be against the (entirely made up, by me) law”.
      - refulgentis 3y ago
        Where? N.b. "godwin's Law" is the famous joke that as an Internet discussion approaches infinity length, hitler will be brought up Not a secret message saying discussion over lol
        basil-rash 3y ago
        The *
whitexn--g28h 3y ago
You are consenting to the experimentation by accessing the software. It’s covered by the terms and conditions.
- esafak 3y ago
  Terms and conditions? It's hard to tell when people are joking. https://www.youtube.com/watch?v=AhsyZ5V1pok https://www.youtube.com/watch?v=AhsyZ5V1pok
- matheusmoreira 3y ago
  > terms and conditions Nobody reads that stuff.
stanleydrew 3y ago
I don't expect you to back off of your take, but you should really consider how and why you came to this conclusion. If I put two different marketing messages on two different billboards to test whether one is more effective than the other, is that unethical non-consensual experimentation? If not, how is it different from A/B testing?
- csydas 3y ago
  In strict terms yes, if you didn't get informed consent from your test subjects that would be unethical. Research has a lot of policies and systems set up to ensure that if your testing involves people, you must get informed consent from the persons before even trying to do the test, and it's really not hard to imagine why this is a stringent standard -- it's very easy to miss how "simple tests" can and often are adverse to those participating in the test or have unintended consequences that the researchers didn't accommodate for, regardless of the reason they did not. Ads are often portrayed as harmless but, like, there's a reason there are restrictions on advertising for certain highly addictive products and regulations against false or misleading advertising, or certain tactics aren't allowed.
  - jfim 3y ago
    I think that's the crux of the matter here. A/B testing can be anything from which page layout leads people to complete their shopping check out process to which ad campaign has the best ad click through rate. The former is pretty inoffensive, but the latter could be bad if it involves gambling/alcoholic beverage ads to people with gambling addiction or alcoholism, for example.
  - 13of40 3y ago
    > unethical If this is based on the possibility that one or more of the ads is harmful, how is it less ethical than the time-honored alternative, which is skipping the study and just running the ads?
    - matheusmoreira 3y ago
erichanson 3y ago
You know the Nazis had pieces of flair that they made the Jews wear.
gopher_space 3y ago
Narrowly true, but what's the difference between this and a diner trying out new blueberry pancake recipes?
- mewpmewp2 3y ago
  It's unethical for a diner to try out new recipes. Per OpenAI policy you need to ask consent before trying out new recipes.
concordDance 3y ago
We do non-consensual human experimentation all the time. Whenever you try a new outfit you're doing it. It's an extremely broad category that contains good things (installing cycle lanes to see if they encourage cycling), neutral things (making both flower mugs and wave mugs and seeing which sells) and bad things (use your imagination).
maxbond 3y ago
I wouldn't make as strong a claim as the parent comment myself, but someone pointed out to me recently that A/B testing is really similar to cold reading. Is it morally equivalent to suggest to someone, in bad faith, that you're able to deliver messages from their dead loved ones, and to perform an A/B test of switching around menu items or change up some language to try and get fewer people to abandon their carts? I lean towards "no" but I have trouble either accepting or rejecting the proposition. It's hard for me to say that A/B testing is done in bad faith, but it's also hard for me to say it's entirely unmanipulative, either.

deleted 3y ago

[deleted]

xattt 3y ago

My best guess is the algorithm has been tweaked to return exact results maybe 1/10 or 1/20 times, like a slot machine with the psychological manipulation and “reward centre activation” that comes with it.