4 ms·

Ask HN: What's the most absurd A/B test result you've seen?

24 points by sAbakumoff 9y ago

Jemaclus 9y ago
I don't know if I have a good answer for this specifically, but as a consumer, I have a little rant. You know, Amazon must A/B test the shit out of their homepage... but if I buy a shower curtain, my Amazon home page is nothing but shower curtains for weeks. How many shower curtains do they think I need? This works with other once-in-a-blue-moon items, too: plungers, watches, windshield wipers, phone cases. What they should do is say "Hey, this guy bought a shower curtain, which means A) he has a bathroom, and B) either moved in or is redecorating... and that means he probably wants a bath mat, a toothbrush holder, a plunger, a towel rack..." But nope... shower curtains for days. And you know that somehow this works out great for Amazon, or else they'd change it.
- bdevine 9y ago
  "Turns out the last shower curtain you bought from us isn't working out so great, is it? Well, check out all these top-ranked shower curtains that your search didn't return!"
  - deleted 9y ago
    [deleted]
- randomString1 9y ago
  Brand recognition by repetition? Even if the brands themselves aren't buying ads, maybe there's a bigger chance that in the future you'll feel more comfortable buying that item again. So it's better for Amazon too.
  - Jemaclus 9y ago
    Potentially!
- tyingq 9y ago
  I suspect "shower curtains for days" A/B tests better than anything else they have to test it against. Not that it's the absolute best thing, just the best thing they can automatically suggest. A bath mat that's a color/pattern match for that shower curtain, for example, might convert really well. But one that's not will convert poorly. I am surprised, as you hint, that they can't tell "one time" versus recurring. Where it should pitch me AA batteries I bought 6 months ago versus the more recent shower curtains, since I will likely buy the former again.
  - Jemaclus 9y ago
    Like I said, I'm positive they A/B test the crap out of it, so I'm sure it brings in more conversions... but that is sooo counter-intuitive, I'm not sure what the rationale is.
- bemmu 9y ago
  Maybe just showing you more shower curtains is enough to remind you to shop for those other bathroom items?
  - Jemaclus 9y ago
    It's possible. Like I said, I'm sure they A/B tested the shit out of this... I just don't know why this keeps happening. Your hypothesis is as good as any right now!
    - sorokod 9y ago
      Perhaps B was.much,much worse.
      - Jemaclus 9y ago
        That's my assumption! I think A is a bizarre solution, but I absolutely think they've tested the crap out of this and that the way it works is the one that makes the most conversions for them. Still... crazy!
- glandium 9y ago
  Relatedly, it would be good for their spam system to be aware of purchases. Case in point: I recently purchased (in pre-order) a movie Bluray on Amazon. Only a few days later, I received spam telling me that very same movie was now available for rent on Amazon video (after the release date and after I actually received the disk).
  - Jemaclus 9y ago
    This happens to me all the time, as well. I read a lot, and I buy a book on my Kindle and two days later get a promo email with the same book in it. I imagine their emails are just less sophisticated and target tens of thousands of customers per batch, rather than sending individuals perfectly tailored emails... Still, you're right. It's annoying.
- DanBC 9y ago
  https://twitter.com/actioncookbook/status/834439563032555521 https://twitter.com/actioncookbook/status/834439563032555521 ME: [views product online] Hmm. Nah. [changes website] PRODUCT: we meet again ME: Sorry, no [changes again] PRODUCT: bitch this ain't over ME: FINE. FINE. I will buy this rug. Just leave me at peace. REST OF INTERNET: this dude loves rugs, let's get him, boys
- Rjevski 9y ago
  Had the same issue when I bought an umbrella recently. My Amazon homepage is nothing but umbrellas now. I know I live in the UK but come on, this is overkill, I don't need that many umbrellas.
deleted 9y ago
[deleted]
muzani 9y ago
This article covers it well: https://theawl.com/a-complete-taxonomy-of-internet-chum-de0b7a070a2d https://theawl.com/a-complete-taxonomy-of-internet-chum-de0b... It's about the ads you usually find at the side and bottom of dodgy sites. An exact quote from the site: "Like everything else on the internet, traffic flowing through chumboxes must be tracked in order for everyone to be paid. Each box in the grid’s performance can be tracked both individually and in context of its neighbors. This allows them to be highly optimized; some chum is clearly better than others. As a byproduct of this optimization, an aesthetic has arisen. An effective chumbox clearly plays on reflex and the subconscious. The chumbox aesthetic broadcasts our most basic, libidinal, electrical desires back at us. And gets us to click." So you get all kinds of disgusting, disturbing and yet fascinating things that entice you to click. Clicking the most disturbing thing encourages more content like that.
cm2012 9y ago
I knew a background check website that tested two funnels. One gave the free info instantly and then upsold to the paid plan, the other gave that same info after an 11 minute B.S. process. The version that took an artificial 11 minutes led to literally 10x conversions to the paid info.
- wingerlang 9y ago
  I think this makes sense. If they can get it instantly without pain there is no incentive to upgrade. Having to wait is extremely annoying. A dating website I used to use does something similar. It makes guys wait 10 minutes between each message while girls can send all they want. I can imagine this works really well because even I considered purchasing it, and I never purchase such stuff.
  - Jeremy1026 9y ago
    You find a nice lady?
Gustomaximus 9y ago
Mostly when significantly different pages make little to no difference on conversion. This has made me a little jaded about sweating over the colour of a button, or wording a statement type thing. Yes, a site needs good architecture and design. But overall its other things than design look/language that become the real needle movers for digital performance, but people often focus here as its the fun stuff.

24 points by sAbakumoff 9y ago

I don't know if I have a good answer for this specifically, but as a consumer, I have a little rant. You know, Amazon must A/B test the shit out of their homepage... but if I buy a shower curtain, my Amazon home page is nothing but shower curtains for weeks. How many shower curtains do they think I need? This works with other once-in-a-blue-moon items, too: plungers, watches, windshield wipers, phone cases. What they should do is say "Hey, this guy bought a shower curtain, which means A) he has a bathroom, and B) either moved in or is redecorating... and that means he probably wants a bath mat, a toothbrush holder, a plunger, a towel rack..." But nope... shower curtains for days. And you know that somehow this works out great for Amazon, or else they'd change it.

bdevine 9y ago
"Turns out the last shower curtain you bought from us isn't working out so great, is it? Well, check out all these top-ranked shower curtains that your search didn't return!"
- deleted 9y ago
  [deleted]
randomString1 9y ago
Brand recognition by repetition? Even if the brands themselves aren't buying ads, maybe there's a bigger chance that in the future you'll feel more comfortable buying that item again. So it's better for Amazon too.
- Jemaclus 9y ago
  Potentially!
tyingq 9y ago
I suspect "shower curtains for days" A/B tests better than anything else they have to test it against. Not that it's the absolute best thing, just the best thing they can automatically suggest. A bath mat that's a color/pattern match for that shower curtain, for example, might convert really well. But one that's not will convert poorly. I am surprised, as you hint, that they can't tell "one time" versus recurring. Where it should pitch me AA batteries I bought 6 months ago versus the more recent shower curtains, since I will likely buy the former again.
- Jemaclus 9y ago
  Like I said, I'm positive they A/B test the crap out of it, so I'm sure it brings in more conversions... but that is sooo counter-intuitive, I'm not sure what the rationale is.
bemmu 9y ago
Maybe just showing you more shower curtains is enough to remind you to shop for those other bathroom items?
- Jemaclus 9y ago
  It's possible. Like I said, I'm sure they A/B tested the shit out of this... I just don't know why this keeps happening. Your hypothesis is as good as any right now!
  - sorokod 9y ago
    Perhaps B was.much,much worse.
    - Jemaclus 9y ago
      That's my assumption! I think A is a bizarre solution, but I absolutely think they've tested the crap out of this and that the way it works is the one that makes the most conversions for them. Still... crazy!
glandium 9y ago
Relatedly, it would be good for their spam system to be aware of purchases. Case in point: I recently purchased (in pre-order) a movie Bluray on Amazon. Only a few days later, I received spam telling me that very same movie was now available for rent on Amazon video (after the release date and after I actually received the disk).
- Jemaclus 9y ago
  This happens to me all the time, as well. I read a lot, and I buy a book on my Kindle and two days later get a promo email with the same book in it. I imagine their emails are just less sophisticated and target tens of thousands of customers per batch, rather than sending individuals perfectly tailored emails... Still, you're right. It's annoying.
DanBC 9y ago
https://twitter.com/actioncookbook/status/834439563032555521 https://twitter.com/actioncookbook/status/834439563032555521 ME: [views product online] Hmm. Nah. [changes website] PRODUCT: we meet again ME: Sorry, no [changes again] PRODUCT: bitch this ain't over ME: FINE. FINE. I will buy this rug. Just leave me at peace. REST OF INTERNET: this dude loves rugs, let's get him, boys
Rjevski 9y ago
Had the same issue when I bought an umbrella recently. My Amazon homepage is nothing but umbrellas now. I know I live in the UK but come on, this is overkill, I don't need that many umbrellas.

deleted 9y ago

[deleted]

muzani 9y ago

This article covers it well: https://theawl.com/a-complete-taxonomy-of-internet-chum-de0b7a070a2d https://theawl.com/a-complete-taxonomy-of-internet-chum-de0b... It's about the ads you usually find at the side and bottom of dodgy sites. An exact quote from the site: "Like everything else on the internet, traffic flowing through chumboxes must be tracked in order for everyone to be paid. Each box in the grid’s performance can be tracked both individually and in context of its neighbors. This allows them to be highly optimized; some chum is clearly better than others. As a byproduct of this optimization, an aesthetic has arisen. An effective chumbox clearly plays on reflex and the subconscious. The chumbox aesthetic broadcasts our most basic, libidinal, electrical desires back at us. And gets us to click." So you get all kinds of disgusting, disturbing and yet fascinating things that entice you to click. Clicking the most disturbing thing encourages more content like that.

cm2012 9y ago

I knew a background check website that tested two funnels. One gave the free info instantly and then upsold to the paid plan, the other gave that same info after an 11 minute B.S. process. The version that took an artificial 11 minutes led to literally 10x conversions to the paid info.

wingerlang 9y ago
I think this makes sense. If they can get it instantly without pain there is no incentive to upgrade. Having to wait is extremely annoying. A dating website I used to use does something similar. It makes guys wait 10 minutes between each message while girls can send all they want. I can imagine this works really well because even I considered purchasing it, and I never purchase such stuff.
- Jeremy1026 9y ago
  You find a nice lady?

Gustomaximus 9y ago

Mostly when significantly different pages make little to no difference on conversion. This has made me a little jaded about sweating over the colour of a button, or wording a statement type thing. Yes, a site needs good architecture and design. But overall its other things than design look/language that become the real needle movers for digital performance, but people often focus here as its the fun stuff.