8 ms·

What works in e-commerce – A meta-analysis of online experiments [pdf]

424 points by sweezyjeezy 9y ago

geetfun 9y ago
Anyone who hangs out on Facebook group e-commerce forums basically have seen the outrageous claims the authors of this paper alludes to. Nice to see an analysis like this for a change.
- sweezyjeezy 9y ago
  Thanks!
- castell 9y ago
  can you recommend good groups?
  - geetfun 9y ago
    A lot of Shopify groups. Search up anything Shopify or ecommerce mastermind. Be aware that a lot of them are set up as groups that simply try to sell you stuff. The best ecommerce folks are basically the best marketers.
- kawsper 9y ago
  My favorite is: "Our buy now button should be orange, that converts best" without testing it, verifying the results, and just pointing at some blogposts.
iagovar 9y ago
It's good to see an analysis, albeit most of this info was common knowledge, maybe except call to action buttons causing a decrease. I thought it was the opposite. Sometimes factoring all variables when doing testing becomes impossible.
- sweezyjeezy 9y ago
  This stuff is less common knowledge than you might expect. When bringing on new clients at Qubit (particularly smaller ones), we still find so many of them obsessing over tiny UI changes expecting it to make an impact. Much of the e-commerce industry has bought into the idea that the cosmetics of a site is the most important thing to get right.
  - mrweasel 9y ago
    I believe that the main focus of any ecommerce site should be pricing, availability and having the payment and delivery options your customers expects. E-commerce sites want to "engage" with their customers, but fail to realise that the customers don't want a relationship. They just want whatever product you're selling as quickly and cheaply as possible. Most of you customers will be coming via price comparison sites or Google, so they aren't going to you directly (unless you're Amazon or eBay). For most e-commerce sites the customers aren't going to stay long enough to notice imperfections in the UI. What you do need is a dead simple checkout (no signup required) and an equally easy return form. Everything else will be used by only a tiny percentage of your customers, often those you don't want to deal with anyway. For some weird reason making the checkout to simply will result in a large number of purchases cancelled right after placement. In my experience customers are more likely to cancel within 15 minutes of placing the order than at any other time.
    - brazzy 9y ago
      > For most e-commerce sites the customers aren't going to stay long enough to notice imperfections in the UI. > What you do need is a dead simple checkout (no signup required) These two statements are contradicting each other, because "imperfections" in the UI can in fact greatly affect the perceived simplicity of the checkout process - and optimizing that is a large part of what people want to achieve with A/B testing.
    - Silhouette 9y ago
      You seem to be taking a very narrow definition of e-commerce here. What you say seems plausible enough for sites selling low value or commodity products to casual purchasers, but I wouldn't assume the same situation for someone selling luxury products, services, digital subscriptions, etc.
tomarr 9y ago
If you want to see pretty much all the most effective measures in action, go to a ticket reseller like Viagogo and follow something through to the basket. It is both impressive and amazing the amount of psychological steers there are on the site.
- gruez 9y ago
  Or flight/hotel/car rental booking websites. Google flights is a notable exception.
  - _jal 9y ago
    The easiest way to make me go somewhere else is to adopt these tactics. They make me feel like I'm locked in a battle of wits with an autistic con-artist, or maybe a suspicious Fed - if weird rituals are not faithfully observed under conditions of vast information asymmetry and artificial stress, I will suffer. There's no hope of avoiding that with airlines for me - I don't fly enough to avoid the prole line. But literally any storefront where I can avoid that shit, I do, and it breeds not just dislike - it makes me want see them to fail.
    - lucaspm98 9y ago
      The people with the knowledge of these tricks are so few that unfortunately in most cases the additional revenue is too much to keep the minority happy. It's fascinating to look at ecommerce storefronts and search for all of the subtle tactics they're trying to use on you.
    - deleted 9y ago
      [deleted]
  - tyingq 9y ago
    I don't see these tactics on many airline or hotel owned sites. They seem to be more common on aggregators.
- deleted 9y ago
  [deleted]
Silhouette 9y ago
Thanks for sharing some real data. It's always interesting to see. I too am surprised to see the conclusions come out as negative as this over such a large overall data set. Just from our own experience, even making quite modest changes to small web sites, it's not that unusual to see the kinds of change that came out with a negative mean in this report actually making a very noticeable positive difference. I wonder whether this is partly a matter of interpretation and presentation. A lot of the treatments that had a slightly negative mean also had a lot of variance, which suggests that quite often those treatments do work but it's not reliable and requires experimentation to make sure you only keep the genuinely beneficial cases. It seems plausible that there were a few of the "75% improvement in our case study!" kinds of results lost in the long tails, but that what the data is telling us is that those really are outliers and don't happen nearly as often as we might wish.
joe-stanton 9y ago
This is a really useful article. It's a shame that so much development time is wasted on large numbers of fruitless optimisations just because they are "easy" (eg. tweaking the colour of a CTA). That being said, I'm surprised many of the results are so negative. It would be great to also see the max uplift achieved for each category. A number of retailers I've worked with have been able to beat these uplifts by quite a bit. I wonder if it might be significantly skewed by the kind of clients Qubit has?
- Silhouette 9y ago
  It would be great to see the max uplift achieved for each category Indeed. What matters most with these kinds of experiments isn't really the average results, but what is possible and the distribution among beneficial results only. After all, the whole point of A/B testing is to try experiments and then either keep the changes if they improve results or stay with what you've already got if the changes didn't bring an improvement. Surely all the treatments that led to negative changes would just have been discarded in practice? It's still important to see the full picture as well, if only to guide decisions about which experiments are even worth trying, but I think there's another side that doesn't fully come through here.
  - jacquesm 9y ago
    I think the big error in A/B testing is that expectations are quite often very unrealistic. Designers typically have a reasonably good idea about what will work and what will not. Finding 'million dollar buttons' is rare. Of course a couple of percent or even 10's of percents of improvement is nothing to sneeze at. But thinking that by A/B testing forever you're going to make a shrub grow into a tree is imo not realistic. Aside from the detail that a continuously changing user interface is often in itself a barrier to sales. Ironically, the companies that have benefited most from A/B testing were the ones that were doing a terrible job of it in the first place so then there is lots of low hanging fruit making the consultants look good. Yet another item often missed: A/B testing success is a direct function of the length of the lever you are pulling. If that lever commands billions of dollars then it is easy to make it pay for itself. But if you're trying to turn $10000 into $11500 then you likely are wasting your time.
ifcologne 9y ago
Lot's of interesting data and some findings I didn't expected. Thanks. What I have missed in this paper is the impact of customer reviews (a 4.5 ranking has a higher uplift potential than a 5.0 star ranking - according to some studies). And the number of reviews has impact as well. Not enough studies around for a meta analysis?
lostphilosopher 9y ago
This was really interesting. Thanks for posting it. Does anyone know of a place to find similar content? (Analysis of web trends and practices from a data driven perspective.)
- brownesauce 9y ago
  We're pretty sure this is the only analysis of its kind at the moment. You have to have a pretty large number of experiments to be able to do this analysis. Hopefully we see more soon (co-author speaking)
- mitchdoogle 9y ago
  Nielsen Norman Group ( https://www.nngroup.com https://www.nngroup.com ) is a good place to look for somewhat similar content. NNG focuses more on user experience, and so their studies are usually based on direct observation of users, rather than large amounts of data gathered through analytics. Still, it has proven a very valuable resource for me when trying to explain a decision I've made in a new website design. They have many free articles that offer some good insights, as well as some more in-depth reports about specific sectors that will cost a few hundred dollars each.
JorgeGT 9y ago
The PwC assurance report URL gives a 404: http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assurance.pdf http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assur...
- sweezyjeezy 9y ago
  Apologies : http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assurance_0.pdf http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assur...
  - Raphmedia 9y ago
    Now that's pretty bad. There is a modal that open before the document is displayed. "Chrome PDF Viewer Click YES to accept the terms of the disclaimer on page 1 of this document. If you click NO the document will close. [OK] [CANCEL]" Not only does this want me to agree to something before displaying it, the legally binding options (Yes / No) aren't even choices in the modal.
    - thefalcon 9y ago
      Thankfully, clicking CANCEL opens the PDF anyway.
Iv 9y ago
tl;dr: The 3 items possibly statistically significant are: - Saying there are just a handful items left in stock (+2.9% revenue per client) - Saying other people are watching this product (+2.3%) - Time limited offer (+1.5%) I did not see mention of combining these factors. I doubt the gains are cumulative. My main takeaway is that most optimizations are not worthy if you have the opportunity to spend your time/money on something else to bring value to the consumer. Also, I think #1 and #3 are dick moves and #2 needs some good crafting to not be. I doubt the cost in reputation is worth the increase in revenue.
- robbiemitchell 9y ago
  > I doubt the gains are cumulative Why do you doubt that? Aren't they presented as independent results? > bring value to the consumer This is vague, but also a separate goal. Perhaps another way of representing customer value is churn reduction. In any case, it could be appropriate to invest in gaining 3% more revenue per additional customer, then use the gains to invest in more customer value. > #1 and #3 are dick moves and #2 needs some good crafting Real scarcity exists, and there's value to communicating it to the customer. For example, saying there are two copies of something available, and the rest are backordered for three weeks, can be pretty useful information.
  - dkersten 9y ago
    Real scarcity exists, and there's value to communicating it to the customer. I assume what was meant is that they're dick moves if done purely for sales optimisation (i.e. If they are lying) and not if the scarcity or time limitations actually naturally exist.
  - Iv 9y ago
    I think they are not purely cumulative because I assume that consumer have a it of slack in their decision/budget and that each optimization consumes some of this slack. The basic idea behind 1 and 3 is to stress out the consumer into believing that he may miss a sale. I think that stressing them out with 1 or 3 more or less gives the same result and does not need to be applied twice. Real scarcity exists, but this paper gives an incentive to create artificial scarcity. I have seen a lot of websites mention low stocks but none mentioning backorders or restocking estimates. Saying "Only 3 items left" has a very different effect than "only 3 items left but 100 more arrive in 2 weeks".
cylinder 9y ago
Well, running a rudimentary eBay store you realize these things help pretty quickly, but it's good to have data. However... Starting a new e-commerce property? Good luck finding traffic in anything profitable. Amazon and other Giants dominate search rankings so I'm not sure how you will find your traffic unless you create a new niche. Maybe you're a thought leader in a hobbyist space, that can work... But you're not going to be succeeding because of these tricks
- nik736 9y ago
  That's simply false. There are always categories or "side"-categories that can be highly profitable and big stores only care about as a side business (small category in their store). Being a small category for a huge store means they don't care about that category too much but it doesn't mean there isn't a lot of cash to be made. I've helped several companies to build an online store that did exactly that and they are all doing 5 digit revenue month after month. It's easy to rank higher in search engine for something you are 100% focused on compared to a big stores where it's only a small thing that they don't highlight.
  - kossae 9y ago
    Agreed. I work for a company in such a niche and there are definitely side-niche markets still out there for people to find and dominate, sometimes even on SEO alone. A small piece of a huge pie is still a very large piece to me.
    - cylinder 9y ago
      Have a good example of one that took that piece? Outside of Amazon
      - deleted 9y ago
        [deleted]
  - pbowyer 9y ago
    Can you give any examples of small categories? I can't think of any, let alone get my brain round what kind of thing they would be.
bartkappenburg 9y ago
Great research with lots of data and tangible results. I'm missing 'authority' as way to improve conversion. Funny that they use it themselves in two ways: letting PWC check the results and, in a milder form, using a scientific way of communicating the results (LaTeX, article layout etc). The impact would have been less if it was just a blog post :-). I'm curious about results on authority, maybe someone from qubit can give us some insights on that? At my company[0] we offer a solution to sites to implement these strategies through notifications/nudges. Having said that: we firmly believe in A/B testing but we believe even more in recognizing (we do that through machine learning) what technique works best on a personal level. This means that a site can have, for example, two strategies and that we apply none, either one or both on the visitor. That way you can reach higher uplifts. [0] https://www.conversify.com https://www.conversify.com
ssharp 9y ago
In my experience, the A/B tests that are most likely to win are the ones where you make UX changes designed to make it easier for visitors to do what you want them to do. These not only improve your conversion rates, they are also less spammy and intrusive as things like exit-intent modals. They are also the types of gains that do compound. Want an easy win? Make mobile checkout better. It's generally the worst. I was on a fairly large, publicly traded, retailer's site over the weekend and had a goofy error that was extremely easy to make on their mobile checkout page. While I was alerted to the error, it also emptied my shopping cart and erased all the address and payment info I spent time typing in.
1k 9y ago
I'm surprised free shipping has a negative impact on revenue. Worst case I imagine would be that revenue would increase but with less or negative profit. Likewise most GUI-related tweaks seem to have a negative effect (mobile friendliness, search, navigation). Assuming it gives a better mobile experience, why would anyone spend less - unless the goal is to get them off mobile and onto the desktop.
- Theodores 9y ago
  I think that the GUI-related tweaks must be on sites that are done properly and well established. For these guys the customers know what to expect so a change in design will be received negatively by people that are used to the old site. Although not in ecommerce, the redesign of the BBC site is like that, people want to cling on to the old site and instinctively don't like the new. In time they adjust or get a touchscreen computer and 'get' the changes and what the redesign is about. Most people are working on sites other than the top ones. With these websites where customers are not regulars, change to the GUI is received differently. Sensible updates to the UX will convert.
ronack 9y ago
This is interesting but I definitely question some of the results, for instance reporting a negative impact for changing search results. Many businesses have been built on improving conversion through search result optimization.
- robbiemitchell 9y ago
  I interpreted this as them creating/improving on-site search. Not related to adwords or display optimization.
  - ronack 9y ago
    Sure, but what were the improvements to on-site search? More relevant results? Tweaking the display? I know from experience that improving search result relevance in e-commerce has a (sometimes drastic) improvement in conversion.
inopinatus 9y ago
Inconsistencies to report. Page 2: • scarcity (stock pointers) +2.9% uplift • urgency (countdown timers) +2.3% uplift • social proof (informing users of others’ behaviour) +1.5% uplift Page 6 table 2.2: scarcity 2.9% social proof 2.3% urgency 1.5%
- sweezyjeezy 9y ago
  facepalm. The table is correct. Will correct this tomorrow. Thanks for the heads up.
Systemic33 9y ago
A lot of these are brilliantly displayed with varying degrees of integration and success on airline pages. [1],[2] [1] http://sas.dk http://sas.dk (Search for e.g. Copenhagen -> London) [2] http://lufthansa.com http://lufthansa.com (Search for e.g. Frankfurt -> Copenhagen)
jameslk 9y ago
I wish they would have evaluated the effect of dynamic pricing. That is, showing different prices to different visitors for the same product. Perhaps not enough online retailers employ the practice, although it seems to be an important tool for retailers like Amazon[0]. 0. https://www.theatlantic.com/magazine/archive/2017/05/how-online-shopping-makes-suckers-of-us-all/521448/?single_page=true https://www.theatlantic.com/magazine/archive/2017/05/how-onl...
- ripberge 9y ago
  What you are describing is an A/B test of a price, not typically what industry calls "dynamic" or "variable" pricing. Amazon tried A/B testing prices and caught hell for it. They no longer do it, but they will do dynamic pricing.
- have_faith 9y ago
  Finding out a company employs this technique is the quickest way for me to not buy anything from them.
hahamrfunnyguy 9y ago
I've done the countdown timer thing before, but not in a sleazy way like Viagogo. I've done limited-time sales where the item starts at a percentage off and gradually increases to full price. It seems to work well.
nocoder 9y ago
Very interesting thanks for sharing. As someone working in e-commerce, I was smiling when I saw some of them but it is frustrating on how often these ineffective experiments are repeated. I would have loved to see the cut of performance by industry/sector. My hunch is some of the things would work really well in travel but not as well in others especially low involvement categories and categories with lower average selling price. It would also be interesting to know the average duration of these A/B tests, I think some of things like scarcity and urgency will have larger effect in the shorter time duration vs others like UI changes which will take a while to produce substantial results, mostly because customers will have to learn new behaviours. Product recommendations is interesting because it is notoriously difficult to get them right and feel they tend to work better in long tail categories like media vs. head heavy categories like mobiles or laptops. They may also not work well in categories where brand influence is high and are generally high involvement and high cost.

424 points by sweezyjeezy 9y ago

Anyone who hangs out on Facebook group e-commerce forums basically have seen the outrageous claims the authors of this paper alludes to. Nice to see an analysis like this for a change.

sweezyjeezy 9y ago
Thanks!
castell 9y ago
can you recommend good groups?
- geetfun 9y ago
  A lot of Shopify groups. Search up anything Shopify or ecommerce mastermind. Be aware that a lot of them are set up as groups that simply try to sell you stuff. The best ecommerce folks are basically the best marketers.
kawsper 9y ago
My favorite is: "Our buy now button should be orange, that converts best" without testing it, verifying the results, and just pointing at some blogposts.

iagovar 9y ago

It's good to see an analysis, albeit most of this info was common knowledge, maybe except call to action buttons causing a decrease. I thought it was the opposite. Sometimes factoring all variables when doing testing becomes impossible.

sweezyjeezy 9y ago
This stuff is less common knowledge than you might expect. When bringing on new clients at Qubit (particularly smaller ones), we still find so many of them obsessing over tiny UI changes expecting it to make an impact. Much of the e-commerce industry has bought into the idea that the cosmetics of a site is the most important thing to get right.
- mrweasel 9y ago
  I believe that the main focus of any ecommerce site should be pricing, availability and having the payment and delivery options your customers expects. E-commerce sites want to "engage" with their customers, but fail to realise that the customers don't want a relationship. They just want whatever product you're selling as quickly and cheaply as possible. Most of you customers will be coming via price comparison sites or Google, so they aren't going to you directly (unless you're Amazon or eBay). For most e-commerce sites the customers aren't going to stay long enough to notice imperfections in the UI. What you do need is a dead simple checkout (no signup required) and an equally easy return form. Everything else will be used by only a tiny percentage of your customers, often those you don't want to deal with anyway. For some weird reason making the checkout to simply will result in a large number of purchases cancelled right after placement. In my experience customers are more likely to cancel within 15 minutes of placing the order than at any other time.
  - brazzy 9y ago
    > For most e-commerce sites the customers aren't going to stay long enough to notice imperfections in the UI. > What you do need is a dead simple checkout (no signup required) These two statements are contradicting each other, because "imperfections" in the UI can in fact greatly affect the perceived simplicity of the checkout process - and optimizing that is a large part of what people want to achieve with A/B testing.
  - Silhouette 9y ago
    You seem to be taking a very narrow definition of e-commerce here. What you say seems plausible enough for sites selling low value or commodity products to casual purchasers, but I wouldn't assume the same situation for someone selling luxury products, services, digital subscriptions, etc.

tomarr 9y ago

If you want to see pretty much all the most effective measures in action, go to a ticket reseller like Viagogo and follow something through to the basket. It is both impressive and amazing the amount of psychological steers there are on the site.

gruez 9y ago
Or flight/hotel/car rental booking websites. Google flights is a notable exception.
- _jal 9y ago
  The easiest way to make me go somewhere else is to adopt these tactics. They make me feel like I'm locked in a battle of wits with an autistic con-artist, or maybe a suspicious Fed - if weird rituals are not faithfully observed under conditions of vast information asymmetry and artificial stress, I will suffer. There's no hope of avoiding that with airlines for me - I don't fly enough to avoid the prole line. But literally any storefront where I can avoid that shit, I do, and it breeds not just dislike - it makes me want see them to fail.
  - lucaspm98 9y ago
    The people with the knowledge of these tricks are so few that unfortunately in most cases the additional revenue is too much to keep the minority happy. It's fascinating to look at ecommerce storefronts and search for all of the subtle tactics they're trying to use on you.
  - deleted 9y ago
    [deleted]
- tyingq 9y ago
  I don't see these tactics on many airline or hotel owned sites. They seem to be more common on aggregators.
deleted 9y ago
[deleted]

Silhouette 9y ago

Thanks for sharing some real data. It's always interesting to see. I too am surprised to see the conclusions come out as negative as this over such a large overall data set. Just from our own experience, even making quite modest changes to small web sites, it's not that unusual to see the kinds of change that came out with a negative mean in this report actually making a very noticeable positive difference. I wonder whether this is partly a matter of interpretation and presentation. A lot of the treatments that had a slightly negative mean also had a lot of variance, which suggests that quite often those treatments do work but it's not reliable and requires experimentation to make sure you only keep the genuinely beneficial cases. It seems plausible that there were a few of the "75% improvement in our case study!" kinds of results lost in the long tails, but that what the data is telling us is that those really are outliers and don't happen nearly as often as we might wish.

joe-stanton 9y ago

This is a really useful article. It's a shame that so much development time is wasted on large numbers of fruitless optimisations just because they are "easy" (eg. tweaking the colour of a CTA). That being said, I'm surprised many of the results are so negative. It would be great to also see the max uplift achieved for each category. A number of retailers I've worked with have been able to beat these uplifts by quite a bit. I wonder if it might be significantly skewed by the kind of clients Qubit has?

Silhouette 9y ago
It would be great to see the max uplift achieved for each category Indeed. What matters most with these kinds of experiments isn't really the average results, but what is possible and the distribution among beneficial results only. After all, the whole point of A/B testing is to try experiments and then either keep the changes if they improve results or stay with what you've already got if the changes didn't bring an improvement. Surely all the treatments that led to negative changes would just have been discarded in practice? It's still important to see the full picture as well, if only to guide decisions about which experiments are even worth trying, but I think there's another side that doesn't fully come through here.
- jacquesm 9y ago
  I think the big error in A/B testing is that expectations are quite often very unrealistic. Designers typically have a reasonably good idea about what will work and what will not. Finding 'million dollar buttons' is rare. Of course a couple of percent or even 10's of percents of improvement is nothing to sneeze at. But thinking that by A/B testing forever you're going to make a shrub grow into a tree is imo not realistic. Aside from the detail that a continuously changing user interface is often in itself a barrier to sales. Ironically, the companies that have benefited most from A/B testing were the ones that were doing a terrible job of it in the first place so then there is lots of low hanging fruit making the consultants look good. Yet another item often missed: A/B testing success is a direct function of the length of the lever you are pulling. If that lever commands billions of dollars then it is easy to make it pay for itself. But if you're trying to turn $10000 into $11500 then you likely are wasting your time.

ifcologne 9y ago

Lot's of interesting data and some findings I didn't expected. Thanks. What I have missed in this paper is the impact of customer reviews (a 4.5 ranking has a higher uplift potential than a 5.0 star ranking - according to some studies). And the number of reviews has impact as well. Not enough studies around for a meta analysis?

lostphilosopher 9y ago

This was really interesting. Thanks for posting it. Does anyone know of a place to find similar content? (Analysis of web trends and practices from a data driven perspective.)

brownesauce 9y ago
We're pretty sure this is the only analysis of its kind at the moment. You have to have a pretty large number of experiments to be able to do this analysis. Hopefully we see more soon (co-author speaking)
mitchdoogle 9y ago
Nielsen Norman Group ( https://www.nngroup.com https://www.nngroup.com ) is a good place to look for somewhat similar content. NNG focuses more on user experience, and so their studies are usually based on direct observation of users, rather than large amounts of data gathered through analytics. Still, it has proven a very valuable resource for me when trying to explain a decision I've made in a new website design. They have many free articles that offer some good insights, as well as some more in-depth reports about specific sectors that will cost a few hundred dollars each.

JorgeGT 9y ago

The PwC assurance report URL gives a 404: http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assurance.pdf http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assur...

sweezyjeezy 9y ago
Apologies : http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assurance_0.pdf http://www.qubit.com/sites/default/files/pdf/pwc-qubit-assur...
- Raphmedia 9y ago
  Now that's pretty bad. There is a modal that open before the document is displayed. "Chrome PDF Viewer Click YES to accept the terms of the disclaimer on page 1 of this document. If you click NO the document will close. [OK] [CANCEL]" Not only does this want me to agree to something before displaying it, the legally binding options (Yes / No) aren't even choices in the modal.
  - thefalcon 9y ago
    Thankfully, clicking CANCEL opens the PDF anyway.

Iv 9y ago

tl;dr: The 3 items possibly statistically significant are: - Saying there are just a handful items left in stock (+2.9% revenue per client) - Saying other people are watching this product (+2.3%) - Time limited offer (+1.5%) I did not see mention of combining these factors. I doubt the gains are cumulative. My main takeaway is that most optimizations are not worthy if you have the opportunity to spend your time/money on something else to bring value to the consumer. Also, I think #1 and #3 are dick moves and #2 needs some good crafting to not be. I doubt the cost in reputation is worth the increase in revenue.

robbiemitchell 9y ago
> I doubt the gains are cumulative Why do you doubt that? Aren't they presented as independent results? > bring value to the consumer This is vague, but also a separate goal. Perhaps another way of representing customer value is churn reduction. In any case, it could be appropriate to invest in gaining 3% more revenue per additional customer, then use the gains to invest in more customer value. > #1 and #3 are dick moves and #2 needs some good crafting Real scarcity exists, and there's value to communicating it to the customer. For example, saying there are two copies of something available, and the rest are backordered for three weeks, can be pretty useful information.
- dkersten 9y ago
  Real scarcity exists, and there's value to communicating it to the customer. I assume what was meant is that they're dick moves if done purely for sales optimisation (i.e. If they are lying) and not if the scarcity or time limitations actually naturally exist.
- Iv 9y ago
  I think they are not purely cumulative because I assume that consumer have a it of slack in their decision/budget and that each optimization consumes some of this slack. The basic idea behind 1 and 3 is to stress out the consumer into believing that he may miss a sale. I think that stressing them out with 1 or 3 more or less gives the same result and does not need to be applied twice. Real scarcity exists, but this paper gives an incentive to create artificial scarcity. I have seen a lot of websites mention low stocks but none mentioning backorders or restocking estimates. Saying "Only 3 items left" has a very different effect than "only 3 items left but 100 more arrive in 2 weeks".

cylinder 9y ago

Well, running a rudimentary eBay store you realize these things help pretty quickly, but it's good to have data. However... Starting a new e-commerce property? Good luck finding traffic in anything profitable. Amazon and other Giants dominate search rankings so I'm not sure how you will find your traffic unless you create a new niche. Maybe you're a thought leader in a hobbyist space, that can work... But you're not going to be succeeding because of these tricks

nik736 9y ago
That's simply false. There are always categories or "side"-categories that can be highly profitable and big stores only care about as a side business (small category in their store). Being a small category for a huge store means they don't care about that category too much but it doesn't mean there isn't a lot of cash to be made. I've helped several companies to build an online store that did exactly that and they are all doing 5 digit revenue month after month. It's easy to rank higher in search engine for something you are 100% focused on compared to a big stores where it's only a small thing that they don't highlight.
- kossae 9y ago
  Agreed. I work for a company in such a niche and there are definitely side-niche markets still out there for people to find and dominate, sometimes even on SEO alone. A small piece of a huge pie is still a very large piece to me.
  - cylinder 9y ago
    Have a good example of one that took that piece? Outside of Amazon
    - deleted 9y ago
      [deleted]
- pbowyer 9y ago
  Can you give any examples of small categories? I can't think of any, let alone get my brain round what kind of thing they would be.

bartkappenburg 9y ago

Great research with lots of data and tangible results. I'm missing 'authority' as way to improve conversion. Funny that they use it themselves in two ways: letting PWC check the results and, in a milder form, using a scientific way of communicating the results (LaTeX, article layout etc). The impact would have been less if it was just a blog post :-). I'm curious about results on authority, maybe someone from qubit can give us some insights on that? At my company[0] we offer a solution to sites to implement these strategies through notifications/nudges. Having said that: we firmly believe in A/B testing but we believe even more in recognizing (we do that through machine learning) what technique works best on a personal level. This means that a site can have, for example, two strategies and that we apply none, either one or both on the visitor. That way you can reach higher uplifts. [0] https://www.conversify.com https://www.conversify.com

ssharp 9y ago

In my experience, the A/B tests that are most likely to win are the ones where you make UX changes designed to make it easier for visitors to do what you want them to do. These not only improve your conversion rates, they are also less spammy and intrusive as things like exit-intent modals. They are also the types of gains that do compound. Want an easy win? Make mobile checkout better. It's generally the worst. I was on a fairly large, publicly traded, retailer's site over the weekend and had a goofy error that was extremely easy to make on their mobile checkout page. While I was alerted to the error, it also emptied my shopping cart and erased all the address and payment info I spent time typing in.

1k 9y ago

I'm surprised free shipping has a negative impact on revenue. Worst case I imagine would be that revenue would increase but with less or negative profit. Likewise most GUI-related tweaks seem to have a negative effect (mobile friendliness, search, navigation). Assuming it gives a better mobile experience, why would anyone spend less - unless the goal is to get them off mobile and onto the desktop.

Theodores 9y ago
I think that the GUI-related tweaks must be on sites that are done properly and well established. For these guys the customers know what to expect so a change in design will be received negatively by people that are used to the old site. Although not in ecommerce, the redesign of the BBC site is like that, people want to cling on to the old site and instinctively don't like the new. In time they adjust or get a touchscreen computer and 'get' the changes and what the redesign is about. Most people are working on sites other than the top ones. With these websites where customers are not regulars, change to the GUI is received differently. Sensible updates to the UX will convert.

ronack 9y ago

This is interesting but I definitely question some of the results, for instance reporting a negative impact for changing search results. Many businesses have been built on improving conversion through search result optimization.

robbiemitchell 9y ago
I interpreted this as them creating/improving on-site search. Not related to adwords or display optimization.
- ronack 9y ago
  Sure, but what were the improvements to on-site search? More relevant results? Tweaking the display? I know from experience that improving search result relevance in e-commerce has a (sometimes drastic) improvement in conversion.

inopinatus 9y ago

Inconsistencies to report. Page 2: • scarcity (stock pointers) +2.9% uplift • urgency (countdown timers) +2.3% uplift • social proof (informing users of others’ behaviour) +1.5% uplift Page 6 table 2.2: scarcity 2.9% social proof 2.3% urgency 1.5%

sweezyjeezy 9y ago
facepalm. The table is correct. Will correct this tomorrow. Thanks for the heads up.

Systemic33 9y ago

A lot of these are brilliantly displayed with varying degrees of integration and success on airline pages. [1],[2] [1] http://sas.dk http://sas.dk (Search for e.g. Copenhagen -> London) [2] http://lufthansa.com http://lufthansa.com (Search for e.g. Frankfurt -> Copenhagen)

jameslk 9y ago

I wish they would have evaluated the effect of dynamic pricing. That is, showing different prices to different visitors for the same product. Perhaps not enough online retailers employ the practice, although it seems to be an important tool for retailers like Amazon[0]. 0. https://www.theatlantic.com/magazine/archive/2017/05/how-online-shopping-makes-suckers-of-us-all/521448/?single_page=true https://www.theatlantic.com/magazine/archive/2017/05/how-onl...

ripberge 9y ago
What you are describing is an A/B test of a price, not typically what industry calls "dynamic" or "variable" pricing. Amazon tried A/B testing prices and caught hell for it. They no longer do it, but they will do dynamic pricing.
have_faith 9y ago
Finding out a company employs this technique is the quickest way for me to not buy anything from them.

hahamrfunnyguy 9y ago

I've done the countdown timer thing before, but not in a sleazy way like Viagogo. I've done limited-time sales where the item starts at a percentage off and gradually increases to full price. It seems to work well.

nocoder 9y ago

Very interesting thanks for sharing. As someone working in e-commerce, I was smiling when I saw some of them but it is frustrating on how often these ineffective experiments are repeated. I would have loved to see the cut of performance by industry/sector. My hunch is some of the things would work really well in travel but not as well in others especially low involvement categories and categories with lower average selling price. It would also be interesting to know the average duration of these A/B tests, I think some of things like scarcity and urgency will have larger effect in the shorter time duration vs others like UI changes which will take a while to produce substantial results, mostly because customers will have to learn new behaviours. Product recommendations is interesting because it is notoriously difficult to get them right and feel they tend to work better in long tail categories like media vs. head heavy categories like mobiles or laptops. They may also not work well in categories where brand influence is high and are generally high involvement and high cost.