6 ms·

Optimizely (YC W10) Increases Homepage Conversion Rate by 29%

146 points by dsiroker 15y ago

espadagroup 15y ago
Does anyone know of a central repository for all of these little landing page optimization tweaks? I know each site is different and just to test to know for certain but there should be some general decisions vetted that would make a good boilerplate.
- kristofferR 15y ago
  http://www.abtests.com/ http://www.abtests.com/ is one
- petercooper 15y ago
  abtests (mentioned above) is awesome but there's also http://whichtestwon.com/ http://whichtestwon.com/
JackWebbHeller 15y ago
I got a lifetime free Optimizely account as part of an AppSumo deal. I can successfully say it's one of the simplest yet most powerful and effective web products I've ever used. The ease with which you can make changes is astonishing and there's no limits if you know a little jQuery.
AJ007 15y ago
They increased the number of people submitting their URL, there is no telling if that actually resulted in higher leads for them. What I have found is a simple landing page, that tells the user exactly what you are providing, and is free of any confusion, works the best over the long term. I've run hundreds of thousands of website visitors through Google Website Optimizer in multi-variate tests and what I've found is that over time there is little to no difference in conversion rate for minor landing page changes. The biggest jumps come from eliminating content in the design and clarifying the message. Looking at the small amount of users they sent to this landing page, I would call the results inconclusive. You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do. (its really chilling when you see how many drugs the FDA has approved over tiny margins of change/success.)
- gnaritas 15y ago
  > Looking at the small amount of users they sent to this landing page, I would call the results inconclusive The statistic say otherwise. A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result. > You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do. And you can rattle off personal anecdotes like this all day long, and the statistics are still more correct than you are and assert their margins of error and accuracy. The statistics are more correct than your intuition.
  - mcn 15y ago
    Keep in mind that the 1% margin of error applies to the measurements of the conversion rates. The 29% has a significantly higher margin of error.
  - Silhouette 15y ago
    > A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result. Well, no, it's a set of numbers with percentage signs after them. Perhaps the documentation for Optimizely specifies how their error margins etc. are derived, but nothing in the linked article does as far as I can see. Without knowing that underlying reasoning, all those pretty graphs and percentages are just a load of gobbledegook, apart from the original data points and the percentage increase figures derived directly from dividing them.
    - gnaritas 15y ago
      What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.
      - Silhouette 15y ago
        I did. A Google search for "error bars" site:optimizely.com turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written. If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested. Otherwise, I stand by my earlier comments.
sgrove 15y ago
We've been using Optimizely more and more extensively (we have a slightly unusual use case for it), and it's been fantastic at successfully ratcheting up conversions. We use it in concert with mixpanel when we need to push people along a funnel. The tool's insanely easy to implement, a joy to use, and I get to rely on them to tell me when something is statistically meaningful.
- walrus 15y ago
  What is your slightly unusual use case?
  - sgrove 15y ago
    We actually deploy it across a number of client apps, rather than just our own site. We use it to run a dozen or so tests at any given time across multiple apps, and to quickly iterate on others' products. It's an interesting combination with mixpanel. If we could get one more thing from Mixpanel, we'd really be set - I'm bugging them, we'll see if they come out with it!
kristofferR 15y ago
How does Optimizely compare with Visual Website Optimizer?
- dsiroker 15y ago
  VWO is a great product but here are some of the unique benefits of using Optimizely: First is ease-of-use and simplicity: we hope Optimizely is a product anyone can use, not just IT or engineers. Paras Chopra, the creator of VWO recently said on Quora: "Optimizely is a great tool and perhaps even bit easier than VWO, so I would recommend you to signup for a free trial of both the products and see which one is suited best to your needs." Secondly, we enable power-users: all of the changes you make in Optimizely can be transformed into JavaScript statements that correspond to those changes. We enable you to see the JavaScript that gets generated and even edit it or write your own if you want to create sophisticated transformations of your page (click the "Edit Code" button in the bottom-right corner in the editor). Third, we enable you to track any numbers of goals, even if you decide you want to measure a goal AFTER you've started your experiment. You can track any pageview, JavaScript event, or custom event and match these goals exactly, using a substring or regular expression. We keep all of the data you've sent us so you can always add and remove goals even if the experiment is still running. This gives you the flexibility and power to measure any goal you have on your website in real-time. In addition to these three we are also unique in offering automated cross-browser testing, behavioral targeting, undo/redo, and unlimited simultaneous experiments.
- yahelc 15y ago
  Without diving into the arcane specifics, I can safely say that I wouldn't have my sanity today without Optimizely. It's mind-numbingly simple to use, but also hugely powerful, and intelligently implemented. Many people seem to assume the power/appeal is in the visual editor, but they've advanced so far ahead of just being "visually created experiments", in both experiment design, targetting, implementation, and customization; they've entirely eliminated the headaches and fudging that usually comes with online testing tools. I'm as happy of a customer as they come.
- paraschopra 15y ago
  I'm the founder of Visual Website Optimizer. While Optimizely is strictly A/B testing, VWO is a much more comprehensive testing tool. In addition to A/B testing, we support Multivariate testing, Split URL testing, Heatmap/Clickmap, Targeting/Segmentation and Revenue Tracking. VWO is being used by companies that have sophisticated and comprehensive testing needs (like AMD, Microsoft, Groupon, etc. and thousands of SMBs) That said Optimizely may be slightly easier to use but with VWO our focus is on both: comprehensiveness of feature set _AND_ ease of use. Since both companies offer free trials, best is to signup for both of them and see which one is better suited to your needs.
usagi7 15y ago
I wish I could apply Optimizely AB Tests in everything I do. Like measuring the best way to word movie choices so that the one you secretly want gets chosen (conversion)! Anyway, interesting statistics. Larger sample size I think would definitely be more dramatic.
rokhayakebe 15y ago
How do you know AB test works if you aren't testing with the same user?
- Raykhenberg 15y ago
  A/B testing works by randomly distributing your traffic. We can't assume that every person is the same, but by splitting the traffic randomly we can assume that the two groups are similar. For a little bit more information on how our particular flavor of AB testing works check out http://optimizely.appspot.com/works http://optimizely.appspot.com/works
201studio 15y ago
I'd be interested to see some testing around using a '!' in a conversion button text as opposed to a period.
pkamb 15y ago
That "Enter Your website URL" field is really annoying. The easiest way to change the default "http://www.example.com http://www.example.com to your website's URL would normally be to double click the "example" and type your address, leaving the boilerplate "http://www. http://www. and ".com" and such. But you can't do that due to the fancy javascript and everything. Have to type the whole thing yourself. If the field cleared when you gave it focus, it wouldn't really matter and I'd just type in the URL myself. But the text remains, in the background, taunting you. It even appears to highlight the "example" part if you double click it.
- dsiroker 15y ago
  Sorry about that. The URL will fade out slightly when you focus on the textbox but I agree it is better to just clear it completely. We'll make that change.
  - ceejayoz 15y ago
    Why clear it? Why not leave http:// http:// in there? Might be worth A/B testing it. :-p
joeyespo 15y ago
It would be very cool if Optimizely could gather the A/B test results across all customers, run some statistical analysis, and publish which optimizations are significant. Sort of like an OKTrends for websites. That would be a great resource for initial usability and design decisions. After which, can be tweaked and further optimized by their product.
- dsiroker 15y ago
  Great idea! We'll do this.
  - gnaritas 15y ago
    That would rock.
destraynor 15y ago
As always you need to be wary of how these results are reported. AJ already pointed out that they're not measuring "conversions" in the sense of converted to paying customers, but converting to "entering a url in a field" The 29% increase is always misunderstood (by clients at least) The original page had a conversion rate of 8.9% The page they ended up with had a conversion rate of 11.5% The change is that an additional 2.6% of customers are now entering their urls in a field and clicking a button.
petercooper 15y ago
This is sorta related but since I concluded the test today and this post is here, I thought I'd share. I ran a 5 way split test for 9 days on my newsletter's signup page (JavaScript Weekly). My original page was the 2nd best performing but an identical page just without the subscriber count got a 8% higher conversion (or about 20% more signups in all) with a 90% confidence at the end of testing. The worst performer? A signup page with no screenshot preview of the newsletter. Sent conversions from about 37% down to a mere 3% (!!) Lesson learned? Always have visuals or screenshots on pages where you're trying to get people to sign up for things they aren't sure about.
- StavrosK 15y ago
  Were you looking at the statistics all the time to see when they passed 90% confidence? That invalidates them, just so you know. Basically, if you're watching it, it won't science.
  - petercooper 15y ago
    I'm not sure how tongue in cheek that was ;-) but 90% confidence was not a goal, at least. I just got bored after a week and wanted to move on. C'est la vie..
garindra 15y ago
Kinda out of topic, but I tried out top websites that use long-polling (like Quora), and it always fails. I guess the app waits for all the resources to load completely, which for long-polling websites, happens rather long after the DOMready event.
Eliezer 15y ago
Dear Optimizely: Your statistics do not tell you that B has a 95% chance of being better than A. Your statistics tell you when the excess of B over A has less than 5% probability, assuming B and A are actually equally effective. A Bayesian would understand this in terms of prior probabilities and likelihood ratios, but to put it into nontechnical terms, suppose that you tried out 15 different alterations and none of them seem to work. Then on the 16th, your detector goes off and says, "Less than 5% probability of these results arising by chance!" Do you conclude that it's 95% likely that this version is genuinely better? No, because the first 15 failed attempts told you that improving this webpage is actually pretty hard (the prior probability of an effective improvement is low), and now when you see that the 16th attempt has a result with a less than 5% probability of arising from chance, you figure "Eh, it's worth testing further, but probably it is just chance." Another extremely important point is that the classical statistics you learned to use to decide that something was <5% likely to arise by chance, only apply if you decided in advance to do exactly that many trials and then stop. Your chance of finding, on some trial, that your running total of results is "statistically significant", when A and B are actually identically effective, is considerably greater than 5%. See http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ... - a trial I ran with 500 fair coinflips had at least one step where the cumulative data "rejected the null hypothesis with p < 0.05" 30% of the time. You're not really to blame for this mistake, because the horrid non-Bayesian classical statistics taught in college are just about impossible to understand clearly; but it does sound to me like someone at your org needs to study (a) Bayes's Theorem (b) the case for reporting likelihood ratios rather than p-values (likelihood ratios are objective, p-values decidedly not) and (c) the beta distribution conjugate prior (which would make progress toward having priors and likelihood ratios over "These two pages have a single unknown conversion rate" or "These two pages have different unknown conversion rates"). Or in simpler terms, "Someone at your company needs to study Bayesian statistics, stat."
- meric 15y ago
  The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"? >> Another extremely important point is that the classical >> statistics you learned to use to decide that something >> was <5% likely to arise by chance, only apply if you >> decided in advance to do exactly that many trials and >> then stop. I agree. Not doing that I'd say it's "fudging with the numbers". I can't find on their article where they did this though.

146 points by dsiroker 15y ago

Does anyone know of a central repository for all of these little landing page optimization tweaks? I know each site is different and just to test to know for certain but there should be some general decisions vetted that would make a good boilerplate.

kristofferR 15y ago
http://www.abtests.com/ http://www.abtests.com/ is one
petercooper 15y ago
abtests (mentioned above) is awesome but there's also http://whichtestwon.com/ http://whichtestwon.com/

JackWebbHeller 15y ago

I got a lifetime free Optimizely account as part of an AppSumo deal. I can successfully say it's one of the simplest yet most powerful and effective web products I've ever used. The ease with which you can make changes is astonishing and there's no limits if you know a little jQuery.

AJ007 15y ago

They increased the number of people submitting their URL, there is no telling if that actually resulted in higher leads for them. What I have found is a simple landing page, that tells the user exactly what you are providing, and is free of any confusion, works the best over the long term. I've run hundreds of thousands of website visitors through Google Website Optimizer in multi-variate tests and what I've found is that over time there is little to no difference in conversion rate for minor landing page changes. The biggest jumps come from eliminating content in the design and clarifying the message. Looking at the small amount of users they sent to this landing page, I would call the results inconclusive. You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do. (its really chilling when you see how many drugs the FDA has approved over tiny margins of change/success.)

gnaritas 15y ago
> Looking at the small amount of users they sent to this landing page, I would call the results inconclusive The statistic say otherwise. A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result. > You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do. And you can rattle off personal anecdotes like this all day long, and the statistics are still more correct than you are and assert their margins of error and accuracy. The statistics are more correct than your intuition.
- mcn 15y ago
  Keep in mind that the 1% margin of error applies to the measurements of the conversion rates. The 29% has a significantly higher margin of error.
- Silhouette 15y ago
  > A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result. Well, no, it's a set of numbers with percentage signs after them. Perhaps the documentation for Optimizely specifies how their error margins etc. are derived, but nothing in the linked article does as far as I can see. Without knowing that underlying reasoning, all those pretty graphs and percentages are just a load of gobbledegook, apart from the original data points and the percentage increase figures derived directly from dividing them.
  - gnaritas 15y ago
    What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.
    - Silhouette 15y ago
      I did. A Google search for "error bars" site:optimizely.com turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written. If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested. Otherwise, I stand by my earlier comments.

sgrove 15y ago

We've been using Optimizely more and more extensively (we have a slightly unusual use case for it), and it's been fantastic at successfully ratcheting up conversions. We use it in concert with mixpanel when we need to push people along a funnel. The tool's insanely easy to implement, a joy to use, and I get to rely on them to tell me when something is statistically meaningful.

walrus 15y ago
What is your slightly unusual use case?
- sgrove 15y ago
  We actually deploy it across a number of client apps, rather than just our own site. We use it to run a dozen or so tests at any given time across multiple apps, and to quickly iterate on others' products. It's an interesting combination with mixpanel. If we could get one more thing from Mixpanel, we'd really be set - I'm bugging them, we'll see if they come out with it!

kristofferR 15y ago

How does Optimizely compare with Visual Website Optimizer?

dsiroker 15y ago
VWO is a great product but here are some of the unique benefits of using Optimizely: First is ease-of-use and simplicity: we hope Optimizely is a product anyone can use, not just IT or engineers. Paras Chopra, the creator of VWO recently said on Quora: "Optimizely is a great tool and perhaps even bit easier than VWO, so I would recommend you to signup for a free trial of both the products and see which one is suited best to your needs." Secondly, we enable power-users: all of the changes you make in Optimizely can be transformed into JavaScript statements that correspond to those changes. We enable you to see the JavaScript that gets generated and even edit it or write your own if you want to create sophisticated transformations of your page (click the "Edit Code" button in the bottom-right corner in the editor). Third, we enable you to track any numbers of goals, even if you decide you want to measure a goal AFTER you've started your experiment. You can track any pageview, JavaScript event, or custom event and match these goals exactly, using a substring or regular expression. We keep all of the data you've sent us so you can always add and remove goals even if the experiment is still running. This gives you the flexibility and power to measure any goal you have on your website in real-time. In addition to these three we are also unique in offering automated cross-browser testing, behavioral targeting, undo/redo, and unlimited simultaneous experiments.
yahelc 15y ago
Without diving into the arcane specifics, I can safely say that I wouldn't have my sanity today without Optimizely. It's mind-numbingly simple to use, but also hugely powerful, and intelligently implemented. Many people seem to assume the power/appeal is in the visual editor, but they've advanced so far ahead of just being "visually created experiments", in both experiment design, targetting, implementation, and customization; they've entirely eliminated the headaches and fudging that usually comes with online testing tools. I'm as happy of a customer as they come.
paraschopra 15y ago
I'm the founder of Visual Website Optimizer. While Optimizely is strictly A/B testing, VWO is a much more comprehensive testing tool. In addition to A/B testing, we support Multivariate testing, Split URL testing, Heatmap/Clickmap, Targeting/Segmentation and Revenue Tracking. VWO is being used by companies that have sophisticated and comprehensive testing needs (like AMD, Microsoft, Groupon, etc. and thousands of SMBs) That said Optimizely may be slightly easier to use but with VWO our focus is on both: comprehensiveness of feature set _AND_ ease of use. Since both companies offer free trials, best is to signup for both of them and see which one is better suited to your needs.

usagi7 15y ago

I wish I could apply Optimizely AB Tests in everything I do. Like measuring the best way to word movie choices so that the one you secretly want gets chosen (conversion)! Anyway, interesting statistics. Larger sample size I think would definitely be more dramatic.

rokhayakebe 15y ago

How do you know AB test works if you aren't testing with the same user?

Raykhenberg 15y ago
A/B testing works by randomly distributing your traffic. We can't assume that every person is the same, but by splitting the traffic randomly we can assume that the two groups are similar. For a little bit more information on how our particular flavor of AB testing works check out http://optimizely.appspot.com/works http://optimizely.appspot.com/works

201studio 15y ago

I'd be interested to see some testing around using a '!' in a conversion button text as opposed to a period.

pkamb 15y ago

That "Enter Your website URL" field is really annoying. The easiest way to change the default "http://www.example.com http://www.example.com to your website's URL would normally be to double click the "example" and type your address, leaving the boilerplate "http://www. http://www. and ".com" and such. But you can't do that due to the fancy javascript and everything. Have to type the whole thing yourself. If the field cleared when you gave it focus, it wouldn't really matter and I'd just type in the URL myself. But the text remains, in the background, taunting you. It even appears to highlight the "example" part if you double click it.

dsiroker 15y ago
Sorry about that. The URL will fade out slightly when you focus on the textbox but I agree it is better to just clear it completely. We'll make that change.
- ceejayoz 15y ago
  Why clear it? Why not leave http:// http:// in there? Might be worth A/B testing it. :-p

joeyespo 15y ago

It would be very cool if Optimizely could gather the A/B test results across all customers, run some statistical analysis, and publish which optimizations are significant. Sort of like an OKTrends for websites. That would be a great resource for initial usability and design decisions. After which, can be tweaked and further optimized by their product.

dsiroker 15y ago
Great idea! We'll do this.
- gnaritas 15y ago
  That would rock.

destraynor 15y ago

As always you need to be wary of how these results are reported. AJ already pointed out that they're not measuring "conversions" in the sense of converted to paying customers, but converting to "entering a url in a field" The 29% increase is always misunderstood (by clients at least) The original page had a conversion rate of 8.9% The page they ended up with had a conversion rate of 11.5% The change is that an additional 2.6% of customers are now entering their urls in a field and clicking a button.

petercooper 15y ago

This is sorta related but since I concluded the test today and this post is here, I thought I'd share. I ran a 5 way split test for 9 days on my newsletter's signup page (JavaScript Weekly). My original page was the 2nd best performing but an identical page just without the subscriber count got a 8% higher conversion (or about 20% more signups in all) with a 90% confidence at the end of testing. The worst performer? A signup page with no screenshot preview of the newsletter. Sent conversions from about 37% down to a mere 3% (!!) Lesson learned? Always have visuals or screenshots on pages where you're trying to get people to sign up for things they aren't sure about.

StavrosK 15y ago
Were you looking at the statistics all the time to see when they passed 90% confidence? That invalidates them, just so you know. Basically, if you're watching it, it won't science.
- petercooper 15y ago
  I'm not sure how tongue in cheek that was ;-) but 90% confidence was not a goal, at least. I just got bored after a week and wanted to move on. C'est la vie..

garindra 15y ago

Kinda out of topic, but I tried out top websites that use long-polling (like Quora), and it always fails. I guess the app waits for all the resources to load completely, which for long-polling websites, happens rather long after the DOMready event.

Eliezer 15y ago

Dear Optimizely: Your statistics do not tell you that B has a 95% chance of being better than A. Your statistics tell you when the excess of B over A has less than 5% probability, assuming B and A are actually equally effective. A Bayesian would understand this in terms of prior probabilities and likelihood ratios, but to put it into nontechnical terms, suppose that you tried out 15 different alterations and none of them seem to work. Then on the 16th, your detector goes off and says, "Less than 5% probability of these results arising by chance!" Do you conclude that it's 95% likely that this version is genuinely better? No, because the first 15 failed attempts told you that improving this webpage is actually pretty hard (the prior probability of an effective improvement is low), and now when you see that the 16th attempt has a result with a less than 5% probability of arising from chance, you figure "Eh, it's worth testing further, but probably it is just chance." Another extremely important point is that the classical statistics you learned to use to decide that something was <5% likely to arise by chance, only apply if you decided in advance to do exactly that many trials and then stop. Your chance of finding, on some trial, that your running total of results is "statistically significant", when A and B are actually identically effective, is considerably greater than 5%. See http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ... - a trial I ran with 500 fair coinflips had at least one step where the cumulative data "rejected the null hypothesis with p < 0.05" 30% of the time. You're not really to blame for this mistake, because the horrid non-Bayesian classical statistics taught in college are just about impossible to understand clearly; but it does sound to me like someone at your org needs to study (a) Bayes's Theorem (b) the case for reporting likelihood ratios rather than p-values (likelihood ratios are objective, p-values decidedly not) and (c) the beta distribution conjugate prior (which would make progress toward having priors and likelihood ratios over "These two pages have a single unknown conversion rate" or "These two pages have different unknown conversion rates"). Or in simpler terms, "Someone at your company needs to study Bayesian statistics, stat."

meric 15y ago
The p-value represents "Probability this is all luck". So why isn't 1 - p "Probability this isn't all luck"? >> Another extremely important point is that the classical >> statistics you learned to use to decide that something >> was <5% likely to arise by chance, only apply if you >> decided in advance to do exactly that many trials and >> then stop. I agree. Not doing that I'd say it's "fudging with the numbers". I can't find on their article where they did this though.