4 ms·

I don't understand people using a CMS thinking that Google Website Optimizer (GWO) for A/B tests is hard I got myself a plugin for WordPress that lets you inse

I don't understand people using a CMS thinking that Google Website Optimizer (GWO) for A/B tests is hard I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts I created a variation page on WordPress, identified the original one and the conversion page. On the site i created the test, started it and it's just working. But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard? Edit: clarified

dsiroker 16y ago
This is a great question! The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target. The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment. We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger. Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!
- paolomaffei 16y ago
  I see you're talking of A/B tests for BIG and medium sites then. If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it? Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
  - petekoomen 16y ago
    Great question. Even when it's just copying (and later removing) several javascript snippets on a page this can be a large hassle for several reasons: 1. For anyone who is non-technical, this can be daunting. Are the snippets in the right places? Was this deployed correctly? Have I set up my conversion goals correctly (GWO only allows for a single goal)? More often than not, tasks like this are passed from marketing onto IT, who are usually overworked as it is. 2. Even for technical folks, deploying and verifying a new version of one's site just to enable (and subsequently disable) a test can be a big hassle. In our experience, it's often the marketers, inside sales, and product managers in an organization who have the strongest interest in A/B testing. We're trying to eliminate the burden (and subsequent pain) that testing places on IT/engineering, freeing those stakeholders to test to their hearts' content! :)
paolomaffei 16y ago
Can't edit anymore: this is the plugin http://wordpress.org/extend/plugins/google-website-optimizer-for-wordpress/ http://wordpress.org/extend/plugins/google-website-optimizer...
patio11 16y ago
Using variation pages causes SEO and site problems when the test ends unless you manage your 301s properly, and the marginal load for the redirect to B artificially biases test results. Also, I did a bit of an eye popper demo at a company I just consulted with: deploying an AB test live on my site took two minutes twenty six seconds while slowing my typing speed to explain every step, while the company reports their experiments with GWO require 15 minutes at a minimum. That gives the typical engineer 12 reasons not to AB test.
- dsiroker 16y ago
  Hi Patrick, glad to hear that! Curious what you think about client-side A/B testing products in general (Optimizely being just one of them) vs. server-side products (like A/Bingo). Do you face pain points with A/Bingo that might be solved with a client-side javascript based solution?
  - patio11 16y ago
    In general, I feel server side is far superior for developers and client side with a WYSIWYG UI is a fabulous option for other stakeholders. I could talk on this subject for a while but my plane is about to take off. Suffice it to say you can't match power, flexibility, and speed of server side with client side testing. You can move away from just twiddling text to actually redoing workflows or business logic. I didn't write ABingo with a pain point included, but for Appointment Reminder it will be helpful to do AB testing in non-Rails marketing pages, too. I wouldn't append too much effort optimizing your solution for the needs of people who have written AB test frameworks though.
- bryanh 16y ago
  I would like to see an A/B test where A is no test and B is a traditional A/B test. ;-) My guess is, similar tests favor the "non-test" or quicker result/load, though I bet it would take A LOT of views to irk this out. Google always talk about these test on shades of blue, I imagine those tests are similar in the number of runs they require (hint: a lot).
  - 16y ago

dsiroker 16y ago

This is a great question! The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target. The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment. We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger. Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!

paolomaffei 16y ago
I see you're talking of A/B tests for BIG and medium sites then. If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it? Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
- petekoomen 16y ago
  Great question. Even when it's just copying (and later removing) several javascript snippets on a page this can be a large hassle for several reasons: 1. For anyone who is non-technical, this can be daunting. Are the snippets in the right places? Was this deployed correctly? Have I set up my conversion goals correctly (GWO only allows for a single goal)? More often than not, tasks like this are passed from marketing onto IT, who are usually overworked as it is. 2. Even for technical folks, deploying and verifying a new version of one's site just to enable (and subsequently disable) a test can be a big hassle. In our experience, it's often the marketers, inside sales, and product managers in an organization who have the strongest interest in A/B testing. We're trying to eliminate the burden (and subsequent pain) that testing places on IT/engineering, freeing those stakeholders to test to their hearts' content! :)

paolomaffei 16y ago

Can't edit anymore: this is the plugin http://wordpress.org/extend/plugins/google-website-optimizer-for-wordpress/ http://wordpress.org/extend/plugins/google-website-optimizer...

patio11 16y ago

Using variation pages causes SEO and site problems when the test ends unless you manage your 301s properly, and the marginal load for the redirect to B artificially biases test results. Also, I did a bit of an eye popper demo at a company I just consulted with: deploying an AB test live on my site took two minutes twenty six seconds while slowing my typing speed to explain every step, while the company reports their experiments with GWO require 15 minutes at a minimum. That gives the typical engineer 12 reasons not to AB test.

dsiroker 16y ago
Hi Patrick, glad to hear that! Curious what you think about client-side A/B testing products in general (Optimizely being just one of them) vs. server-side products (like A/Bingo). Do you face pain points with A/Bingo that might be solved with a client-side javascript based solution?
- patio11 16y ago
  In general, I feel server side is far superior for developers and client side with a WYSIWYG UI is a fabulous option for other stakeholders. I could talk on this subject for a while but my plane is about to take off. Suffice it to say you can't match power, flexibility, and speed of server side with client side testing. You can move away from just twiddling text to actually redoing workflows or business logic. I didn't write ABingo with a pain point included, but for Appointment Reminder it will be helpful to do AB testing in non-Rails marketing pages, too. I wouldn't append too much effort optimizing your solution for the needs of people who have written AB test frameworks though.
bryanh 16y ago
I would like to see an A/B test where A is no test and B is a traditional A/B test. ;-) My guess is, similar tests favor the "non-test" or quicker result/load, though I bet it would take A LOT of views to irk this out. Google always talk about these test on shades of blue, I imagine those tests are similar in the number of runs they require (hint: a lot).
- 16y ago