7 ms·

A/B Testing You'll Actually Use: Optimizely (YC W10) Launches Out of Beta

249 points by dsiroker 16y ago

It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public! Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started. We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback@optimizely.com and let us know what you think!

deleted 16y ago
[deleted]
petekoomen 16y ago
It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public! Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started. We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback at optimizely dot com and let us know what you think!
- CWIZO 16y ago
  A undo functionality would be super sweet! "Also on the front page: when you enter a URL and hit enter on your keyboard you are taken to some TechCrunch article :)" ... well scratch that, it does not happen any more. Great job otherwise, I'm "selling" this to my boss tomorrow :)
  - dsiroker 16y ago
    Thanks for the feedback! We've actually just implemented Undo! It didn't make it into the demo video but if you click on the little upside triangle next to a variation's title (e.g. "Variation #1") you have an option to Undo or Redo. There are keyboard shortcuts as well! As for the front page, is it possible you accidentally hit tab and then enter? If you hit enter right away, it works. If you hit tab, it changes focus to the first testimonial link on the page which is that TC article. We'll fix this so it doesn't happen in the future. Thanks again for the feedback!
    - CWIZO 16y ago
      It's quite possible I hit tab first yes. As for the undo: great. To be honest, I haven't even noticed the triangles, so that's why I wasn't aware of the feature.
- tnorthcutt 16y ago
  I actually got excited watching that video, and I don't even have something to use it for right now. Looks fantastic. I also really like how you have the plans setup to record stats from all visitors, but only unlock them if you buy the appropriate plan. I assume that means you can start out with the small plan, then upgrade and see stats that have already been recorded? If so, very cool. Based on the results in the promo video, looks like you should have your video autoplay! ;-)
  - dsiroker 16y ago
    I assume that means you can start out with the small plan, then upgrade and see stats that have already been recorded? If so, very cool. That's exactly right! Glad you like it. You can also start at a Silver or Gold plan for your 30-day free trial and downgrade at the end of your free trial if you like.
- petenixey 16y ago
  Awesome stuff Pete - even better than I expected it to be and the video's really good. Great to see you guys out in the wild!
avichal 16y ago
This is a really sweet product. Congrats on the launch!
xal 16y ago
It's a great product, we have been using it at Shopify with great success. Fantastic news that it's available to everyone now!
deleted 16y ago
[deleted]
rumpelstiltskin 16y ago
Anyone do a comparison of Optimizely and Visual Website Optimzer?
- paraschopra 16y ago
  DISCLAIMER: I'm the founder of Visual Website Optimizer but I will try to give a fair and honest comparision. Optimizely team, please feel free to correct me if I am wrong in my understanding. What VWO has that Optimizely doesn't - VWO has (JavaScript based) A/B and multivariate testing and split URL (based on traffic redirect) on while Optimizely is strictly (JavaScript based) A/B testing. You cannot do multivariate testing with it. - VWO has heatmap and clickmap reports while Optimizely doesn't - VWO has various advanced features such as visitor segmentation, cross-domain tracking, testing behind login wall pages, test results notifications, variation screen shot generation, etc. - VWO has team collaboration such as multiple-permission based logins and subaccounts. Optimizely doesn't have that. - In VWO, for all plans, CEO equally shares the support work ;) What Optimizely has that VWO doesn't - Changing layout of the page. Optimizely has a drag and drop feature in its test designer which is useful if you want to test different layouts (say sidebar to the right). To achieve the same in VWO, you need to enter some CSS (say float:right v/s float:left) as there is no drag-drop support (yet) - Since Optimizely exclusively focuses on A/B testing (and not MVT), in its test designer you can change several elements of the page (say headline, image and text) at once to create a variation. In VWO (currently), you need to change one element at once for A/B test and multiple elements for multivariate test. - In optimizely, results update on the page in real time. In VWO, results are also realtime but you have to refresh the report to see latest results (no auto-refreshing) I hope this was a fair comparision. I wish Optimizely team great luck for their product. I hope A/B testing market is big enough for multiple players to survive!
  - dsiroker 16y ago
    Great honest and fair comparison, Paras. One benefit you forgot to mention is that your cheapest plan ($49) comes out to $0.0049 per visitor. Optimizely's cheapest plan ($19) comes out to $0.0095 per visitor. One thing I think we do well is allow you to try Optimizely and create your experiment without needing to create an account or give us your email address. The only time you need to create an account is when you want to save. We also offer cross-browser testing and uptime monitoring and reporting. One small correction: we do support testing behind login wall pages. You just need to put the embed code on your page and the tool will load your site even if it's behind a login wall or on your own local intranet behind a firewall. Our biggest competitor is non-consumption. I'm glad there are other startups out there helping to educate the market about the benefits of A/B testing. A rising tide floats all boats!
paolomaffei 16y ago
I don't understand people using a CMS thinking that Google Website Optimizer (GWO) for A/B tests is hard I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts I created a variation page on WordPress, identified the original one and the conversion page. On the site i created the test, started it and it's just working. But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard? Edit: clarified
- dsiroker 16y ago
  This is a great question! The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target. The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment. We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger. Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!
  - paolomaffei 16y ago
    I see you're talking of A/B tests for BIG and medium sites then. If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it? Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
    - petekoomen 16y ago
deleted 16y ago
[deleted]
zacharyc 16y ago
This is a company of really smart and talented people. I have had limited experience with the beta, but have been following the project with much interest. I expect great things from Optimizely.
moe 16y ago
This looks extremely slick and well executed. Hats off, my feeling is you guys will be quite wealthy in short order.
- ericflo 16y ago
  I certainly hope so. Great product idea, well executed.
hammerdr 16y ago
Suggestions: In a couple of spots, the edit HTML dialog overlaid the items I was trying to edit. I attempted and expected to be able to move the edit dialog around the page and out of the way of what I was editing. On my own site, I was particularly wanting to play with colors/themes. Not sure what the best way to implement this is, but the only way I could do that through this interface was with style tags. Awesome product! I will likely use it in the near future.
- dsiroker 16y ago
  This is a great suggestion. We try to automatically put the dialog in the right place depending on the action you are trying to perform. For example, if you use drag-and-drop and move something to the right-edge of the window, we move the instructions dialog to the left-hand side of the element. This certainly isn't perfect and we'll work to improve it. Would you mind sending me the URL you were trying and some steps to repro the frustration you had? Feel free to email us at feedback@optimizely.com Thanks!
tsbaron 16y ago
This website is great. I was able to jump in and mess around with my website's home page within seconds after watching the video. I'm not a developer and being able to help out with A/B testing will be a huge help and one less thing our developers will have to deal with. Looking forward to using it!
Abid 16y ago
Fantastic stuff. I was wondering if users are just shown the different pages randomly or is there more granular control involved?
- dsiroker 16y ago
  Great question! Visitors are bucketed randomly into each variation. Once they have been bucketed they are cookied and will see the same variation over and over again if they reload the page You specify how much traffic you want to allocate to each bucket. You can do this when editing an experiment and going to "Advanced Settings" :: "Percentage of Traffic to Variations"
ssharp 16y ago
Can someone A/B test a web startup's name with words ending phonetically as "ly" or "er" versus a more "normal" name? Kidding aside, the product looks really great - congratulations on launching! For a critique, I'll say that the intro video is very dry. A/B it with something less Ben Stein? One thing I'm always interested in is pricing. How did you go about determining your various plans and rates?
- dsiroker 16y ago
  Sorry you thought the video was dry. I'm not the best voice actor in the world. We were thinking of putting a song from the Inception soundtrack in the background or using a real voice actor from http://www.voices.com/ http://www.voices.com/ Do you think either would help? Also, would love to share how we went about determining our various plans and rates. We did a lot of customer development on this and it might be interesting to other entrepreneurs. Let me know if you are interested and we'll blog about it.
  - btucker 16y ago
    Skip the soundtrack, but a professional voice (and more importantly professional recording) could do wonders for the perceived polishedness of the video. That being said, the content was enough to get me interested!
  - ssharp 16y ago
    I actually would be interested in seeing results of testing this video vs. a voice actor and maybe even against a set of videos that were split up and shorter.
    - dsiroker 16y ago
      Great idea! We're actually running an Optimizely experiment on the http://www.optimizely.com http://www.optimizely.com homepage right now. I won't give away what we're experimenting with but we'll blog about it once the experiment is over and share what we learned.
jyu 16y ago
Does this work well on https pages?
- dsiroker 16y ago
  Yes! https pages are a great place to start experimenting because they are so close to the conversion event most businesses care about: making money. Optimizely works seamlessly on https and http pages alike.
kevintwohy 16y ago
Just gave it a spin and it's a really cool product - feels a bit buggy at the moment but for the right application this would be a really valuable tool to have. Great job, guys.
- petekoomen 16y ago
  Hi Kevin, great to hear! I'm curious what felt buggy. Feel free to email us at support@optimizely.com and give us a heads up. We'd love to look into it!
justinchen 16y ago
Any chance this would work with dynamic content? i.e. a listings page that has similar structure from page to page but different content?
- dsiroker 16y ago
  Yes, we designed Optimizely with precisely this use case in mind. Here is the documentation on how to set this up: http://support.optimizely.com/faqs/getting-started/experiment-urls http://support.optimizely.com/faqs/getting-started/experimen... To clarify: If you want to run one experiment on a single dynamic page (not multiple dynamic pages) you can do that as long as the structure of the page does not change. The best way to find out is to just try it. Load one of these dynamic pages in the tool and create your variations. Then put the embed code on your site and click "Preview variation" from the variation drop-down menu (you can get there by clicking the small upside down triangle next to the variation title, e.g. "Variation #1"). If you have the embed code on your site this will show you the page on your live site. You do NOT need to start the experiment to do this. Hope that answers your question
  - justinchen 16y ago
    Awesome, I'll give it a try!
- deleted 16y ago
  [deleted]
cperciva 16y ago
You might want to adjust the pricing tiers. With $19/$79/$399 per month tiers, I have to guess that most people will start by trying the lowest tier; but that's limited to 2k visitors. If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky. You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around. EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.
- dsiroker 16y ago
  All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans. We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
  - cperciva 16y ago
    All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans. True, but that doesn't help people who don't get that much traffic within a 30 day period. We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month. Offering refunds is great, but it's even better if they're not needed. :-)
    - dsiroker 16y ago
      Offering refunds is great, but it's even better if they're not needed. :-) Certainly agree! :)
      - 16y ago
usagi7 16y ago
Looks really exciting. Sounds like a great idea. I'm looking forward to checking it out.
enjo 16y ago
Do any of these A/B testing tools support offline conversions in any meaningful way? For us a phone call is a conversion. I'd love to be able to build the logic to support that once (give me an API call along with GET parameters to the page that I can post back to you), and then be able to use tools like this to actually try different tests.
- petekoomen 16y ago
  Hi Enjo, great question. I'm curious--how would you use an API to track a phone call? If you can describe the process as you'd implement it in your org, we might be able to suggest (or build!) a solution..
- jonknee 16y ago
  I imagine it would be quite difficult to match up the phone call with what version of the website was being used. One idea: depending on how your incoming calls work you could try using different extensions on each version which would let you track it. A different phone number all together would work too, but might be confusing and bad for SEO.
- alastair 16y ago
  Couldn't you setup 2 numbers and just A/B that along with whatever else you're changing?
Ataraxy 16y ago
What's the USP on Optimzely when compared to the already quite formidably awesome Visual Website Optimizer? Also, as someone that really wants to incorporate this tech into my larger scale marketing efforts I am put off by the pricing models to both of the above mentioned sources. When only one of several ad campaigns are doing 100k visitors a day you can see how that pricing model does me no good. Particularly when you take into consideration that I never stop testing -something- on a campaigns landing page. Also would love an API so that something like this could be tightly integrated into my custom conversion analytics solution. Congratulations on your launch nonetheless.
- dsiroker 16y ago
  Right now our biggest competitor is non-consumption. Businesses see the value in A/B testing but just aren't doing it. The pain points we are trying to solve are setup & implementation, easy-to-understand real-time results, and resolving the marketing & IT impasse within organizations when it comes to doing A/B testing: marketing wants to do it, IT doesn't have time to help. We want to enable marketers to run A/B tests without having to rely on engineering or IT. Great feedback on the pricing model. I agree this isn't perfect. We do offer a 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason, we'd be happy to offer you a full refund for the month.
  - mahmud 16y ago
    Right now our biggest competitor is non-consumption You think like a winner, my man. You had your opportunity to rag at the "competition" but you were wise enough to see the big picture. Cheers!
theospears 16y ago
Disclaimer: I have watched the video but not tried the product, so some of these observations may not apply to the the product itself. Overall, this looks like a great product. I know a number of people who find Google Website Optimizer complicated to use, and I would definitely recommend this to them as a simpler option. I love how slick the browser interface to edit pages is, and I think having the default 'engagement' metric so people can see results without having to set up a goal page is a brilliant idea. There are a few things you have done which I would consider doing differently. 1. It looks like your mission is to make A/B testing really easy, but your pricing page at the moment doesn't really reflect that. Number of visitors tested is an easy metric, but one that it is hard for me to interpret without lots of knowledge of A/B testing. How many tests does this mean I can run how quickly? I would also reconsider the additional features you offer in premium packages. Cross-browser testing sounds complex and makes me worry that your site edits will fail in IE6. I don't want to have to test it, I just want it to work. With uptime monitoring, what does this have to do with A/B testing? Bigger sites probably already have some form of monitoring already anyway, so it looks like they are going to pay for something they don't need. I think your core product is strong enough that you don't need to offer these. 2. Showing me the percentage significance level appeals to my inner stats nerd, but I suspect the sort of people I think will benefit most from Optimizely will have difficulty interpreting this number. What level is 'ok'? Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I need 5/5? Google deal with this very well with their bars which turn red or green when they reach significance. 3. The 'select container' option to expand the selection seems non-obvious, and isn't how multi-select works in any other interface I've seen. Maybe allow people to select multiple components and then take their deepest common parent? There are also some additional features I personally would like to see 1. It would be great if you gave an estimate for how long until my experiment will reach an appropriate significance level (obviously based on % change and traffic seen so far). 2. I would like to be able to choose my conversion action by clicking on a form button or link in your page editor. 3. It would be amazingly useful to have some automatic suggestions for how a page could be changed - on many occasions I've seen people resist A/B testing because the options are so wide and they don't know what to do. Doing this for some simple suggestions sounds possible - e.g. making key links bigger and moving them up the page. Doing anything more sophisticated could be a good challenge though :-)
mcotton 16y ago
This looks great. It looks like you are using google app engine, maybe a future post about how well it is working out for you.
- dsiroker 16y ago
  Great idea! I'll try to convince my co-founder to write a blog post about his experience going from product manager for Google App Engine to coder using GAE every day as a customer!
reedlaw 16y ago
I just went to dropbox.com and the play button got larger and larger each time I hovered over it. Could this be a bug, or a weird optimization test?
- petekoomen 16y ago
  Hi Reedlaw--I'm not able to reproduce this. I'm assuming you mean you created an experiment on dropbox.com and that the play button somehow gets larger every time you hover your mouse over it. What browser are you using? Thanks! Pete
aqwan 16y ago
Very impressive. I will definitely share this among my peers.
senko 16y ago
Great stuff, the video made me want to jump right in and test stuff; too bad I don't have any medium/big sites to test :) For us folks with small-traffic websites (ie. your Bronze plan), an useful addition would be a suggestion to people to not try too many variations at the same time. (I don't have the math, but I assume less variations means less datapoints until you get significant results) Perhaps, as another commenter suggested, by estimating how long (in time or visitors) it will take - and by showing this estimate while adding variations.
- dsiroker 16y ago
  You are correct, the more variations, the longer the experiment will need to run to reach statistical significance. Estimating how long it will take to finish is a great suggestion and something we plan on doing.
JonM 16y ago
This looks brilliant, you just need to nail your pricing structure.. perhaps consider per test or limit the number of tests and variations? Either way, congratulations, some serious skills on show form your engineer(s)!
brc 16y ago
I get the impression that home page has been optimised to within an inch of it's life and converts like crazy. A very impressive landing page, I had the overwhelming urge to put in my Url and try it out. Well done.
- prawn 16y ago
  I didn't believe it could possibly be that easy until I watched the video. Subsequently, I think they should add text to that effect near the video. After all the positive talk on HN, I was expecting something clever but was still surprised by the implementation (single line to paste in!). I doubt I'm the only one who doesn't always jump to watch demo videos (unless referred by HN) or throw my URL into a strange form. For Ask HN's, I usually just check out the design and try and get a rough idea of the concept, FWIW.
rmoriz 16y ago
Will it be possible to define URLs to different, already existing versions (as google does)? Your online editing is awesome esp. for small things but sometimes one might be able to render already existing pages…
Swoopey 16y ago
This is great, just in time for my testing :), I'll definitely give it a try! I would love for one of your pricing packages to include speaking directly with an optimization consultant, or perhaps receiving a detailed report that includes optimization recommendations.
SudarshanP 16y ago
Really cool A/B Testing tool :) Your variation editor looks really awesome too. How much effort does it take create something like that? I mean just the HTML hover/manipulation etc. I am planning to write a scraper related tool similar to http://open.dapper.net/ http://open.dapper.net/ Did you encounter any serious cross browser issues? Or any other notable info you would like to share with the HN community. Is there any open source basic implementation that can be used as a starting point? Do you preprocess the page on the server or is it all Javascript magic on the client side?
- photon_off 16y ago
  Dapper is pretty cool. Good luck in making a similar scraper tool. As for doing the manipulation: They're using jQuery for all of that. The current hover/manipulation stuff they're doing seems pretty simple. So simple, that it makes me want to make a competing service. For the hover effects, you can do something like: <style> .bordered { border: 4px solid blue; } </style> <script> var curHltdObj = null; $("div").live("mouseover mouseout", function(e){ if (curHltdObj) curHltdObj.removeClass("bordered"); if (e.type=="mouseover"){ $(this).addClass("bordered") curHltdObj = $(this); } }); </script> This is just a starting point. What I would do is put an overlay on top of the hovered item instead. This gives you move flexibility in terms of styling and event control. I just checked out optimizely, and this is actually what they do. Though, curiously, they put their overlay element in before the <body> of the iframe document, but nevertheless it works (in FF, at least). That actual page editing is incredibly simple as well. For the most part, you position everything relative to where it was. For normal (non positioned) elements, you just set them to position:relative, and set their top/left to be however far the user dragged the item. For other elements, you tack on how the user moved the item to their already existent top/left coordinates. Again, I'm sure it's more complicated than that, but that's the gist of it.
- dsiroker 16y ago
  Our biggest challenge in building this tool was compatibility. Every webpage is different and even seemingly innocuous things things like frame busters and javascript redirects presented quite a challenge for us. We were fortunate to have a great set of early beta customers that gave us feedback when things didn't work and we took it one website at a time. We'd love to share the lessons we learned along the way with the HN community in a blog post. Let me know what you're most interested in and we'll share what we've learned.
revorad 16y ago
Great job on the product and video - really easy and slick. What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - http://news.ycombinator.com/item?id=1778104 http://news.ycombinator.com/item?id=1778104.
- dsiroker 16y ago
  Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days. Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.
  - revorad 16y ago
    During the Obama campaign Whoaa why isn't there a crazy big testimonial screaming that on the homepage? [EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.] I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand. The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.
- btilly 16y ago
  That thread is seriously wrong. Here is an excerpt from an email discussion I had recently that touched on http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html. Evan Miller has a point, but not as good of one as he thinks. It is true that multiple peeks mean that eventually any test will find significance at any level you want. However in A/B tests the peeks are not independent. This greatly weakens the effect he is talking about. Section 7 of my presentation, starting at http://elem.com/~btilly/effective-ab-testing/#slide59 http://elem.com/~btilly/effective-ab-testing/#slide59, is about the question of how long it takes for a test to complete. For that I ran numerical experiments with constant peeking, literally every time you add one to A and one to B you peek again. You can see graphs of how many errors there were, and how long it takes to get an answer. Here are key points: - Be suspicious of tests that end quickly. Run them a bit longer on general principal. (In general I'd call 500 people a very small test.) - Nobody can predict how long a test will take. Even if you know the actual improvement, you still can't predict time to within an order of magnitude. - If a test has been running for a long time you know the true difference is small, so there is no harm in accepting whatever answer it gives.
davidnelson 16y ago
nice idea. I tried to setup a new a/b test for a site and it became 'not easy' when I wanted to track an adsense click as a goal. I wrote a manual a/b test for the same site about 7 years ago, and got adsense click goals working with 40 variations of the page, after about a week of working on it after work at night after my regular job. There is some jquery code you can add to track javascript events which would probably track adsense clicks, but it's 'not easy', ie: I might as well not pay for it. Cool idea though! I would pay for it if I could track adsense clicks without having to write code. Also, having an option to select multiple dom elements at once would be nice. In addition, logging into the web site and editing the html to add the code in the <head> tag would be nice too, especially since it's kind of a hassle for me to open eclipse and re-deploy to app engine at 1am when I am tired. haha, hope this helps.
- dsiroker 16y ago
  Certainly something we are well aware of and trying to improve! We'd love to automatically track ad clicks for you and provide the corresponding reporting to let you know which variation gets you the most clicks. Please email us at feedback@optimizely.com and let us know if you're interested in this and I'd love to learn more about this use case.
  - photon_off 16y ago
    I think it'd be amazing if you could work smething out with the ppl at mouseflow or thier ilk. They could track the clicks and mouse movements for you. Maybe offer it as an add on for a test.
dav-id 16y ago
Why do all of these companies create cookie cutter names like 'Optimizely' is 'ly' the next .com? To me ly says 'We have no track record, use with caution' I would feel more comfortable using something with a name that isn't following some online fashion.
bretthellman 16y ago
Optimizely sounds like a great service but after entering my URL it took a long time to load. It would be great if you could speed that up... If I hadn't heard of the link from Hacker News, I would have closed the tab after a few seconds.
- dsiroker 16y ago
  Great feedback and certainly something we plan to improve. Thanks for bearing with us!

249 points by dsiroker 16y ago

deleted 16y ago

[deleted]

petekoomen 16y ago

CWIZO 16y ago
A undo functionality would be super sweet! "Also on the front page: when you enter a URL and hit enter on your keyboard you are taken to some TechCrunch article :)" ... well scratch that, it does not happen any more. Great job otherwise, I'm "selling" this to my boss tomorrow :)
- dsiroker 16y ago
  Thanks for the feedback! We've actually just implemented Undo! It didn't make it into the demo video but if you click on the little upside triangle next to a variation's title (e.g. "Variation #1") you have an option to Undo or Redo. There are keyboard shortcuts as well! As for the front page, is it possible you accidentally hit tab and then enter? If you hit enter right away, it works. If you hit tab, it changes focus to the first testimonial link on the page which is that TC article. We'll fix this so it doesn't happen in the future. Thanks again for the feedback!
  - CWIZO 16y ago
    It's quite possible I hit tab first yes. As for the undo: great. To be honest, I haven't even noticed the triangles, so that's why I wasn't aware of the feature.
tnorthcutt 16y ago
I actually got excited watching that video, and I don't even have something to use it for right now. Looks fantastic. I also really like how you have the plans setup to record stats from all visitors, but only unlock them if you buy the appropriate plan. I assume that means you can start out with the small plan, then upgrade and see stats that have already been recorded? If so, very cool. Based on the results in the promo video, looks like you should have your video autoplay! ;-)
- dsiroker 16y ago
  I assume that means you can start out with the small plan, then upgrade and see stats that have already been recorded? If so, very cool. That's exactly right! Glad you like it. You can also start at a Silver or Gold plan for your 30-day free trial and downgrade at the end of your free trial if you like.
petenixey 16y ago
Awesome stuff Pete - even better than I expected it to be and the video's really good. Great to see you guys out in the wild!

avichal 16y ago

This is a really sweet product. Congrats on the launch!

xal 16y ago

It's a great product, we have been using it at Shopify with great success. Fantastic news that it's available to everyone now!

deleted 16y ago

[deleted]

rumpelstiltskin 16y ago

Anyone do a comparison of Optimizely and Visual Website Optimzer?

paraschopra 16y ago
DISCLAIMER: I'm the founder of Visual Website Optimizer but I will try to give a fair and honest comparision. Optimizely team, please feel free to correct me if I am wrong in my understanding. What VWO has that Optimizely doesn't - VWO has (JavaScript based) A/B and multivariate testing and split URL (based on traffic redirect) on while Optimizely is strictly (JavaScript based) A/B testing. You cannot do multivariate testing with it. - VWO has heatmap and clickmap reports while Optimizely doesn't - VWO has various advanced features such as visitor segmentation, cross-domain tracking, testing behind login wall pages, test results notifications, variation screen shot generation, etc. - VWO has team collaboration such as multiple-permission based logins and subaccounts. Optimizely doesn't have that. - In VWO, for all plans, CEO equally shares the support work ;) What Optimizely has that VWO doesn't - Changing layout of the page. Optimizely has a drag and drop feature in its test designer which is useful if you want to test different layouts (say sidebar to the right). To achieve the same in VWO, you need to enter some CSS (say float:right v/s float:left) as there is no drag-drop support (yet) - Since Optimizely exclusively focuses on A/B testing (and not MVT), in its test designer you can change several elements of the page (say headline, image and text) at once to create a variation. In VWO (currently), you need to change one element at once for A/B test and multiple elements for multivariate test. - In optimizely, results update on the page in real time. In VWO, results are also realtime but you have to refresh the report to see latest results (no auto-refreshing) I hope this was a fair comparision. I wish Optimizely team great luck for their product. I hope A/B testing market is big enough for multiple players to survive!
- dsiroker 16y ago
  Great honest and fair comparison, Paras. One benefit you forgot to mention is that your cheapest plan ($49) comes out to $0.0049 per visitor. Optimizely's cheapest plan ($19) comes out to $0.0095 per visitor. One thing I think we do well is allow you to try Optimizely and create your experiment without needing to create an account or give us your email address. The only time you need to create an account is when you want to save. We also offer cross-browser testing and uptime monitoring and reporting. One small correction: we do support testing behind login wall pages. You just need to put the embed code on your page and the tool will load your site even if it's behind a login wall or on your own local intranet behind a firewall. Our biggest competitor is non-consumption. I'm glad there are other startups out there helping to educate the market about the benefits of A/B testing. A rising tide floats all boats!

paolomaffei 16y ago

I don't understand people using a CMS thinking that Google Website Optimizer (GWO) for A/B tests is hard I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts I created a variation page on WordPress, identified the original one and the conversion page. On the site i created the test, started it and it's just working. But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard? Edit: clarified

dsiroker 16y ago
This is a great question! The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target. The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment. We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger. Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!
- paolomaffei 16y ago
  I see you're talking of A/B tests for BIG and medium sites then. If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it? Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
  - petekoomen 16y ago

deleted 16y ago

[deleted]

zacharyc 16y ago

This is a company of really smart and talented people. I have had limited experience with the beta, but have been following the project with much interest. I expect great things from Optimizely.

moe 16y ago

This looks extremely slick and well executed. Hats off, my feeling is you guys will be quite wealthy in short order.

ericflo 16y ago
I certainly hope so. Great product idea, well executed.

hammerdr 16y ago

Suggestions: In a couple of spots, the edit HTML dialog overlaid the items I was trying to edit. I attempted and expected to be able to move the edit dialog around the page and out of the way of what I was editing. On my own site, I was particularly wanting to play with colors/themes. Not sure what the best way to implement this is, but the only way I could do that through this interface was with style tags. Awesome product! I will likely use it in the near future.

dsiroker 16y ago
This is a great suggestion. We try to automatically put the dialog in the right place depending on the action you are trying to perform. For example, if you use drag-and-drop and move something to the right-edge of the window, we move the instructions dialog to the left-hand side of the element. This certainly isn't perfect and we'll work to improve it. Would you mind sending me the URL you were trying and some steps to repro the frustration you had? Feel free to email us at feedback@optimizely.com Thanks!

tsbaron 16y ago

This website is great. I was able to jump in and mess around with my website's home page within seconds after watching the video. I'm not a developer and being able to help out with A/B testing will be a huge help and one less thing our developers will have to deal with. Looking forward to using it!

Abid 16y ago

Fantastic stuff. I was wondering if users are just shown the different pages randomly or is there more granular control involved?

dsiroker 16y ago
Great question! Visitors are bucketed randomly into each variation. Once they have been bucketed they are cookied and will see the same variation over and over again if they reload the page You specify how much traffic you want to allocate to each bucket. You can do this when editing an experiment and going to "Advanced Settings" :: "Percentage of Traffic to Variations"

ssharp 16y ago

Can someone A/B test a web startup's name with words ending phonetically as "ly" or "er" versus a more "normal" name? Kidding aside, the product looks really great - congratulations on launching! For a critique, I'll say that the intro video is very dry. A/B it with something less Ben Stein? One thing I'm always interested in is pricing. How did you go about determining your various plans and rates?

dsiroker 16y ago
Sorry you thought the video was dry. I'm not the best voice actor in the world. We were thinking of putting a song from the Inception soundtrack in the background or using a real voice actor from http://www.voices.com/ http://www.voices.com/ Do you think either would help? Also, would love to share how we went about determining our various plans and rates. We did a lot of customer development on this and it might be interesting to other entrepreneurs. Let me know if you are interested and we'll blog about it.
- btucker 16y ago
  Skip the soundtrack, but a professional voice (and more importantly professional recording) could do wonders for the perceived polishedness of the video. That being said, the content was enough to get me interested!
- ssharp 16y ago
  I actually would be interested in seeing results of testing this video vs. a voice actor and maybe even against a set of videos that were split up and shorter.
  - dsiroker 16y ago
    Great idea! We're actually running an Optimizely experiment on the http://www.optimizely.com http://www.optimizely.com homepage right now. I won't give away what we're experimenting with but we'll blog about it once the experiment is over and share what we learned.

jyu 16y ago

Does this work well on https pages?

dsiroker 16y ago
Yes! https pages are a great place to start experimenting because they are so close to the conversion event most businesses care about: making money. Optimizely works seamlessly on https and http pages alike.

kevintwohy 16y ago

Just gave it a spin and it's a really cool product - feels a bit buggy at the moment but for the right application this would be a really valuable tool to have. Great job, guys.

petekoomen 16y ago
Hi Kevin, great to hear! I'm curious what felt buggy. Feel free to email us at support@optimizely.com and give us a heads up. We'd love to look into it!

justinchen 16y ago

Any chance this would work with dynamic content? i.e. a listings page that has similar structure from page to page but different content?

dsiroker 16y ago
Yes, we designed Optimizely with precisely this use case in mind. Here is the documentation on how to set this up: http://support.optimizely.com/faqs/getting-started/experiment-urls http://support.optimizely.com/faqs/getting-started/experimen... To clarify: If you want to run one experiment on a single dynamic page (not multiple dynamic pages) you can do that as long as the structure of the page does not change. The best way to find out is to just try it. Load one of these dynamic pages in the tool and create your variations. Then put the embed code on your site and click "Preview variation" from the variation drop-down menu (you can get there by clicking the small upside down triangle next to the variation title, e.g. "Variation #1"). If you have the embed code on your site this will show you the page on your live site. You do NOT need to start the experiment to do this. Hope that answers your question
- justinchen 16y ago
  Awesome, I'll give it a try!
deleted 16y ago
[deleted]

cperciva 16y ago

You might want to adjust the pricing tiers. With $19/$79/$399 per month tiers, I have to guess that most people will start by trying the lowest tier; but that's limited to 2k visitors. If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky. You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around. EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.

dsiroker 16y ago
All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans. We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
- cperciva 16y ago
  All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans. True, but that doesn't help people who don't get that much traffic within a 30 day period. We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month. Offering refunds is great, but it's even better if they're not needed. :-)
  - dsiroker 16y ago
    Offering refunds is great, but it's even better if they're not needed. :-) Certainly agree! :)
    - 16y ago

usagi7 16y ago

Looks really exciting. Sounds like a great idea. I'm looking forward to checking it out.

enjo 16y ago

Do any of these A/B testing tools support offline conversions in any meaningful way? For us a phone call is a conversion. I'd love to be able to build the logic to support that once (give me an API call along with GET parameters to the page that I can post back to you), and then be able to use tools like this to actually try different tests.

petekoomen 16y ago
Hi Enjo, great question. I'm curious--how would you use an API to track a phone call? If you can describe the process as you'd implement it in your org, we might be able to suggest (or build!) a solution..
jonknee 16y ago
I imagine it would be quite difficult to match up the phone call with what version of the website was being used. One idea: depending on how your incoming calls work you could try using different extensions on each version which would let you track it. A different phone number all together would work too, but might be confusing and bad for SEO.
alastair 16y ago
Couldn't you setup 2 numbers and just A/B that along with whatever else you're changing?

Ataraxy 16y ago

What's the USP on Optimzely when compared to the already quite formidably awesome Visual Website Optimizer? Also, as someone that really wants to incorporate this tech into my larger scale marketing efforts I am put off by the pricing models to both of the above mentioned sources. When only one of several ad campaigns are doing 100k visitors a day you can see how that pricing model does me no good. Particularly when you take into consideration that I never stop testing -something- on a campaigns landing page. Also would love an API so that something like this could be tightly integrated into my custom conversion analytics solution. Congratulations on your launch nonetheless.

dsiroker 16y ago
Right now our biggest competitor is non-consumption. Businesses see the value in A/B testing but just aren't doing it. The pain points we are trying to solve are setup & implementation, easy-to-understand real-time results, and resolving the marketing & IT impasse within organizations when it comes to doing A/B testing: marketing wants to do it, IT doesn't have time to help. We want to enable marketers to run A/B tests without having to rely on engineering or IT. Great feedback on the pricing model. I agree this isn't perfect. We do offer a 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason, we'd be happy to offer you a full refund for the month.
- mahmud 16y ago
  Right now our biggest competitor is non-consumption You think like a winner, my man. You had your opportunity to rag at the "competition" but you were wise enough to see the big picture. Cheers!

theospears 16y ago

Disclaimer: I have watched the video but not tried the product, so some of these observations may not apply to the the product itself. Overall, this looks like a great product. I know a number of people who find Google Website Optimizer complicated to use, and I would definitely recommend this to them as a simpler option. I love how slick the browser interface to edit pages is, and I think having the default 'engagement' metric so people can see results without having to set up a goal page is a brilliant idea. There are a few things you have done which I would consider doing differently. 1. It looks like your mission is to make A/B testing really easy, but your pricing page at the moment doesn't really reflect that. Number of visitors tested is an easy metric, but one that it is hard for me to interpret without lots of knowledge of A/B testing. How many tests does this mean I can run how quickly? I would also reconsider the additional features you offer in premium packages. Cross-browser testing sounds complex and makes me worry that your site edits will fail in IE6. I don't want to have to test it, I just want it to work. With uptime monitoring, what does this have to do with A/B testing? Bigger sites probably already have some form of monitoring already anyway, so it looks like they are going to pay for something they don't need. I think your core product is strong enough that you don't need to offer these. 2. Showing me the percentage significance level appeals to my inner stats nerd, but I suspect the sort of people I think will benefit most from Optimizely will have difficulty interpreting this number. What level is 'ok'? Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I need 5/5? Google deal with this very well with their bars which turn red or green when they reach significance. 3. The 'select container' option to expand the selection seems non-obvious, and isn't how multi-select works in any other interface I've seen. Maybe allow people to select multiple components and then take their deepest common parent? There are also some additional features I personally would like to see 1. It would be great if you gave an estimate for how long until my experiment will reach an appropriate significance level (obviously based on % change and traffic seen so far). 2. I would like to be able to choose my conversion action by clicking on a form button or link in your page editor. 3. It would be amazingly useful to have some automatic suggestions for how a page could be changed - on many occasions I've seen people resist A/B testing because the options are so wide and they don't know what to do. Doing this for some simple suggestions sounds possible - e.g. making key links bigger and moving them up the page. Doing anything more sophisticated could be a good challenge though :-)

mcotton 16y ago

This looks great. It looks like you are using google app engine, maybe a future post about how well it is working out for you.

dsiroker 16y ago
Great idea! I'll try to convince my co-founder to write a blog post about his experience going from product manager for Google App Engine to coder using GAE every day as a customer!

reedlaw 16y ago

I just went to dropbox.com and the play button got larger and larger each time I hovered over it. Could this be a bug, or a weird optimization test?

petekoomen 16y ago
Hi Reedlaw--I'm not able to reproduce this. I'm assuming you mean you created an experiment on dropbox.com and that the play button somehow gets larger every time you hover your mouse over it. What browser are you using? Thanks! Pete

aqwan 16y ago

Very impressive. I will definitely share this among my peers.

senko 16y ago

Great stuff, the video made me want to jump right in and test stuff; too bad I don't have any medium/big sites to test :) For us folks with small-traffic websites (ie. your Bronze plan), an useful addition would be a suggestion to people to not try too many variations at the same time. (I don't have the math, but I assume less variations means less datapoints until you get significant results) Perhaps, as another commenter suggested, by estimating how long (in time or visitors) it will take - and by showing this estimate while adding variations.

dsiroker 16y ago
You are correct, the more variations, the longer the experiment will need to run to reach statistical significance. Estimating how long it will take to finish is a great suggestion and something we plan on doing.

JonM 16y ago

This looks brilliant, you just need to nail your pricing structure.. perhaps consider per test or limit the number of tests and variations? Either way, congratulations, some serious skills on show form your engineer(s)!

brc 16y ago

I get the impression that home page has been optimised to within an inch of it's life and converts like crazy. A very impressive landing page, I had the overwhelming urge to put in my Url and try it out. Well done.

prawn 16y ago
I didn't believe it could possibly be that easy until I watched the video. Subsequently, I think they should add text to that effect near the video. After all the positive talk on HN, I was expecting something clever but was still surprised by the implementation (single line to paste in!). I doubt I'm the only one who doesn't always jump to watch demo videos (unless referred by HN) or throw my URL into a strange form. For Ask HN's, I usually just check out the design and try and get a rough idea of the concept, FWIW.

rmoriz 16y ago

Will it be possible to define URLs to different, already existing versions (as google does)? Your online editing is awesome esp. for small things but sometimes one might be able to render already existing pages…

Swoopey 16y ago

This is great, just in time for my testing :), I'll definitely give it a try! I would love for one of your pricing packages to include speaking directly with an optimization consultant, or perhaps receiving a detailed report that includes optimization recommendations.

SudarshanP 16y ago

Really cool A/B Testing tool :) Your variation editor looks really awesome too. How much effort does it take create something like that? I mean just the HTML hover/manipulation etc. I am planning to write a scraper related tool similar to http://open.dapper.net/ http://open.dapper.net/ Did you encounter any serious cross browser issues? Or any other notable info you would like to share with the HN community. Is there any open source basic implementation that can be used as a starting point? Do you preprocess the page on the server or is it all Javascript magic on the client side?

photon_off 16y ago
Dapper is pretty cool. Good luck in making a similar scraper tool. As for doing the manipulation: They're using jQuery for all of that. The current hover/manipulation stuff they're doing seems pretty simple. So simple, that it makes me want to make a competing service. For the hover effects, you can do something like: <style> .bordered { border: 4px solid blue; } </style> <script> var curHltdObj = null; $("div").live("mouseover mouseout", function(e){ if (curHltdObj) curHltdObj.removeClass("bordered"); if (e.type=="mouseover"){ $(this).addClass("bordered") curHltdObj = $(this); } }); </script> This is just a starting point. What I would do is put an overlay on top of the hovered item instead. This gives you move flexibility in terms of styling and event control. I just checked out optimizely, and this is actually what they do. Though, curiously, they put their overlay element in before the <body> of the iframe document, but nevertheless it works (in FF, at least). That actual page editing is incredibly simple as well. For the most part, you position everything relative to where it was. For normal (non positioned) elements, you just set them to position:relative, and set their top/left to be however far the user dragged the item. For other elements, you tack on how the user moved the item to their already existent top/left coordinates. Again, I'm sure it's more complicated than that, but that's the gist of it.
dsiroker 16y ago
Our biggest challenge in building this tool was compatibility. Every webpage is different and even seemingly innocuous things things like frame busters and javascript redirects presented quite a challenge for us. We were fortunate to have a great set of early beta customers that gave us feedback when things didn't work and we took it one website at a time. We'd love to share the lessons we learned along the way with the HN community in a blog post. Let me know what you're most interested in and we'll share what we've learned.

revorad 16y ago

Great job on the product and video - really easy and slick. What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - http://news.ycombinator.com/item?id=1778104 http://news.ycombinator.com/item?id=1778104.

dsiroker 16y ago
Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make. During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days. Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.
- revorad 16y ago
  During the Obama campaign Whoaa why isn't there a crazy big testimonial screaming that on the homepage? [EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.] I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand. The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.
btilly 16y ago
That thread is seriously wrong. Here is an excerpt from an email discussion I had recently that touched on http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/how-not-to-run-an-ab-test.html. Evan Miller has a point, but not as good of one as he thinks. It is true that multiple peeks mean that eventually any test will find significance at any level you want. However in A/B tests the peeks are not independent. This greatly weakens the effect he is talking about. Section 7 of my presentation, starting at http://elem.com/~btilly/effective-ab-testing/#slide59 http://elem.com/~btilly/effective-ab-testing/#slide59, is about the question of how long it takes for a test to complete. For that I ran numerical experiments with constant peeking, literally every time you add one to A and one to B you peek again. You can see graphs of how many errors there were, and how long it takes to get an answer. Here are key points: - Be suspicious of tests that end quickly. Run them a bit longer on general principal. (In general I'd call 500 people a very small test.) - Nobody can predict how long a test will take. Even if you know the actual improvement, you still can't predict time to within an order of magnitude. - If a test has been running for a long time you know the true difference is small, so there is no harm in accepting whatever answer it gives.

davidnelson 16y ago

nice idea. I tried to setup a new a/b test for a site and it became 'not easy' when I wanted to track an adsense click as a goal. I wrote a manual a/b test for the same site about 7 years ago, and got adsense click goals working with 40 variations of the page, after about a week of working on it after work at night after my regular job. There is some jquery code you can add to track javascript events which would probably track adsense clicks, but it's 'not easy', ie: I might as well not pay for it. Cool idea though! I would pay for it if I could track adsense clicks without having to write code. Also, having an option to select multiple dom elements at once would be nice. In addition, logging into the web site and editing the html to add the code in the <head> tag would be nice too, especially since it's kind of a hassle for me to open eclipse and re-deploy to app engine at 1am when I am tired. haha, hope this helps.

dsiroker 16y ago
Certainly something we are well aware of and trying to improve! We'd love to automatically track ad clicks for you and provide the corresponding reporting to let you know which variation gets you the most clicks. Please email us at feedback@optimizely.com and let us know if you're interested in this and I'd love to learn more about this use case.
- photon_off 16y ago
  I think it'd be amazing if you could work smething out with the ppl at mouseflow or thier ilk. They could track the clicks and mouse movements for you. Maybe offer it as an add on for a test.

dav-id 16y ago

Why do all of these companies create cookie cutter names like 'Optimizely' is 'ly' the next .com? To me ly says 'We have no track record, use with caution' I would feel more comfortable using something with a name that isn't following some online fashion.

bretthellman 16y ago

Optimizely sounds like a great service but after entering my URL it took a long time to load. It would be great if you could speed that up... If I hadn't heard of the link from Hacker News, I would have closed the tab after a few seconds.

dsiroker 16y ago
Great feedback and certainly something we plan to improve. Thanks for bearing with us!