5 ms·

ABalytics.js: Client-side A/B Testing With Google Analytics

60 points by danmaz74 14y ago

ckluis 14y ago
Very Nice! Personally, I would have loved a code example inside the post. Thanks for going MIT.
- danmaz74 14y ago
  You're welcome! I didn't think about the code example in the post... maybe I'll add it and resubmit the post ;)
- pella 14y ago
  more information about usage: https://github.com/danmaz74/ABalytics https://github.com/danmaz74/ABalytics
  - ckluis 14y ago
    Oh, I opened that. I just meant my favorite - check out my cool tool posts include samples of cool tool being cool. :)
jentulman 14y ago
looks nice and this is just an initial thought, it looks like the holding content in the HTML is replaced by either case A or B from the script, will it run with the holding text being case A and swapping out only when B is picked by the script therefore avoiding a possible content flicker some of the time? of course this may not be desirable on larger content.
- danmaz74 14y ago
  Yes, it would work - but I wouldn't advice doing so: * If the flicker isn't noticeable, it doesn't matter either way * If the flicker is noticeable and disturbing, having it only for one version could skew the results in favor of the non-flickering version
btilly 14y ago
My first thought upon seeing this was, "Where are the statistics so that you know if you are just ahead by chance?" My second thought was, "Testing is so much better than not testing, this is still worthwhile." Seriously, if you aren't testing, do it. If you are, this is something to graduate out of.
- danmaz74 14y ago
  OP here: That was the spirit of the thing. It isn't a fully fledged solution, but it's a very easy way to start testing, and in GA you can combine this data with all the other data you have there, if you want.
- yahelc 14y ago
  I made a bookmarklet for easily calculating statistical significance, specifically for this use case. After enabling the bookmarklet, you click the numbers you need and then it sends it to an A/B testing calculator hosted by the folks who made ABBA Project: https://github.com/yahelc/ABBA-bookmarklet https://github.com/yahelc/ABBA-bookmarklet Sends to here: http://www.thumbtack.com/labs/abba/ http://www.thumbtack.com/labs/abba/
Timmy_C 14y ago
Checking out the source code, this isn't going to work in IE8 and earlier without some small tweaks. IE didn't support `document.getElementsByClassName(string)` until IE9. It would be a small rewrite to make it work with IDs instead of classes (or you could use a shim). [EDIT]: . . . But I like the idea. Might even use it next time I'm working with GA.
- danmaz74 14y ago
  Thanks for the warning! I initially developed this with jQuery, and only after seeing how little of it was needed decided to get rid of it - but I forgot to make all the due diligence. Later or this weekend I'll fix this - but if in the meantime anybody wants to send a fix or a pull request, I'll be glad to use it.
- danmaz74 14y ago
  LOL, I had forgotten: I already use a custom getElementsByClassName if there isn't one! So the it already works in IE < 9... check out the source code :)
theseanstewart 14y ago
Just me, or does anyone else miss Multivariate testing? It's impossible for me to use A/B testing with shopping cart software when it loads dynamic content (like the cart or checkout processes). Unless I'm doing it wrong?
- siddharthdeswal 14y ago
  Content that loads dynamically won't force you to use Multivariate Testing. In fact, most commercial services implement both A/B and Multivariate tests using client side JS. Also, you can A/B test dynamically loading content so I think you just need to find out how to do it using your chosen software.
ohashi 14y ago
Great job. I came up with my own ghetto solution using event tracking, yours is definitely better.
yahelc 14y ago
You can't reliably use Array#concat on the _gaq object. Once Google Analytics loads, it replaces the _gaq array with a custom object that has it's own push() method. That object doesn't have concat() on it. So, your core coding style will only work with very particular GA implementations; anything that deviates from that could very easily break or be subject to race conditions. You'll also get better results if you use events rather than custom variables; custom variable reporting isn't as flexible and can be very confusing (ie, visit start). It'll also mitigate the need to use concat() to cut the line, since Events can be fired at any time (whereas custom variables need to happen before the first hit of the session to be well-reported). I also really like the idea, though. I actually pass all my Optimizely test results as Google Analytics events with this snippet: if (window.optimmizely && optimizely.variationNamesMap) { var optmap = optimizely.variationNamesMap; for (var expt in optmap) { if (optmap.hasOwnProperty(expt)) { _gaq.push(["_trackEvent", "Optimizely", optimizely.allExperiments[expt].name, optmap[expt], 0, true]); } } }
- danmaz74 14y ago
  Concat is only called before GA is loaded, so I don't think any race condition could happen.
  - yahelc 14y ago
    Only if everyone loads it the same way. But there are lots of different ways to load GA; the only reason _gaq.push() isn't scary and doesn't raise the specter of race conditions is because of the 'spoofing' GA does. There are LOTS of different versions of the GA snippet floating around the internet; the assumptions this configuration makes would only work for some of them. For example, some people load GA like this: <script async src="//google-analytics.com/ga.js"> </script> <script> var _gaq = _gaq || []; ... </script> Folks who do this will break at minimum with older browsers that don't support async; not sure what it would do with async support, though. It seems unnecessarily risky.
    - deleted 14y ago
      [deleted]

60 points by danmaz74 14y ago

ckluis 14y ago

Very Nice! Personally, I would have loved a code example inside the post. Thanks for going MIT.

danmaz74 14y ago
You're welcome! I didn't think about the code example in the post... maybe I'll add it and resubmit the post ;)
pella 14y ago
more information about usage: https://github.com/danmaz74/ABalytics https://github.com/danmaz74/ABalytics
- ckluis 14y ago
  Oh, I opened that. I just meant my favorite - check out my cool tool posts include samples of cool tool being cool. :)

jentulman 14y ago

looks nice and this is just an initial thought, it looks like the holding content in the HTML is replaced by either case A or B from the script, will it run with the holding text being case A and swapping out only when B is picked by the script therefore avoiding a possible content flicker some of the time? of course this may not be desirable on larger content.

danmaz74 14y ago
Yes, it would work - but I wouldn't advice doing so: * If the flicker isn't noticeable, it doesn't matter either way * If the flicker is noticeable and disturbing, having it only for one version could skew the results in favor of the non-flickering version

btilly 14y ago

My first thought upon seeing this was, "Where are the statistics so that you know if you are just ahead by chance?" My second thought was, "Testing is so much better than not testing, this is still worthwhile." Seriously, if you aren't testing, do it. If you are, this is something to graduate out of.

danmaz74 14y ago
OP here: That was the spirit of the thing. It isn't a fully fledged solution, but it's a very easy way to start testing, and in GA you can combine this data with all the other data you have there, if you want.
yahelc 14y ago
I made a bookmarklet for easily calculating statistical significance, specifically for this use case. After enabling the bookmarklet, you click the numbers you need and then it sends it to an A/B testing calculator hosted by the folks who made ABBA Project: https://github.com/yahelc/ABBA-bookmarklet https://github.com/yahelc/ABBA-bookmarklet Sends to here: http://www.thumbtack.com/labs/abba/ http://www.thumbtack.com/labs/abba/

Timmy_C 14y ago

Checking out the source code, this isn't going to work in IE8 and earlier without some small tweaks. IE didn't support `document.getElementsByClassName(string)` until IE9. It would be a small rewrite to make it work with IDs instead of classes (or you could use a shim). [EDIT]: . . . But I like the idea. Might even use it next time I'm working with GA.

danmaz74 14y ago
Thanks for the warning! I initially developed this with jQuery, and only after seeing how little of it was needed decided to get rid of it - but I forgot to make all the due diligence. Later or this weekend I'll fix this - but if in the meantime anybody wants to send a fix or a pull request, I'll be glad to use it.
danmaz74 14y ago
LOL, I had forgotten: I already use a custom getElementsByClassName if there isn't one! So the it already works in IE < 9... check out the source code :)

theseanstewart 14y ago

Just me, or does anyone else miss Multivariate testing? It's impossible for me to use A/B testing with shopping cart software when it loads dynamic content (like the cart or checkout processes). Unless I'm doing it wrong?

siddharthdeswal 14y ago
Content that loads dynamically won't force you to use Multivariate Testing. In fact, most commercial services implement both A/B and Multivariate tests using client side JS. Also, you can A/B test dynamically loading content so I think you just need to find out how to do it using your chosen software.

ohashi 14y ago

Great job. I came up with my own ghetto solution using event tracking, yours is definitely better.

yahelc 14y ago

You can't reliably use Array#concat on the _gaq object. Once Google Analytics loads, it replaces the _gaq array with a custom object that has it's own push() method. That object doesn't have concat() on it. So, your core coding style will only work with very particular GA implementations; anything that deviates from that could very easily break or be subject to race conditions. You'll also get better results if you use events rather than custom variables; custom variable reporting isn't as flexible and can be very confusing (ie, visit start). It'll also mitigate the need to use concat() to cut the line, since Events can be fired at any time (whereas custom variables need to happen before the first hit of the session to be well-reported). I also really like the idea, though. I actually pass all my Optimizely test results as Google Analytics events with this snippet: if (window.optimmizely && optimizely.variationNamesMap) { var optmap = optimizely.variationNamesMap; for (var expt in optmap) { if (optmap.hasOwnProperty(expt)) { _gaq.push(["_trackEvent", "Optimizely", optimizely.allExperiments[expt].name, optmap[expt], 0, true]); } } }

danmaz74 14y ago
Concat is only called before GA is loaded, so I don't think any race condition could happen.
- yahelc 14y ago
  Only if everyone loads it the same way. But there are lots of different ways to load GA; the only reason _gaq.push() isn't scary and doesn't raise the specter of race conditions is because of the 'spoofing' GA does. There are LOTS of different versions of the GA snippet floating around the internet; the assumptions this configuration makes would only work for some of them. For example, some people load GA like this: <script async src="//google-analytics.com/ga.js"> </script> <script> var _gaq = _gaq || []; ... </script> Folks who do this will break at minimum with older browsers that don't support async; not sure what it would do with async support, though. It seems unnecessarily risky.
  - deleted 14y ago
    [deleted]