3 ms·

Ask HN: Please help with an AB test. (Live results in comments.)

4 points by notauser 17y ago

notauser 17y ago
My last day at my full time job was Tuesday, now I'm working on my own projects. It's a lot of fun so far (except for testing in Internet Explorer - this works in 8, haven't tried 7 yet). I wanted to get something up quickly to test my tool chain/environments and I thought that running a simple experiment was a great way to do that. I'm AB testing two versions of requesting e-mail addresses from visitors and I'm happy to share the results with HN. I'll post a more detailed (anonymous) breakdown later on but you can see the initial results here: http://theplanis.com/app/abtestresults/ http://theplanis.com/app/abtestresults/
- ErrantX 17y ago
  just as important are users who DIDNT respond :) are you collecting that information?
  - notauser 17y ago
    Good point, yes I am - It's not going into the live results but I'll put that data up too at an appropriate point. (For the interested the short link points to a tiny Django web app that's doing both the tracking and handling AJAX e-mail uploads. Everything else is client side JavaScript.)
- seldo 17y ago
  The first time I hit the page the request for an email address just didn't turn up at all; not sure if you're capturing that possibility. (FF 3.5.3, OSX 10.5.8)
jsm386 17y ago
a/b testing is a great tool, but wouldn't telling people they are participating in an a/b test trying to determine a specific question 'requesting e-mail addresses from visitors' bias the results?
- notauser 17y ago
  I'm logging referrers so I can see how much of an effect that has - if it turns out to be significant I'll segment the results.
  - jsm386 17y ago
    interesting..i await the results
    - notauser 17y ago
      After 20 hours: 310 unique visits / 22 addresses captured (7%) 50% of traffic from news.yc - and generally didn't leave an email address at all. Users from other sources were 10 times more likely to leave an addresses if shown the pop up on page load.

3 ms·

Ask HN: Please help with an AB test. (Live results in comments.)

4 points by notauser 17y ago

notauser 17y ago
My last day at my full time job was Tuesday, now I'm working on my own projects. It's a lot of fun so far (except for testing in Internet Explorer - this works in 8, haven't tried 7 yet). I wanted to get something up quickly to test my tool chain/environments and I thought that running a simple experiment was a great way to do that. I'm AB testing two versions of requesting e-mail addresses from visitors and I'm happy to share the results with HN. I'll post a more detailed (anonymous) breakdown later on but you can see the initial results here: http://theplanis.com/app/abtestresults/ http://theplanis.com/app/abtestresults/
- ErrantX 17y ago
  just as important are users who DIDNT respond :) are you collecting that information?
  - notauser 17y ago
    Good point, yes I am - It's not going into the live results but I'll put that data up too at an appropriate point. (For the interested the short link points to a tiny Django web app that's doing both the tracking and handling AJAX e-mail uploads. Everything else is client side JavaScript.)
- seldo 17y ago
  The first time I hit the page the request for an email address just didn't turn up at all; not sure if you're capturing that possibility. (FF 3.5.3, OSX 10.5.8)
jsm386 17y ago
a/b testing is a great tool, but wouldn't telling people they are participating in an a/b test trying to determine a specific question 'requesting e-mail addresses from visitors' bias the results?
- notauser 17y ago
  I'm logging referrers so I can see how much of an effect that has - if it turns out to be significant I'll segment the results.
  - jsm386 17y ago
    interesting..i await the results
    - notauser 17y ago
      After 20 hours: 310 unique visits / 22 addresses captured (7%) 50% of traffic from news.yc - and generally didn't leave an email address at all. Users from other sources were 10 times more likely to leave an addresses if shown the pop up on page load.