4 ms·

Show HN: A/Bingo for ASP.NET

32 points by jasonkester 16y ago

jasonkester 16y ago
I've been using this for the past 6 months for S3stat, Twiddla and FairTutor, with some pretty impressive results. Since it's "November Sprint" time, I finally motivated myself to package it up and release it as Open Source. It's essentially a simplified version of patio's A/Bingo, with one major departure in that participation data is stored in the users' browser rather than a local database. That helps it scale out better and means that you can pretty much just drop the code into your project and have it start working without having to configure anything. Let me know what you think!
- patio11 16y ago
  I think a prominent client of mine may owe you a beer. Thanks for bringing the light of A/B testing to platforms without great options. Nobody should have to use Google Website Optimizer.
  - jasonkester 16y ago
    Thanks for that, but I'm afraid that beer would get forwarded right back to you, since it was your comment that pushed me from "I should build one of those" to "I should build one of those and release it as open source": http://news.ycombinator.com/item?id=1124839 http://news.ycombinator.com/item?id=1124839
- rubyrescue 16y ago
  I guess this isn't totally off topic: http://www.fairtutor.com/about/how-it-works http://www.fairtutor.com/about/how-it-works is broken unless that's part of the a/b test :) Seriously, FairTutor is a great looking site, would love a post on how you built it, found the tutors, etc.
  - jasonkester 16y ago
    You caught me pushing a new build. Try again now. FairTutor will get its own "Rate My Startup" post, and probably a few "Making of" entries on the Expat Software blog when I get closer to a public launch. I've never been one to publicize my stuff when the front page is just collecting emails. Even though there's a full-blown website running beta classes back there, I'll wait until there's something you can actually see before asking people to take a look at it.
    - jasonkester 16y ago
      As for finding tutors, I went to the source. Last year I spent 9 months backpacking my way across Peru, Ecuador, Colombia and Panama. Along the way I spent nearly as many days talking with local Spanish language schools as I did surfing. Assuming any success whatsoever when the site launches, I'll try to follow up with a Central America tour next year. It's only the responsible thing to do, and besides, the surfing is better.
    - rubyrescue 16y ago
      Sites where you're building a marketplace of clients and providers are always a bit harder to bootstrap so i find it interesting; also i live in South America and thought about building a product like this.
- euroclydon 16y ago
  [EDIT] Just saw your reply to forcer's question. My page was stale [/EDIT] That storing the participation data in the user's browser is a bit of a red herring for me. Would you please describe the tradeoffs between doing it that way, and doing it Patrick's way? Also, there's no license. I potentially would like to contribute, but that's a show stopper. Any chance of giving it an MIT lic? Feel free to shoot me an email at joshpearce[at]beechtreetech[dot]com
  - jasonkester 16y ago
    I don't do open source licenses, since they just strike me as pretentious. The code is out there on the internet, and clearly labeled as free to use. I don't particularly care if you don't credit me, use it in a paid app, or whatever. It's in the public domain, so go nuts with it. If you make changes, let me know and I'll see about rolling them back in.
jeroen 16y ago
http://www.fairtutor.com/fairlycertain/fairlycertain-details http://www.fairtutor.com/fairlycertain/fairlycertain-details says "Using the &ltab:Test> UserControl"
- jasonkester 16y ago
  Are you pointing out a typo on the site (if so, I can't find it), or are you lamenting that it requires UserControls to work? If it's the latter, read down to the next code sample, which shows how to call the library directly from an ASP.NET MVC View or anywhere else. The UserControl is just there as a convenience.
  - pmjordan 16y ago
    There is a typo: you're missing the semicolon at the end of the < entity. Some browsers are permissive enough to allow that, some aren't. You must be using one that is.
    - jasonkester 16y ago
      Thanks for that. Fixed.
  - Huppie 16y ago
    Check the source: "<h3>Using the &ltab:Test> UserControl</h3>" It probably should be: "<h3>Using the <ab:Test> UserControl</h3>"
  - jeroen 16y ago
    Sorry, just the typo.
thinkzig 16y ago
You rock. This weekend I was planning to start evaluating A/B testing frameworks for my ASP.NET-based travel site. I was really hoping I could find something like Patrick's framework and here it just dropped in my lap. Thanks for releasing this. I'll let you know how it goes. HN delivers again!
forcer 16y ago
Great job Jason. I was looking for something like this for a long time. I have few questions which are not explained much on your site: - you say it does not use database but stores result on the client? how does it work? can I loose the results when IIS is restarted etc. ? - any screenshot of your dashboard? I know its probably not your main business so you dont want to waste lots of time on it but I think it is a great idea and you should make a business out of it. We currently pay $150 a month for VWO and its a fantastic tool but it limits us to test on certain websites which have lots of traffic as VWO and any other tool on the market charges per number of visitors tested. Your tool as its self-hosted solution wouldn't have these economics so you can charge few hundred dollars for a license with unlimited tests etc.
- jasonkester 16y ago
  It uses cookies to store participation and conversion data, so as a visitor you'll come away with a cookie that looks like this: ab:ID=1212802769|Tests=header_caption,register_payment_message,account_instructions,setup_navigation|Conversions=header_caption That way, we can simply ask the visitor whether he's participating in a given test, and we can know not to score him twice for anything. All that lives on the server is a file at the root called tests.ab holding a few k worth of participation and conversion tallies. Every time anything changes, it checks to see if it's saved its state in the last minute, and if not it simply serializes itself to XML and dumps that to disk. If it ever finds itself uninitialized (such as on a server reset), it attempts to load its last saved state. So long as you remember to ignore that tests.ab file as part of your deploy, it'll survive intact. As to selling it, I'm all about bringing in the cash, but a hosted option wouldn't really work for this. First, it's architected specifically so there's hardly anything to host. Second, its big advantage over something like Visual Website Optimizer is that your pages don't hang while requesting resources from a 3rd party. Don't worry. I've got other stuff bringing in revenue (like the site this is hosted on, for instance), but thanks for the suggestion!
  - forcer 16y ago
    Thanks for the explanation - now I know it serializes to a file which is fine - I wasn't sure how it can keep the state only by storing on the user side. Self-hosted solution - I probably didn't explain myself well I meant that its great it can be self-hosted rather than relying on 3rd party website like VWO.
spoiledtechie 16y ago
Can someone please describe what AB testing is or point me to something that explains it?
- jasonkester 16y ago
  Try a few of the links here: http://www.bingocardcreator.com/abingo/resources http://www.bingocardcreator.com/abingo/resources Basically, it's testing one version of your website against another version, so that you can measure which one better compels users to do something. So, for example, you might try testing your normal "Buy Now" button against a double-sized, bright red shiny button. You'd show one version to half your visitors, the other version to the other half. Libraries like this one let you keep track of who saw which version, and whether they actually bought your thing. After a week or so collecting data, you can see that, for example, 4.8% of visitors seeing your old button clicked it, whereas 6.6% of those who saw the big red one clicked it. That's valuable. As in, measurable in dollars valuable, so you need to be doing it if you run a website that sells things to people or otherwise gets value out of people doing stuff on it.
  - spoiledtechie 16y ago
    thanks. That helps.
weaksauce 16y ago
Is there an ab testing tool worth looking at on the django side of the fence?

32 points by jasonkester 16y ago

I've been using this for the past 6 months for S3stat, Twiddla and FairTutor, with some pretty impressive results. Since it's "November Sprint" time, I finally motivated myself to package it up and release it as Open Source. It's essentially a simplified version of patio's A/Bingo, with one major departure in that participation data is stored in the users' browser rather than a local database. That helps it scale out better and means that you can pretty much just drop the code into your project and have it start working without having to configure anything. Let me know what you think!

patio11 16y ago
I think a prominent client of mine may owe you a beer. Thanks for bringing the light of A/B testing to platforms without great options. Nobody should have to use Google Website Optimizer.
- jasonkester 16y ago
  Thanks for that, but I'm afraid that beer would get forwarded right back to you, since it was your comment that pushed me from "I should build one of those" to "I should build one of those and release it as open source": http://news.ycombinator.com/item?id=1124839 http://news.ycombinator.com/item?id=1124839
rubyrescue 16y ago
I guess this isn't totally off topic: http://www.fairtutor.com/about/how-it-works http://www.fairtutor.com/about/how-it-works is broken unless that's part of the a/b test :) Seriously, FairTutor is a great looking site, would love a post on how you built it, found the tutors, etc.
- jasonkester 16y ago
  You caught me pushing a new build. Try again now. FairTutor will get its own "Rate My Startup" post, and probably a few "Making of" entries on the Expat Software blog when I get closer to a public launch. I've never been one to publicize my stuff when the front page is just collecting emails. Even though there's a full-blown website running beta classes back there, I'll wait until there's something you can actually see before asking people to take a look at it.
  - jasonkester 16y ago
    As for finding tutors, I went to the source. Last year I spent 9 months backpacking my way across Peru, Ecuador, Colombia and Panama. Along the way I spent nearly as many days talking with local Spanish language schools as I did surfing. Assuming any success whatsoever when the site launches, I'll try to follow up with a Central America tour next year. It's only the responsible thing to do, and besides, the surfing is better.
  - rubyrescue 16y ago
    Sites where you're building a marketplace of clients and providers are always a bit harder to bootstrap so i find it interesting; also i live in South America and thought about building a product like this.
euroclydon 16y ago
[EDIT] Just saw your reply to forcer's question. My page was stale [/EDIT] That storing the participation data in the user's browser is a bit of a red herring for me. Would you please describe the tradeoffs between doing it that way, and doing it Patrick's way? Also, there's no license. I potentially would like to contribute, but that's a show stopper. Any chance of giving it an MIT lic? Feel free to shoot me an email at joshpearce[at]beechtreetech[dot]com
- jasonkester 16y ago
  I don't do open source licenses, since they just strike me as pretentious. The code is out there on the internet, and clearly labeled as free to use. I don't particularly care if you don't credit me, use it in a paid app, or whatever. It's in the public domain, so go nuts with it. If you make changes, let me know and I'll see about rolling them back in.

jeroen 16y ago

http://www.fairtutor.com/fairlycertain/fairlycertain-details http://www.fairtutor.com/fairlycertain/fairlycertain-details says "Using the &ltab:Test> UserControl"

jasonkester 16y ago
Are you pointing out a typo on the site (if so, I can't find it), or are you lamenting that it requires UserControls to work? If it's the latter, read down to the next code sample, which shows how to call the library directly from an ASP.NET MVC View or anywhere else. The UserControl is just there as a convenience.
- pmjordan 16y ago
  There is a typo: you're missing the semicolon at the end of the < entity. Some browsers are permissive enough to allow that, some aren't. You must be using one that is.
  - jasonkester 16y ago
    Thanks for that. Fixed.
- Huppie 16y ago
  Check the source: "<h3>Using the &ltab:Test> UserControl</h3>" It probably should be: "<h3>Using the <ab:Test> UserControl</h3>"
- jeroen 16y ago
  Sorry, just the typo.

thinkzig 16y ago

You rock. This weekend I was planning to start evaluating A/B testing frameworks for my ASP.NET-based travel site. I was really hoping I could find something like Patrick's framework and here it just dropped in my lap. Thanks for releasing this. I'll let you know how it goes. HN delivers again!

forcer 16y ago

Great job Jason. I was looking for something like this for a long time. I have few questions which are not explained much on your site: - you say it does not use database but stores result on the client? how does it work? can I loose the results when IIS is restarted etc. ? - any screenshot of your dashboard? I know its probably not your main business so you dont want to waste lots of time on it but I think it is a great idea and you should make a business out of it. We currently pay $150 a month for VWO and its a fantastic tool but it limits us to test on certain websites which have lots of traffic as VWO and any other tool on the market charges per number of visitors tested. Your tool as its self-hosted solution wouldn't have these economics so you can charge few hundred dollars for a license with unlimited tests etc.

jasonkester 16y ago
It uses cookies to store participation and conversion data, so as a visitor you'll come away with a cookie that looks like this: ab:ID=1212802769|Tests=header_caption,register_payment_message,account_instructions,setup_navigation|Conversions=header_caption That way, we can simply ask the visitor whether he's participating in a given test, and we can know not to score him twice for anything. All that lives on the server is a file at the root called tests.ab holding a few k worth of participation and conversion tallies. Every time anything changes, it checks to see if it's saved its state in the last minute, and if not it simply serializes itself to XML and dumps that to disk. If it ever finds itself uninitialized (such as on a server reset), it attempts to load its last saved state. So long as you remember to ignore that tests.ab file as part of your deploy, it'll survive intact. As to selling it, I'm all about bringing in the cash, but a hosted option wouldn't really work for this. First, it's architected specifically so there's hardly anything to host. Second, its big advantage over something like Visual Website Optimizer is that your pages don't hang while requesting resources from a 3rd party. Don't worry. I've got other stuff bringing in revenue (like the site this is hosted on, for instance), but thanks for the suggestion!
- forcer 16y ago
  Thanks for the explanation - now I know it serializes to a file which is fine - I wasn't sure how it can keep the state only by storing on the user side. Self-hosted solution - I probably didn't explain myself well I meant that its great it can be self-hosted rather than relying on 3rd party website like VWO.

spoiledtechie 16y ago

Can someone please describe what AB testing is or point me to something that explains it?

jasonkester 16y ago
Try a few of the links here: http://www.bingocardcreator.com/abingo/resources http://www.bingocardcreator.com/abingo/resources Basically, it's testing one version of your website against another version, so that you can measure which one better compels users to do something. So, for example, you might try testing your normal "Buy Now" button against a double-sized, bright red shiny button. You'd show one version to half your visitors, the other version to the other half. Libraries like this one let you keep track of who saw which version, and whether they actually bought your thing. After a week or so collecting data, you can see that, for example, 4.8% of visitors seeing your old button clicked it, whereas 6.6% of those who saw the big red one clicked it. That's valuable. As in, measurable in dollars valuable, so you need to be doing it if you run a website that sells things to people or otherwise gets value out of people doing stuff on it.
- spoiledtechie 16y ago
  thanks. That helps.

weaksauce 16y ago

Is there an ab testing tool worth looking at on the django side of the fence?