5 ms·

A/Bingo 1.0.0 Official Release

21 points by stakent 17y ago

patio11 17y ago
I didn't think this would necessarily be HN material so I didn't bother submitting it, but apparently someone disagrees with me. I figured I had to "professionalize" A/Bingo a bit because recently from reading my email I get the impression that it is looking less like the 10% time project from my 10% time project and more like "Oh, effity, if this breaks businesses with employees start losing money." So I did a bit of code cleanup, wrote some tests, and created a versioning plan so that folks depending on it are not totally on their own for evaluating whether a particular commit will break stuff if they pull it in. Since I had the opportunity I also made some changes to the user experience, such as decreasing the amount of code you have to write even further, and incorporating the Minimum Viable Dashboard. If you aren't A/B testing yet, I want to change that. Tell me what I need to do to make it happen.
- stakent 17y ago
  I'm this "someone". Why? Because I believe your "If you aren't A/B testing yet, I want to change that." is beneficial for significant part of HN'ers.
- dlib 17y ago
  I'm going to have a look at this in a few weeks. I only knew about Vanity so I'm glad I came across this post. Does A/Bingo also auto-select an alternative if there is a 95% reliability that it's better?
  - patio11 17y ago
    No, it does not. Vanity and A/Bingo have some fundamentally different takes on the problem. This is totally fine -- there are a lot of good ways to do A/B testing just like there are a lot of good ways to parse an HTML document. Use whichever fits your needs. I'm happy to answer specific implementation questions, but the bumper sticker version is "In general, Vanity wants to be your one-stop measurement shop. A/Bingo wants to make A/B testing so easy you do it automatically, every single time."
    - michael_dorfman 17y ago
      How about "A/Bingo vs Google Website Optimizer"? What's the bumper sticker on that comparison?
      - patio11 17y ago
        A/Bingo: Written by programmers, for programmers. GWO: written by an advertising company, to be used by non-technical stakeholders in marketing. GWO makes significant compromises on usability -- both for the site's programmers and the site's users. Previously I recommended it only very hesitatingly, to non-technical marketer types. These days Visual Website Optimizer exists and it is embarrassingly better in every possible way for that audience, including the WYSIWYG interface. You can read my thoughts on GWO at length in the A/Bingo documentation. http://www.bingocardcreator.com/abingo/compare http://www.bingocardcreator.com/abingo/compare
        paraschopra 17y ago
        Thanks for recommending http://visualwebsiteoptimizer.com/ http://visualwebsiteoptimizer.com/ It exists to make A/B and Multivariate testing dead-simple for non-technical types :)
        dminor 17y ago
        Most of your technical arguments against GWO are obviated by using their multivariate version.
- ehsanul 17y ago
  Would this work in a non-Rails project? Say Sinatra?
  - patio11 17y ago
    As written there is an ActiveRecord dependency, and it depends on code implementing the general contract of ActiveSupport::Cache::Store for proper operation. However, if you've got a day or two to kill, you could probably fix that. After that you'll have to rebuild the dashboard. I'm not sure how Sinatra handles the view layer typically but you may need to redo the syntax sugar for it, as I think my implementation probably depends on ActiveView magic.
euroclydon 17y ago
Would anyone familiar with ASP.NET MVC & ROR care to tell me how portable this would be to ASP.NET MVC?
- patio11 17y ago
  ASP.NET is not my bag, baby, but on the gratuitously painful enterprise-friendly development platform that has put a roof over my head for the last few years (J2EE) I think you could get the moral equivalent of this done in under a month. If you want to write yourself ASP.NET A/B testing stuff, send me an email and I'll tell you what I know about the design tradeoffs.
jasonkester 17y ago
Any chance of offering a download that doesn't require me to install GIT on my computer?
- mattyb 17y ago
  If you use Mercurial, you can use hg-git to talk to git repos without Git. http://hg-git.github.com/ http://hg-git.github.com/ I'm not sure if there's an equivalent for Subversion.
- patio11 17y ago
  Watch out, first you start using SVN over git and then people will assume you don't own a MacBook and then your chances of Rails development jobs are shot :) http://www.bingocardcreator.com/files/abingo.zip http://www.bingocardcreator.com/files/abingo.zip At the moment that is created by hand and so it won't be automatically updated. I'll figure out a more permanent solution later.
  - jasonkester 17y ago
    Thanks for that. Given your writing about the pain of dealing with open source, I would have expected you'd have that autogenerating from the start. For what it's worth, the reason I'm not running git on my Linux box is that I'm a C# guy by choice. ABingo seems like something the .NET world needs, so I want to look into porting it.
    - patio11 17y ago
      I made it available through git because that is the de-facto standard distribution method for Rails coders these days, who are the audience I needed to reach with it. If it were consumer-facing, oh yeah, I would have done something much easier from the getgo. I concur that .NET desperately needs an A/B testing framework. Stack Overflow should have one. Give me their worst .NET engineer and two months -- one for writing A/Boverflow and one for writing tests -- I could increase almost any metric on that site by 10%. What is your worst engineer scheduled to do this month that is more important than that? That's a question I'd like to ask anybody who isn't A/B testing yet. Seriously. What keeps you so busy that you can't adopt a low-risk, battle-tested, low-investment strategy with stupendously scaling returns like A/B testing? (I can only hope that every company with a web page pulled a Google, built one of these internally, and keeps vewwwwy vewwwwy quiet about it for fear of letting people know how easy it is.)
paraschopra 17y ago
Patrick, as you said my tool (VWO) is for non-techies (and I know that) but some of the users have been demanding an API of sorts for A/B and multivariate testing. I have been perplexed on how a language agnostic API for A/B testing can possibly exist. First of all, to make it language agnostic it would probably need to work on HTTP REST but then how would it be different that client side JavaScript based A/B testing. Any thoughts on language independent API design for A/B testing?
recurser 17y ago
Nice writeup, looks great. btw a tiny typo: "I intend to migration my own deployment to Redis"

21 points by stakent 17y ago

I didn't think this would necessarily be HN material so I didn't bother submitting it, but apparently someone disagrees with me. I figured I had to "professionalize" A/Bingo a bit because recently from reading my email I get the impression that it is looking less like the 10% time project from my 10% time project and more like "Oh, effity, if this breaks businesses with employees start losing money." So I did a bit of code cleanup, wrote some tests, and created a versioning plan so that folks depending on it are not totally on their own for evaluating whether a particular commit will break stuff if they pull it in. Since I had the opportunity I also made some changes to the user experience, such as decreasing the amount of code you have to write even further, and incorporating the Minimum Viable Dashboard. If you aren't A/B testing yet, I want to change that. Tell me what I need to do to make it happen.

stakent 17y ago
I'm this "someone". Why? Because I believe your "If you aren't A/B testing yet, I want to change that." is beneficial for significant part of HN'ers.
dlib 17y ago
I'm going to have a look at this in a few weeks. I only knew about Vanity so I'm glad I came across this post. Does A/Bingo also auto-select an alternative if there is a 95% reliability that it's better?
- patio11 17y ago
  No, it does not. Vanity and A/Bingo have some fundamentally different takes on the problem. This is totally fine -- there are a lot of good ways to do A/B testing just like there are a lot of good ways to parse an HTML document. Use whichever fits your needs. I'm happy to answer specific implementation questions, but the bumper sticker version is "In general, Vanity wants to be your one-stop measurement shop. A/Bingo wants to make A/B testing so easy you do it automatically, every single time."
  - michael_dorfman 17y ago
    How about "A/Bingo vs Google Website Optimizer"? What's the bumper sticker on that comparison?
    - patio11 17y ago
      A/Bingo: Written by programmers, for programmers. GWO: written by an advertising company, to be used by non-technical stakeholders in marketing. GWO makes significant compromises on usability -- both for the site's programmers and the site's users. Previously I recommended it only very hesitatingly, to non-technical marketer types. These days Visual Website Optimizer exists and it is embarrassingly better in every possible way for that audience, including the WYSIWYG interface. You can read my thoughts on GWO at length in the A/Bingo documentation. http://www.bingocardcreator.com/abingo/compare http://www.bingocardcreator.com/abingo/compare
      - paraschopra 17y ago
        Thanks for recommending http://visualwebsiteoptimizer.com/ http://visualwebsiteoptimizer.com/ It exists to make A/B and Multivariate testing dead-simple for non-technical types :)
      - dminor 17y ago
        Most of your technical arguments against GWO are obviated by using their multivariate version.
ehsanul 17y ago
Would this work in a non-Rails project? Say Sinatra?
- patio11 17y ago
  As written there is an ActiveRecord dependency, and it depends on code implementing the general contract of ActiveSupport::Cache::Store for proper operation. However, if you've got a day or two to kill, you could probably fix that. After that you'll have to rebuild the dashboard. I'm not sure how Sinatra handles the view layer typically but you may need to redo the syntax sugar for it, as I think my implementation probably depends on ActiveView magic.

euroclydon 17y ago

Would anyone familiar with ASP.NET MVC & ROR care to tell me how portable this would be to ASP.NET MVC?

patio11 17y ago
ASP.NET is not my bag, baby, but on the gratuitously painful enterprise-friendly development platform that has put a roof over my head for the last few years (J2EE) I think you could get the moral equivalent of this done in under a month. If you want to write yourself ASP.NET A/B testing stuff, send me an email and I'll tell you what I know about the design tradeoffs.

jasonkester 17y ago

Any chance of offering a download that doesn't require me to install GIT on my computer?

mattyb 17y ago
If you use Mercurial, you can use hg-git to talk to git repos without Git. http://hg-git.github.com/ http://hg-git.github.com/ I'm not sure if there's an equivalent for Subversion.
patio11 17y ago
Watch out, first you start using SVN over git and then people will assume you don't own a MacBook and then your chances of Rails development jobs are shot :) http://www.bingocardcreator.com/files/abingo.zip http://www.bingocardcreator.com/files/abingo.zip At the moment that is created by hand and so it won't be automatically updated. I'll figure out a more permanent solution later.
- jasonkester 17y ago
  Thanks for that. Given your writing about the pain of dealing with open source, I would have expected you'd have that autogenerating from the start. For what it's worth, the reason I'm not running git on my Linux box is that I'm a C# guy by choice. ABingo seems like something the .NET world needs, so I want to look into porting it.
  - patio11 17y ago
    I made it available through git because that is the de-facto standard distribution method for Rails coders these days, who are the audience I needed to reach with it. If it were consumer-facing, oh yeah, I would have done something much easier from the getgo. I concur that .NET desperately needs an A/B testing framework. Stack Overflow should have one. Give me their worst .NET engineer and two months -- one for writing A/Boverflow and one for writing tests -- I could increase almost any metric on that site by 10%. What is your worst engineer scheduled to do this month that is more important than that? That's a question I'd like to ask anybody who isn't A/B testing yet. Seriously. What keeps you so busy that you can't adopt a low-risk, battle-tested, low-investment strategy with stupendously scaling returns like A/B testing? (I can only hope that every company with a web page pulled a Google, built one of these internally, and keeps vewwwwy vewwwwy quiet about it for fear of letting people know how easy it is.)

paraschopra 17y ago

Patrick, as you said my tool (VWO) is for non-techies (and I know that) but some of the users have been demanding an API of sorts for A/B and multivariate testing. I have been perplexed on how a language agnostic API for A/B testing can possibly exist. First of all, to make it language agnostic it would probably need to work on HTTP REST but then how would it be different that client side JavaScript based A/B testing. Any thoughts on language independent API design for A/B testing?

recurser 17y ago

Nice writeup, looks great. btw a tiny typo: "I intend to migration my own deployment to Redis"