7 ms·
The Most Radical A/B Test I've Ever Done
- travem 16y agoMy first thoughts are that removing the downloadable version will work out well by getting rid of the "Tyranny of Choice" and thus helping with conversions. Has anyone else tried this approach?
- michael_dorfman 16y agoVery interesting. My initial thought (unencumbered by any actual knowledge) is that the folks who download the client version (instead of sign up for the web version) don't do so for any particular reason, so I don't think that removing the option should actually hurt. Sounds like a good fit for Bayesian probability, using the old numbers as priors.
- JoeAltmaier 16y agoDownloaded app can be used offline.
- michael_dorfman 16y agoBut the question is: is that really important to the majority of the users who download it? Or are they acting blindly? Now, it could be that there is a contingent of bingo-loving teachers who absolutely prefer to have the software locally installed, but I'm suspecting (without any good evidence, I grant) that this is not really a key decision-factor for most of Patrick's customers who choose to download.
- patio11 16y agoI believed for years that there was a group of customers who would pay for downloadable versions when they wouldn't pay for web versions. In fact, I put AdWords ads on many free web-based competitors, and those are my best performing ads anywhere. But I've come to believe, after listening to customers and seeing the behavior in tests where I emphasize or deemphasize the options, that this doesn't say that users want to use downloadable applications. It was just that users want easy to use applications that don't suck. Next to the free web apps in 2006, BCC downloadable was easy to use and didn't suck. Next to BCC downloadable, BCC online is easy to use and doesn't suck.
- bmelton 16y agoI'm not paying to use the web version, but if it's written in HTML5 it can leverage the cache-manifest (or Gears, either way) and work offline as well. Do you know one way or another that it isn't?
- patio11 16y agoUnless HTML5 has some previously undisclosed way of running ImageMagick and Prawn, I'm going to go out on a limb and say that this will not work offline that well.
- paraschopra 16y agoAnd I thought the post _had_ the results of this test :) Anyway, interesting test and do keep us posted about the results.
- fierarul 16y agoIsn't it odd to announce the A/B test when you've just started it ? I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.
- paraschopra 16y agoSomewhat agree with you. I recently blogged about why you shouldn't publicize that you are doing an A/B test http://visualwebsiteoptimizer.com/split-testing-blog/3-controversial-ab-testing-tips-that-you-may-not-agree-with/ http://visualwebsiteoptimizer.com/split-testing-blog/3-contr... Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.
- JoachimSchipper 16y agoIn what way would this influence things? Sure, it might draw some additional uninterested visitors, but that will just depress conversion rates all around, not exclusively for A or B. The results remain valid.
- paraschopra 16y agoYes but that changes the real difference between conversion rates of A and B. Even though trend won't get affected but actual numbers will. Not that serious but given that this problem is easily avoidable one may want to not announce the test.
- fierarul 16y agoAssuming no actual possible customer reads this blog, this doesn't influences the number of new customers, but it does changes the overall numbers. But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer. Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.
- Sukotto 16y ago
- px 16y agoAs a teacher, I think this is a smart decision. Desktop applications are increasingly difficult to manage on a school computer. Many teachers return each September to a computer that has been wiped clean. At the same time, the internet is becoming more and more accessible in schools. Teachers will be able to use the web application no matter which room they are in or which computers they have access to. Another challenge for non-tech savvy teachers is syncing work between the school and home. I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.
- maukdaddy 16y agoAs someone who works in information security at a very large district, I will say this is a brilliant decision! =) The last thing I need is school personnel downloading Java apps. Not that yours is malicious - but once someone learns how to install/run something, it becomes exponentially more difficult to teach them to identify what is malicious and what is not. +1 for syncing between work/school. Often the computers at schools are inferior to ones that teachers have a home, or at least a different platform.
- patio11 16y agoMany teachers return each September to a computer that has been wiped clean. I hear that. Frequently, in September, from several dozen teachers asking me "How do I reinstall Bingo Card Creator on my computer which was just wiped clean? Also, do I have to pay?" I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area. It totally solved that problem, where the feature I made in the downloadable version to do the same thing has been a catastrophic flop. (Twenty people have ever used it, and I still get emails asking to explain how to copy files onto a floppy so that they can be read on the machine at school. And then emails the next day saying that the computer doesn't know what a .bcf file is. And then emails the next day saying that they installed Bingo Card Creator and opened the file but it says it is the free trial which is odd because they know they purchased the program and they never had this issue at home [i.e. where they followed directions and put in their Registration Key].)
- 16y ago
- markkanof 16y ago"How do I copy the files to my niece’s computer? By the way it is a Mac and I use a Yahoo. Try using the web application." This gem from the article really illustrates the added difficulty of supporting a desktop application vs a web application. For this particular request what OS she is running doesn't matter, but for many other requests it could be very important to know. Given the technical sophistication of this user, just figuring out what environment the application is running in could be an arduous process.
- fdb 16y agoI love this article. And his article on why he's done making desktop software is even better: http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-apps/ http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-a...
- coolio73 16y agoEnough of this uninteresting Bingo Card Creating program. How do these posts keep getting pushed up to the first page?
- Grinnmarr 16y agoI just wish I had enough rep to down vote your comment :P His posts and complete transparency about growing his business epitomize the type of community contribution that brings me back to this site time and time again.
- coolio73 16y agoprinting bingo cards is a business any 12 year old with a computer could set up. Couldn't you do this on Print Shop and an Apple IIe? Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day. Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable. He should spend some of his blogging energy on developing a more interesting topic/business to blog about.
- Poiesis 16y agoI will feed the troll: He should spend some of his blogging energy on developing a more interesting topic/business to blog about. Patio11 is not your dancing monkey. Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable. Laughing all the way to the bank. Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day. Well, can't say we'd miss you if you stopped contributing, but I'd have to take issue with the first part. At this moment, the front page has 30 entries of course. I can't believe I'm doing this, but they are about Flash, SproutRobot, Falcon 9, bcvi, HTML5, ideas, diamonds, deflate, BigPipe, Falcon9 (again), Titan, YourWorldOfText, fivethirtyeight.com, being wrong, Toxoplasma, Soros, Readability, Google IO, Posterous, Steve Jobs, Wakemate, Amelie(), the FTC, patio11's a/b test, chrome dev tools, twitter employees leaving, Apple and HTML5, Bing cashback, a world cup calendar and John Gruber. I'll leave it to you to determine if "most tech stuff on HN is the cutting edge of web apps." I tend to think that most business stuff on HN is there to help educate someone to run their business, generally technical in nature. Finally, there's printing bingo cards is a business any 12 year old with a computer could set up. Why haven't you, then?
- tomerico 16y agoI've just searched google for: Bingo cards Generator Printable bingo cards Create bingo cards Bingo Creator In all of them you are not one of the top 3 results. It might suggest that a better approach would be to work on your SEO than to increase your ad money.