6 ms·
Happy to answer questions, as always. By the way, roughly 3/4 of A/B tests I participate in (my own and for clients) fail to improve business results. This co
by patio11 15y ago
Happy to answer questions, as always.
By the way, roughly 3/4 of A/B tests I participate in (my own and for clients) fail to improve business results. This compares with approximately 2% of A/B tests I've ever seen with in-depth blog posts, so I tried to redress the balance here.
- maukdaddy 15y agoHow do you break that news to clients after they've paid $$$ for your services? Do you heavily caveat the potential before the engagement?
- wildwood 15y agoOne potential upside is that, as far as the numbers are concerned, the client is free to use whichever version they want.
- nialo 15y agoYou just run more tests, such that 25% of tests run still turns out to make $$$$ at least.
- wildwood 15y agoWell, keep in mind that, with a significance threshold of 5%, one in twenty null-result tests will appear to have significance. So just continuing to test until you get a result is not necessarily going to net you anything.
- patio11 15y agoThat's surprisingly less of a problem than I naively thought it would be. I'm upfront with clients prior to, during, and after engagements, and I try to set expectations accurately. That said, for the overwhelming majority of my clients, if e.g. the entire budget for my wedding was spent on a project whose final conclusion was "Experimental results: we now know one more thing that didn't work" that would not be a catastrophic loss. Businesses routinely spend $$$, $$$$$, and $$$$$$$$$ on things that don't end up working out the way they were planned. They're pretty much OK with that, since they run "portfolio strategies" in terms of directions. Nobody is betting the company on a two-week engagement with me, any more than they bet the company on any particular man-month of engineering time. n.b. This is partially influenced by being picky about clients and working with people who are clueful and have budget. This is one reason why if a startup with $X0,000 in the bank comes to me and asks to roll the dice on an engagement I give them a link to my blog, some suggestions, and invite them to come back when we're a better positioned to help each other out.
- adrianhoward 15y agoI suspect that people hire you not to do "a successful experiment" - but so that they can learn how to experiment in productive ways. It's a skill that seems sadly lacking with many of the folk that I talk to :-)
- gvb 15y agoThe glass half empty is that there was no benefit to the redesign. The glass half full is that you have a more visually pleasing, refreshed web site that is at least as effective as the old design. Patrick also notes that the users tend to associate visual changes with programatic improvements, even when there are no programatic improvements (placebo effect).
- runevault 15y agoMaybe you covered this before and I forgot, but how did you arrive at the number you use for the max cards creatable with a free account? I get the 'more than the # of kids someone would have in a family' but 15 seems... high.
- citizens 15y agoYou're on the right track, but it's more than the number of kids in a classroom — not family. He mentioned it in a podcast somewhere.
- tehayj 15y agoYou sell the bingo card creator 5000 times a month? Never expected that bingo is such a big deal. Congratz! :)
- patio11 15y agoNot sure where you're getting that number from but it is not accurate. http://www.bingocardcreator.com/stats/sales-by-month http://www.bingocardcreator.com/stats/sales-by-month Units on the graph are USD.
- dgant 15y agoYou mentioned that sales were strongly up for reasons unrelated to the post. What are those reasons?
- patio11 15y agoShort version: traffic way the heck up, probably through a combination of Google algorithm changes, just natural growth of the Internet among people who matter to my business, and organic growth.
- sedev 15y agoYou recommend Stripe in a throwaway line in the post - it sounds like your experience with them has been really positive! I'm working on a project where I may end up taking people's money for services on a subscription basis. I've heard that Stripe is better for one-time charges than for re-occuring charges. Is that true in your experience, as someone who's done work with Stripe?
- patio11 15y agoStripe is a dream come true for developers. I use Spreedly (backed by Paypal) on Appointment Reminder to do subscription management, and am very happy with it, too. I hear you can use them together now, which sounds like the PB&J of money if you ask me. P.S. I also don't hate me love Paypal, too.
- sedev 15y agoI'd never heard of Spreedly! Thank you for pointing that out: it makes this project just the tiniest bit more likely to replace my day job. :)
- jcampbell1 15y agoDoes Paypal still have the restriction that if you use them for credit card processing, you must also offer a Paypal checkout? Offering Appointment Reminder though standard PayPal seems like a giant additional headache.
- tomjen3 15y agoAs a user I love paypal because it is really, really easy to end a subscription and there is nothing, whatsoever, the merchant can do about it.
- ktsmith 15y agoSome thing that you might want to look at are the number of free cards you are giving away. With many states having mandatory class size reduction class sizes are smaller and 15 would cover a lot of classes. My wife's class this year has been as small as 12 students and as large as 18 students. With 15 free cards she would just make the other three by hand. A lot of that is due to us already spending a large amount of money on her class due to budget deficiencies and salary freezes for all teachers for three years now. Another issue you might be encountering is that states are adopting the common core standards http://www.corestandards.org/ http://www.corestandards.org/ and word bingo doesn't meet any of the standards for english language arts. Since everything in classrooms is documented, lesson plans are stored and in some cases handed over to the state there's not going to be a lot of use cases for activities that can't be applied to the standards. Even if those activities are just for fun due to a holiday.
- patio11 15y agoI've experimented with 12 vs. 15 before. Null result. I might try pushing it down to 8 sometime. But, realistically speaking, only if it strikes my fancy some morning.
- bhousel 15y agoOk, I freely admit that I know nothing about selling bingo cards.. But it "feels" to me like that number should be closer to 5. I was really surprised to see that you give away 15 cards for free. You could A/B test this too, right?
- joshwa 15y agoAnother idea: start high and ratchet down with each subsequent word list?
- paraschopra 15y agoYep, most A/B tests won't give results. But the ones that do are worth it (provided they were quick and easy to setup). There's always an opportunity and design cost involved with A/B testing, but it is sort of ironical that you can't be judgmental about which A/B tests to run, because if you already know which ones are going to work, why A/B test in the first place?
- StavrosK 15y agoNo questions here, just a small niggle: "Fewer users", not "less".
- mkl 15y agoActually, that's a widespread misconception. See here, for example: http://itre.cis.upenn.edu/~myl/languagelog/archives/003775.html http://itre.cis.upenn.edu/~myl/languagelog/archives/003775.h... That's a linguist's take on it - you can find many more by googling [fewer less languagelog] or similar.
- StavrosK 15y agoThat mainly refers to units of time, amounts, etc. "Less than 5 users", I can maybe accept. "Less users", no.
- Kaizyn 15y agoDo you have any sort of logical explanation or intuition about your stated result - that 75 percent of tests do not improve business results? It seems to suggest that design just has to be good enough. Below some set threshold for visually appeal and/or usability the design hinders money-making. However, above that threshold, further improvement isn't necessary? If this is a reasonable interpretation, that is actually quite sad and suggests that all the new design-centered startups are wasting a lot of money.