5 ms·

I'm going to have to disagree here, based on data. We did A/B testing of our homepage with and without a phone number for months (using odd/even days) and simpl

by spolsky 18y ago

I'm going to have to disagree here, based on data. We did A/B testing of our homepage with and without a phone number for months (using odd/even days) and simply could not see any measurable difference in sales. With all due respect to Ian, he makes a great product, and having a phone number on your web site is a good thing, but having the phone number on the home page (rather than a contact page) did ZERO for us.

zmimon 18y ago
> (using odd/even days) That sounds like an odd way to do it - do your customers typically purchase on the same day they first hit your web site? If not, they likely came back several times and might have been bemused or even frustrated to find the phone number appearing and then disappearing. Or just seeing the phone number one of those days was enough to accrue all the benefit of having the phone number on any day.
- dkokelley 18y ago
  This is a good point. I assume the testing measured how many sales were made during which days. A more thorough test would be 1 month intervals for say 6 months (even better would be for them to figure out how long the sales cycle typically takes from the 1st visit, double it and add a day, then alternate for that period).
- aneesh 18y ago
  Good point. A better way would be to randomly sort visitors to the site into either the A or B group, and make that sorting sticky over time (cookies aren't perfect, but are a reasonable approximation). No magically disappearing phone number!
- jfarmer 18y ago
  I presume he means he puts all users who arrive on an even day into the control and all users who arrive on an odd day into the "show a phone number" bucket. You then use cookies to track which user is in what bucket, because you have to know which treatment the user was in when they converted. As you said, that may or may not be on the same day they were treated. That's still kind of weird, though. You should just randomly assign each user as they arrive to one group or the other. If he's doing what you're suggesting that's bad methodology for all the reasons you said, and I wouldn't trust the results.
webwright 18y ago
Really interesting. It SEEMS sensible that the phone number should have an effect in terms of credibility and actual sales calls that end up with a sale. One theory might be that your audience is comprised of geeks. Less likely to pick up the phone, and less emotionality dependent on voice contact in terms of comfort/credibility. I wonder if A/B testing on Cabelas.com or LLBean.com would yield different results.

by spolsky 18y ago

zmimon 18y ago

> (using odd/even days) That sounds like an odd way to do it - do your customers typically purchase on the same day they first hit your web site? If not, they likely came back several times and might have been bemused or even frustrated to find the phone number appearing and then disappearing. Or just seeing the phone number one of those days was enough to accrue all the benefit of having the phone number on any day.

dkokelley 18y ago
This is a good point. I assume the testing measured how many sales were made during which days. A more thorough test would be 1 month intervals for say 6 months (even better would be for them to figure out how long the sales cycle typically takes from the 1st visit, double it and add a day, then alternate for that period).
aneesh 18y ago
Good point. A better way would be to randomly sort visitors to the site into either the A or B group, and make that sorting sticky over time (cookies aren't perfect, but are a reasonable approximation). No magically disappearing phone number!
jfarmer 18y ago
I presume he means he puts all users who arrive on an even day into the control and all users who arrive on an odd day into the "show a phone number" bucket. You then use cookies to track which user is in what bucket, because you have to know which treatment the user was in when they converted. As you said, that may or may not be on the same day they were treated. That's still kind of weird, though. You should just randomly assign each user as they arrive to one group or the other. If he's doing what you're suggesting that's bad methodology for all the reasons you said, and I wouldn't trust the results.

webwright 18y ago

Really interesting. It SEEMS sensible that the phone number should have an effect in terms of credibility and actual sales calls that end up with a sale. One theory might be that your audience is comprised of geeks. Less likely to pick up the phone, and less emotionality dependent on voice contact in terms of comfort/credibility. I wonder if A/B testing on Cabelas.com or LLBean.com would yield different results.