11 ms·

How we test fake sites on live traffic

165 points by jaf12duke 13y ago

josefresco 13y ago
A theory on why the "ugly banking site" did so well. Out of all of the designs, that was the most traditional, and probably most "comfortable" to the user. The site to me feels like it has depth, and it's friendly to the user, encouraging them to explore. The others while more effective (conversion rate) felt like they were engineered to spoon feed me just the right data, and didn't feel like "complete" sites open to browsing. This of course is all anecdotal based on my years of being a web geek but sometimes the "feel" of the design speaks volumes where data and raw number crunching cannot.
- paul0 13y ago
  Agree. The 'ugly banking site' is a traditional design who is still a reference for a major part of visitors.
- robmcm 13y ago
  Perhaps the "Ugly banking site" makes them think you're not a bunch of kids with some VC money ;) Seriously though, it could be that a less trendy design gives the users some reassurances that you've been around since that design style was in fashion.
- stevesearer 13y ago
  I love that Ugly Banking site is clearly based on US Bank's website, which was actually redesigned in the last 12 months. https://www.usbank.com/ https://www.usbank.com/
  - namenotrequired 13y ago
    It looks no more like that one than most other bank websites I've seen :)
agoandanon 13y ago
The hover version exposed the most information in a usable and visually-understandable format. I am not surprised.
- bentoner 13y ago
  Did you conclude this before reading the answer?
jdmitch 13y ago
How long did you have the fake sites up? I would have thought you couldn't leave them up for very long before frustrating users, but it seems like you would need a good chunk of data before you would know it was statistically significant. Or did you also have some sort of disclaimer for those users so they knew it was a beta test?
- darrennix 13y ago
  We left the fake sites up for 3 weeks. The parts of the design that required a DB connection, like sorting, pagination, contacting a listing, would just redirect to the matching operation on the live site on click.
richardv 13y ago
Looking at a listing[1], I notice that you provide email addresses to users and presumably act like a craiglists for anonymous messaging... I'm just curious though, don't you find that you get spammed out if you do this? I think providing email addresses to users on a marketplace is a great way of keeping discussions/messages boxed up in your platform, but showing the emails on the site must result in way too many spam emails? [1] : http://42floors.com/ny/new-york/25-w-39th-st/3171 http://42floors.com/ny/new-york/25-w-39th-st/3171
- darrennix 13y ago
  We used to expose the raw email address of the underlying user. Now it's a two-way email proxy, which is why the domain is "Floors.me" e.g. rich.farmey.31719@42floors.me We do get spammed but we've been able to keep a handle on it. Mailgun's filters capture most of the spam and we have backup handlers in place to address the rest.
  - underwater 13y ago
    Do users know who they're emailing with the anonymous addresses? The username@42floors.me address may look like they have to communicate via 42floors (meaning more friction) instead of directly with the manager. Craigslist emails are more obviously throw aways.
bullseye 13y ago
> The winning variation was Google Hover Clone I was enjoying the article, but feel like it just abruptly ended. Why was the clone the winner? I would have loved to hear more about your process for judging the successful mockup.
- fat0wl 13y ago
  Bump.... if Ugly Banking Site had lowest bounce rate why wasn't it the "winner"? If you're just looking at a static page what else is judged other than bounce rate? If you're measuring conversions, isn't most of that influenced by the existing site? Or do you attribute it to some lasting impression based on how the site was entered? Glad Ugly Banking Site didn't win, but I am curious about the critical metrics.
  - frogpelt 13y ago
    Fewer bounces doesn't mean more conversions. The winning site might have generated the most tours of office space but the Ugly Banking Site may have generated more overall page views. That's what I got from it.
    - fat0wl 13y ago
      I understand & agree, but my point was that conversions would still be accomplished from the existing site, just entered via a static landing page. The conversions weren't coming from the static page, it was just an entryway into the existing site. Enter = no bounce, Exit = bounce Correlating stats collected once they leave (click anything) with these static pages is a bit less scientific, so I was interested in hearing the logic.
- darrennix 13y ago
  Good point. I just added this note to the end of the article: Our criteria for picking the winner came down to one number: tour request rate. In other words, what percentage of users found a listing they liked and contacted it using a particular design. We use other data points to augment this core metric (bounce rate, time on site, number of listings viewed, search criteria revisions), but ultimately a variation only wins if it's better than the control at getting users to an office space that they like.
robmcm 13y ago
Isn't there a worry that "fake" sights would damage your reputation?
- wikwocket 13y ago
  This reminds me of the worry of the potential A/B tester: won't my users get confused if they see site A from one web browser/computer, but site B from another? I think this is just a self-defeating worry, though. Most people won't know or care about your site or brand, or remember you from one session to another. If they don't like your page, they'll just click the back button. If they do get upset, well, the internet is effectively a pool of infinitely more people who can click your adwords. So exercise caution when changing big features used by paying customers, but don't be afraid to experiment and show things to random visitors.
EvanL 13y ago
Very interesting, thanks for sharing! What specific actions did you measure to determine a winner? Noticed you answered the question above. *Was just clawing my eyes out browsing craigslist for NYC spaces, remember hearing about you guys a little while back, thanks for refreshing my memory ;)
thatthatis 13y ago
Hover clone was the only one that put the locations on a map. This test seems to, more than anything else, have proven that when searching for office space _exact_ location is one of the primary criteria. I wouldn't be at all surprised if "just show me where it is on a map" is a feature most commercial real-estate search sites are missing.
- darrennix 13y ago
  This explanation does not address the fact that all three of the original designs had a heavy map emphasis including the ability in Unified View to make the map full screen. Keep in mind that the split test was 9-way in that each variation had to outperform the control. I think hover1 outperformed all versions because it gives the ability to quickly process massive amounts of data without clicking or changing pages.
  - thatthatis 13y ago
    The ability to make the map full screen != a large easy to use map and definitely != ability to see exact location. What percent of users clicked to make the map full screen? "The ability to quickly process massive amounts of data without clicking or changing pages" sounds like it may be a good design mantra for your business.
makmanalp 13y ago
So didn't users get pissed when they landed on the fake site?
- dangerlibrary 13y ago
  Real data, just pre-populated. Not live. Cached, for weeks. You'd never know unless you tried to book and were told the space was already taken.
lukethomas 13y ago
I really enjoyed this article - the one question I have is if the "winner" has been rolled out to the masses (not only the SEM channel.) I'm a little scared for you guys because the test results show that the design works well for visitors from SEM, not necessarily from other channels. Have you tested this on traffic from other channels (organic, referral, etc)?
level09 13y ago
I kinda liked the map more. I thought moving to this new hover layout was a business decision (e.g: get some clients to pay for being featured on top of the list as the map makes all listings have equal look/chance to be discovered). apparently you were just A/B testing.
gmjoe 13y ago
Getting user feedback on fake stuff is great. Tim Ferris didn't know what to call the book that turned out to be The 4 Hour Workweek, so he spent money on AdWords to see which title more people clicked on. [1] [1] http://futureperfectpublishing.com/2009/01/23/the-sticky-goodness-of-testing-book-titles-with-google-adwords/ http://futureperfectpublishing.com/2009/01/23/the-sticky-goo...
johngrefe 13y ago
What is this gem called? I've seen it on boatbound and MyTime, now on 42floors. Same Rails based layout.
calbear81 13y ago
The hovering map seems to be behaving oddly (Chrome/Mac OS) for me and blocks off a large section of the top left with a beige colored background when I click on "More Map". Otherwise, very surprising to me that this was the winner. Did you see similar results across all browsers and form factors (especially interested in desktop vs. tablet).
trustfundbaby 13y ago
You should really consider changing the title to something a bit more descriptive. Its a very good article about strategies for A/B testing design ideas, but I didn't get that from the title, and would it skipped it if not for the fact that I really enjoy 42floors blog posts.
chenster 13y ago
The keyword is "Fake". The static HTML5 websites take much less time to produce thus reduce programming costs and shortens precious time-to-lauch. In a nutshell: Moqups > PSD > PSD2HTML + Static data > Fake sites
pax 13y ago
what did it mean 'fake'? Weren't all 8 fully working versions of the website?
tux3 13y ago
Google in charge of having good designers : check. Why am I not surprised ?
inthewoods 13y ago
I'm not affiliated with the company, but Visual Website Optimizer offers the ability to do A/B tests between different URLs - rather than just variations within a page (which it also does).

165 points by jaf12duke 13y ago

A theory on why the "ugly banking site" did so well. Out of all of the designs, that was the most traditional, and probably most "comfortable" to the user. The site to me feels like it has depth, and it's friendly to the user, encouraging them to explore. The others while more effective (conversion rate) felt like they were engineered to spoon feed me just the right data, and didn't feel like "complete" sites open to browsing. This of course is all anecdotal based on my years of being a web geek but sometimes the "feel" of the design speaks volumes where data and raw number crunching cannot.

paul0 13y ago
Agree. The 'ugly banking site' is a traditional design who is still a reference for a major part of visitors.
robmcm 13y ago
Perhaps the "Ugly banking site" makes them think you're not a bunch of kids with some VC money ;) Seriously though, it could be that a less trendy design gives the users some reassurances that you've been around since that design style was in fashion.
stevesearer 13y ago
I love that Ugly Banking site is clearly based on US Bank's website, which was actually redesigned in the last 12 months. https://www.usbank.com/ https://www.usbank.com/
- namenotrequired 13y ago
  It looks no more like that one than most other bank websites I've seen :)

agoandanon 13y ago

The hover version exposed the most information in a usable and visually-understandable format. I am not surprised.

bentoner 13y ago
Did you conclude this before reading the answer?

jdmitch 13y ago

How long did you have the fake sites up? I would have thought you couldn't leave them up for very long before frustrating users, but it seems like you would need a good chunk of data before you would know it was statistically significant. Or did you also have some sort of disclaimer for those users so they knew it was a beta test?

darrennix 13y ago
We left the fake sites up for 3 weeks. The parts of the design that required a DB connection, like sorting, pagination, contacting a listing, would just redirect to the matching operation on the live site on click.

richardv 13y ago

Looking at a listing[1], I notice that you provide email addresses to users and presumably act like a craiglists for anonymous messaging... I'm just curious though, don't you find that you get spammed out if you do this? I think providing email addresses to users on a marketplace is a great way of keeping discussions/messages boxed up in your platform, but showing the emails on the site must result in way too many spam emails? [1] : http://42floors.com/ny/new-york/25-w-39th-st/3171 http://42floors.com/ny/new-york/25-w-39th-st/3171

darrennix 13y ago
We used to expose the raw email address of the underlying user. Now it's a two-way email proxy, which is why the domain is "Floors.me" e.g. rich.farmey.31719@42floors.me We do get spammed but we've been able to keep a handle on it. Mailgun's filters capture most of the spam and we have backup handlers in place to address the rest.
- underwater 13y ago
  Do users know who they're emailing with the anonymous addresses? The username@42floors.me address may look like they have to communicate via 42floors (meaning more friction) instead of directly with the manager. Craigslist emails are more obviously throw aways.

bullseye 13y ago

> The winning variation was Google Hover Clone I was enjoying the article, but feel like it just abruptly ended. Why was the clone the winner? I would have loved to hear more about your process for judging the successful mockup.

fat0wl 13y ago
Bump.... if Ugly Banking Site had lowest bounce rate why wasn't it the "winner"? If you're just looking at a static page what else is judged other than bounce rate? If you're measuring conversions, isn't most of that influenced by the existing site? Or do you attribute it to some lasting impression based on how the site was entered? Glad Ugly Banking Site didn't win, but I am curious about the critical metrics.
- frogpelt 13y ago
  Fewer bounces doesn't mean more conversions. The winning site might have generated the most tours of office space but the Ugly Banking Site may have generated more overall page views. That's what I got from it.
  - fat0wl 13y ago
    I understand & agree, but my point was that conversions would still be accomplished from the existing site, just entered via a static landing page. The conversions weren't coming from the static page, it was just an entryway into the existing site. Enter = no bounce, Exit = bounce Correlating stats collected once they leave (click anything) with these static pages is a bit less scientific, so I was interested in hearing the logic.
darrennix 13y ago
Good point. I just added this note to the end of the article: Our criteria for picking the winner came down to one number: tour request rate. In other words, what percentage of users found a listing they liked and contacted it using a particular design. We use other data points to augment this core metric (bounce rate, time on site, number of listings viewed, search criteria revisions), but ultimately a variation only wins if it's better than the control at getting users to an office space that they like.

robmcm 13y ago

Isn't there a worry that "fake" sights would damage your reputation?

wikwocket 13y ago
This reminds me of the worry of the potential A/B tester: won't my users get confused if they see site A from one web browser/computer, but site B from another? I think this is just a self-defeating worry, though. Most people won't know or care about your site or brand, or remember you from one session to another. If they don't like your page, they'll just click the back button. If they do get upset, well, the internet is effectively a pool of infinitely more people who can click your adwords. So exercise caution when changing big features used by paying customers, but don't be afraid to experiment and show things to random visitors.

EvanL 13y ago

Very interesting, thanks for sharing! What specific actions did you measure to determine a winner? Noticed you answered the question above. *Was just clawing my eyes out browsing craigslist for NYC spaces, remember hearing about you guys a little while back, thanks for refreshing my memory ;)

thatthatis 13y ago

Hover clone was the only one that put the locations on a map. This test seems to, more than anything else, have proven that when searching for office space _exact_ location is one of the primary criteria. I wouldn't be at all surprised if "just show me where it is on a map" is a feature most commercial real-estate search sites are missing.

darrennix 13y ago
This explanation does not address the fact that all three of the original designs had a heavy map emphasis including the ability in Unified View to make the map full screen. Keep in mind that the split test was 9-way in that each variation had to outperform the control. I think hover1 outperformed all versions because it gives the ability to quickly process massive amounts of data without clicking or changing pages.
- thatthatis 13y ago
  The ability to make the map full screen != a large easy to use map and definitely != ability to see exact location. What percent of users clicked to make the map full screen? "The ability to quickly process massive amounts of data without clicking or changing pages" sounds like it may be a good design mantra for your business.

makmanalp 13y ago

So didn't users get pissed when they landed on the fake site?

dangerlibrary 13y ago
Real data, just pre-populated. Not live. Cached, for weeks. You'd never know unless you tried to book and were told the space was already taken.

lukethomas 13y ago

I really enjoyed this article - the one question I have is if the "winner" has been rolled out to the masses (not only the SEM channel.) I'm a little scared for you guys because the test results show that the design works well for visitors from SEM, not necessarily from other channels. Have you tested this on traffic from other channels (organic, referral, etc)?

level09 13y ago

I kinda liked the map more. I thought moving to this new hover layout was a business decision (e.g: get some clients to pay for being featured on top of the list as the map makes all listings have equal look/chance to be discovered). apparently you were just A/B testing.

gmjoe 13y ago

Getting user feedback on fake stuff is great. Tim Ferris didn't know what to call the book that turned out to be The 4 Hour Workweek, so he spent money on AdWords to see which title more people clicked on. [1] [1] http://futureperfectpublishing.com/2009/01/23/the-sticky-goodness-of-testing-book-titles-with-google-adwords/ http://futureperfectpublishing.com/2009/01/23/the-sticky-goo...

johngrefe 13y ago

What is this gem called? I've seen it on boatbound and MyTime, now on 42floors. Same Rails based layout.

calbear81 13y ago

The hovering map seems to be behaving oddly (Chrome/Mac OS) for me and blocks off a large section of the top left with a beige colored background when I click on "More Map". Otherwise, very surprising to me that this was the winner. Did you see similar results across all browsers and form factors (especially interested in desktop vs. tablet).

trustfundbaby 13y ago

You should really consider changing the title to something a bit more descriptive. Its a very good article about strategies for A/B testing design ideas, but I didn't get that from the title, and would it skipped it if not for the fact that I really enjoy 42floors blog posts.

chenster 13y ago

The keyword is "Fake". The static HTML5 websites take much less time to produce thus reduce programming costs and shortens precious time-to-lauch. In a nutshell: Moqups > PSD > PSD2HTML + Static data > Fake sites

pax 13y ago

what did it mean 'fake'? Weren't all 8 fully working versions of the website?

tux3 13y ago

Google in charge of having good designers : check. Why am I not surprised ?

inthewoods 13y ago

I'm not affiliated with the company, but Visual Website Optimizer offers the ability to do A/B tests between different URLs - rather than just variations within a page (which it also does).