3 ms·

Very often your current users get used to your design and hate it when you change something. Is there a way to do A/B testing on your current users, or should y

by pisarzp 13y ago

Very often your current users get used to your design and hate it when you change something. Is there a way to do A/B testing on your current users, or should you try it only on new ones?

patio11 13y ago
You can, theoretically, only A/B test things against new users. Most A/B testing tools do not support this behavior out of the box. The one time I made a change which was drastic enough to consider doing that, I just put a revert button on the interface (on both sides) and got ready to tell people how to use it. It turns out nobody asked. In general, though, many of the things which are most valuable to test border on imperceptible for long-term users of the site. For example, do you use Dropbox? (Picking a well-known example locally.) Can you identify the H1 on dropbox.com? Can you identify the button copy on dropbox.com? Most seasoned users can't, and won't notice changes to these elements, yet they're strongly influential on free trial signups. Can you freehand sketch the Dropbox credit card form? How about identifying the button copy on it? These have substantial impact on purchases at the margins. People only see them once (well, typical case for software), and nobody remembers them for more than a few minutes. Think back to your first run experience with Dropbox. How many steps did it have? What were their names? I'm going to bet you that the onboarding process for Dropbox has had more dedicated optimization effort than everything else the company does combined, but the median number of exposures per user is one.
- ampersandy 13y ago
  Great insight! Though I'd expand on your bet and claim that at almost every major site, a majority of testing is channeled into onboarding. It's certainly the case at Twitter and Facebook that a lot of work goes into optimizing a user's first steps.

by pisarzp 13y ago

Very often your current users get used to your design and hate it when you change something. Is there a way to do A/B testing on your current users, or should you try it only on new ones?

patio11 13y ago

You can, theoretically, only A/B test things against new users. Most A/B testing tools do not support this behavior out of the box. The one time I made a change which was drastic enough to consider doing that, I just put a revert button on the interface (on both sides) and got ready to tell people how to use it. It turns out nobody asked. In general, though, many of the things which are most valuable to test border on imperceptible for long-term users of the site. For example, do you use Dropbox? (Picking a well-known example locally.) Can you identify the H1 on dropbox.com? Can you identify the button copy on dropbox.com? Most seasoned users can't, and won't notice changes to these elements, yet they're strongly influential on free trial signups. Can you freehand sketch the Dropbox credit card form? How about identifying the button copy on it? These have substantial impact on purchases at the margins. People only see them once (well, typical case for software), and nobody remembers them for more than a few minutes. Think back to your first run experience with Dropbox. How many steps did it have? What were their names? I'm going to bet you that the onboarding process for Dropbox has had more dedicated optimization effort than everything else the company does combined, but the median number of exposures per user is one.

ampersandy 13y ago
Great insight! Though I'd expand on your bet and claim that at almost every major site, a majority of testing is channeled into onboarding. It's certainly the case at Twitter and Facebook that a lot of work goes into optimizing a user's first steps.