3 ms·

I would love to see the data sets that support the lifts - those little "up 3.49 / down 3.38" can get notoriously tricky in terms of statistical relevancy using

by aresant 15y ago

I would love to see the data sets that support the lifts - those little "up 3.49 / down 3.38" can get notoriously tricky in terms of statistical relevancy using GWO or otherwise. Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns. Sometimes the sheer "newness" can skew the data as folks sitting on the fence (returning visitors) push the convo data up. All that said, I'm a fan of the new look and feel, fun to watch 37signals influence turn the market - expect to see lots of similar looking stuff in the future.

asianmack 15y ago
"Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns." I totally agree. However, this is a marketing site that "established clients" probably won't be visiting often. Once they've signed up the site has done its job right?
jcampbell1 15y ago
> Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns. I have seen this a number of times. I change the design, and it is a statistically significant winner, then I continue to let the experiment run, and the conversion rate falls back to the original level. I have also found that rotating the design, is statistically better than never changing it.
- noelwelsh 15y ago
  There is the problem with repeated hypothesis testing: every test has a probability of error. If you test often enough you will make an error.
  - rmc 15y ago
    There's an xkcd strip for that: http://xkcd.com/882/ http://xkcd.com/882/
    - steve-howard 15y ago
      That strip is less about errors and more about the science news cycle. Another good reference for that: http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174
noelwelsh 15y ago
Heck yes. It is so easy to do A/B testing incorrectly. Without knowing at least the number of hits the results are almost meaningless.
sunir 15y ago
There are a couple important biases where just changing something results in a positive result. * http://en.wikipedia.org/wiki/Novelty_effect http://en.wikipedia.org/wiki/Novelty_effect (everyone loves seeing something new!) * http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect (being a subject of an experiment is motivational) There are a couple ways to deal with these biases. One is to try a large number of different designs, so the bias is spread across a large number of options. Thus the comparative winner is more likely to itself be intrinsically better, at least amongst the new designs. Note that this still negatively punishes the control. The other way is to wait longer until the new design is no longer 'new'. In practice, that is impractical! Really, all you can do is pick the winner and keep monitoring it after the fact to see if there is a regression in the result. I know others have said this, but you can harness these biases. Simply change the design frequently for no reason other than to stay 'fresh'. It keeps people interested and it shows you're alive and kicking.
- chc 15y ago
  The Hawthorne effect probably wouldn't apply since it requires the subject to know he's the subject of an experiment. The novelty of the new designs might be a factor, but since we're measuring new sign-ups, it seems kind of unlikely that a lot of people were familiar enough with the design of the page to realize it was new.
  - jcampbell1 15y ago
    It is highly likely that people know they are looking at a new design. People don't just adopt a CRM on a whim. For my site, (foreign language learning), the customer visits an average of 6 times before buying. New designs drop the visit count temporarily, a clear sign the novelty effect is strong.
chc 15y ago
The article actually notes that those up/down 2% results are statistically insignificant ("a specific person didn’t quite matter among the set of people we tested"). But the lift from no-person to big-smiling-face was very significant: more than 100%.

by aresant 15y ago

asianmack 15y ago

"Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns." I totally agree. However, this is a marketing site that "established clients" probably won't be visiting often. Once they've signed up the site has done its job right?

jcampbell1 15y ago

> Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns. I have seen this a number of times. I change the design, and it is a statistically significant winner, then I continue to let the experiment run, and the conversion rate falls back to the original level. I have also found that rotating the design, is statistically better than never changing it.

noelwelsh 15y ago
There is the problem with repeated hypothesis testing: every test has a probability of error. If you test often enough you will make an error.
- rmc 15y ago
  There's an xkcd strip for that: http://xkcd.com/882/ http://xkcd.com/882/
  - steve-howard 15y ago
    That strip is less about errors and more about the science news cycle. Another good reference for that: http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174

noelwelsh 15y ago

Heck yes. It is so easy to do A/B testing incorrectly. Without knowing at least the number of hits the results are almost meaningless.

sunir 15y ago

There are a couple important biases where just changing something results in a positive result. * http://en.wikipedia.org/wiki/Novelty_effect http://en.wikipedia.org/wiki/Novelty_effect (everyone loves seeing something new!) * http://en.wikipedia.org/wiki/Hawthorne_effect http://en.wikipedia.org/wiki/Hawthorne_effect (being a subject of an experiment is motivational) There are a couple ways to deal with these biases. One is to try a large number of different designs, so the bias is spread across a large number of options. Thus the comparative winner is more likely to itself be intrinsically better, at least amongst the new designs. Note that this still negatively punishes the control. The other way is to wait longer until the new design is no longer 'new'. In practice, that is impractical! Really, all you can do is pick the winner and keep monitoring it after the fact to see if there is a regression in the result. I know others have said this, but you can harness these biases. Simply change the design frequently for no reason other than to stay 'fresh'. It keeps people interested and it shows you're alive and kicking.

chc 15y ago
The Hawthorne effect probably wouldn't apply since it requires the subject to know he's the subject of an experiment. The novelty of the new designs might be a factor, but since we're measuring new sign-ups, it seems kind of unlikely that a lot of people were familiar enough with the design of the page to realize it was new.
- jcampbell1 15y ago
  It is highly likely that people know they are looking at a new design. People don't just adopt a CRM on a whim. For my site, (foreign language learning), the customer visits an average of 6 times before buying. New designs drop the visit count temporarily, a clear sign the novelty effect is strong.

chc 15y ago

The article actually notes that those up/down 2% results are statistically insignificant ("a specific person didn’t quite matter among the set of people we tested"). But the lift from no-person to big-smiling-face was very significant: more than 100%.