2 ms·
> Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns. I have seen this a number of times
by jcampbell1 15y ago
> Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns.
I have seen this a number of times. I change the design, and it is a statistically significant winner, then I continue to let the experiment run, and the conversion rate falls back to the original level.
I have also found that rotating the design, is statistically better than never changing it.
- noelwelsh 15y agoThere is the problem with repeated hypothesis testing: every test has a probability of error. If you test often enough you will make an error.
- rmc 15y agoThere's an xkcd strip for that: http://xkcd.com/882/ http://xkcd.com/882/
- steve-howard 15y agoThat strip is less about errors and more about the science news cycle. Another good reference for that: http://www.phdcomics.com/comics.php?f=1174 http://www.phdcomics.com/comics.php?f=1174