2 ms·
> You ship an experiment that’s +10% in your conversion funnel. Then your revenue/installs/whatever goes up by +10% right? Wrong :( Turns out usually it goes up
by MaxBarraclough 8y ago
> You ship an experiment that’s +10% in your conversion funnel. Then your revenue/installs/whatever goes up by +10% right? Wrong :( Turns out usually it goes up a little bit, or maybe not at all.
Never mind "The difference between high- and low-intent users", this could be explained in terms of regression-toward-the-mean, a phenomenon mention in neither the article, nor the discussion here.
Have 1000 students do an IQ test. Pick the top 20 students. Have them do another IQ test next week. Their mean score second time round will almost certainly be lower than their mean score first time round. The reason they made the top 20 the first time round was a combination of having a high true IQ, and being lucky on the day. Second time round, they aren't 'defined to be lucky', as it were.
It's the reason movie sequels tend to be worse than the original. The reason the sequel was made was that the original movie was far more successful than the average movie, on account of both unusually skillful creators, and unusually good luck. Second time round, you can't count on the luck component again.