4 ms·
"back test data to outcomes" sounds scarily like p-hacking. That is, it seems highly likely you will still not find anything actually meaningful. You are just
by taeric 8y ago
"back test data to outcomes" sounds scarily like p-hacking. That is, it seems highly likely you will still not find anything actually meaningful. You are just guaranteeing that you will be able to find something.
This is especially true if you have retention numbers, already. Why bother asking if I was happy with a purchase, if I never buy again anyway.
Note that I am sympathetic. There is every chance that I will buy something again simply because the exit survey reminded me of your existence. There is also the chance having an exit survey will make me not want to come back. :)
- stuartaxelowen 8y agoNot so - the business is looking for real ROI, and thus will not be satisfied with topics that are "in the noise". Also, you create the measurement that establishes a problem before you apply the treatment (create new feature X or fix bug Y), and then verify after treatment that the desired affect was achieved.
- taeric 8y agoIf you are scanning historic data looking for correlations, you will almost certainly fall for some of the same traps that are common in p hacking. Now, can you mitigate some by carefully doing an experiment again? Certainly. But you should also not constrain experiments to only things to have been measuring already.