3 ms·
This is really not the best way to think about this problem. The issue is not that it's necessarily bad to test many hypotheses, it's that testing more hypothe
by jessriedel 16y ago
This is really not the best way to think about this problem. The issue is not that it's necessarily bad to test many hypotheses, it's that testing more hypotheses requires more data. For instance OP says
>Instead of using a thesaurus to generate 10 ad variants, decide what pain-points or language you think will grab potential customers and test that theory specifically.
In other words, he says test one hypothesis rather than ten. But there's nothing wrong with testing ten variants so long as you have enough data to ensure that the chance that any of the variants produce a false positive because of statistical fluctuation is small.
Given a fixed, finite amount of data, you can do some pretty simple statistics to find out exactly how many hypothesis you can test at a given confidence level.