"The fallacy is that you're searching for a theory in a pile of data, rather than forming a theory and running an experiment to support or disprove it." But loo

"The fallacy is that you're searching for a theory in a pile of data, rather than forming a theory and running an experiment to support or disprove it." But looking at data is an important step in forming new hypotheses -- http://en.wikipedia.org/wiki/Exploratory_data_analysis http://en.wikipedia.org/wiki/Exploratory_data_analysis . You may just want to then gather more data to independently check on your ideas. (And to go back to the runs example, isn't the problem that the announcers weren't willing to seriously consider an alternate hypothesis, ie, they weren't doing enough simultaneous estimation?) "Instead of running multiple AdWords variants each against multiple landing page variants each feeding a different website funnel, run just one experiment at a time, one variable at a time." I think this is bad advice. If there's an interaction between your variables, this will lead you to totally miss it. Even without interactions, it can be a more efficient use of resources to estimate the effects of multiple variables at a time: http://en.wikipedia.org/wiki/Factorial_experiment http://en.wikipedia.org/wiki/Factorial_experiment Also, in this context, it seems worth looking at http://www.stat.columbia.edu/~cook/movabletype/archives/2008/03/why_i_dont_usua_1.html http://www.stat.columbia.edu/~cook/movabletype/archives/2008...