3 ms·
OK, So just to get a handle on this stuff, the two problems you have if you do a test and only look at a low p are: 1) Unusual things do occur. If a million p
by joe_the_user 13y ago
OK,
So just to get a handle on this stuff, the two problems you have if you do a test and only look at a low p are:
1) Unusual things do occur. If a million people do the same test, it's obvious they'll come up with some wrong values. It's less obvious that a similar number of wrong values will come if a million people do a million different tests with a similar small chance of bogus results.
2) The pattern of results may indeed be unusual but not necessarily in the fashion you think it is. There may be a non-random pattern possessed by the data but may not because of your particular hypothesis but a "this is not random" result may seem to say your hypothesis does explain the data.
Does that characterize the problem?