3 ms·
> It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias. So
by CoastalCoder 3y ago
> It’s not that we care about their intent, it’s that we care about whether their conclusions are more likely to represent a genuine effect vs. sample bias.
So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?
- jncfhnb 3y agoThe math required to interpret 20 results is different than the math required to interpret 1 result. It’s not a weakness of the experiment itself. On some level the math required to interpret 20 results may not actually exist. Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive. Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.