The math required to interpret 20 results is different than the math required to interpret 1 result. It’s not a weakness of the experiment itself. On some level

The math required to interpret 20 results is different than the math required to interpret 1 result. It’s not a weakness of the experiment itself. On some level the math required to interpret 20 results may not actually exist. Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive. Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.