3 ms·

Let's say Galileo drops two spheres of independent mass off the Leaning Tower of Pisa, and publishes a study saying they both fall with the same acceleration an

Let's say Galileo drops two spheres of independent mass off the Leaning Tower of Pisa, and publishes a study saying they both fall with the same acceleration and concluding this is true of all objects dropped on Earth, assuming they're heavy enough to ignore air resistance. Is it evidence against this study if you perform the same experiment from the Empire State Building and get a different result? Yes, because it matters that the study's conclusion wasn't specific to one building. It's a bit of a low blow to say changing the country the study was performed in invalidates it as a reproduction, if the study itself made conclusions broader than a single country.

andai 4y ago
Modafinil increases histamine release in the anterior hypothalamus of rats... in Japan!
notahacker 4y ago
On the other hand, if you perform a slightly different experiment from the Empire State Building and get different results whilst someone else replicates it perfectly in Pisa, is it justified to ignore the Pisa replication altogether, write off gravity as a theory and add it to your meta-analysis of replication crises in physics, or should you in fact be looking at how the differences in experiment might affect the results? (Maybe the Empire State Building is too tall or not Italian enough for gravity to take effect, or maybe the problem is you're dropping things that glide...). Especially when most of the failed "replications" were things the original study authors said up front weren't valid replications of what they were studying. It's possible to predict in advance that people from a different culture will have a different reaction than Stanford students to Stanford students talking about a hot button issue on their campus (it is also not surprising culture has more impact on psychology than physics!). And even pure ad-hoc explanations of the effect of differences in experimental design (OK, we think the problem might not be the Empire State Building, it might be that they were dropping gliders!) after an unexpected replication fail are more helpful testable hypotheses for science than writing off the original theory altogether. (Though sometimes good hypotheses include "the construction of the original experiment made this mistake, which could be tested by replicating with and without this source of sampling bias ...")

andai 4y ago

Modafinil increases histamine release in the anterior hypothalamus of rats... in Japan!

notahacker 4y ago

On the other hand, if you perform a slightly different experiment from the Empire State Building and get different results whilst someone else replicates it perfectly in Pisa, is it justified to ignore the Pisa replication altogether, write off gravity as a theory and add it to your meta-analysis of replication crises in physics, or should you in fact be looking at how the differences in experiment might affect the results? (Maybe the Empire State Building is too tall or not Italian enough for gravity to take effect, or maybe the problem is you're dropping things that glide...). Especially when most of the failed "replications" were things the original study authors said up front weren't valid replications of what they were studying. It's possible to predict in advance that people from a different culture will have a different reaction than Stanford students to Stanford students talking about a hot button issue on their campus (it is also not surprising culture has more impact on psychology than physics!). And even pure ad-hoc explanations of the effect of differences in experimental design (OK, we think the problem might not be the Empire State Building, it might be that they were dropping gliders!) after an unexpected replication fail are more helpful testable hypotheses for science than writing off the original theory altogether. (Though sometimes good hypotheses include "the construction of the original experiment made this mistake, which could be tested by replicating with and without this source of sampling bias ...")