4 ms·

Quote: "The issue of replication goes to the heart of the scientific process." Indeed it does, but some fields don't think it's very important. For example, th

by lutusp 10y ago

Quote: "The issue of replication goes to the heart of the scientific process." Indeed it does, but some fields don't think it's very important. For example, there's a surprisingly widespread attitude among psychologists that it's either a waste of time or an attack on the integrity of one's colleagues. Example: http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_science_2014.pdf http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_... Author: Professor Jason Mitchell, Harvard University. Quotes: "Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value." "The field of social psychology can be improved, but not by the publication of negative findings." "Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues." "Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way." It seems not to have occurred to Professor Mitchell that the original study's result might also have resulted from someone bungling something along the way.

therpe1 10y ago
That's because Jason Mitchell is largely right and I suspect most scientists would, if pressed, agree with him. The reality is that science is a largely social process. It "works" because institutions have come together to devise cultures and processes that produce useful new knowledge. Now it may be that many people would rather not know how the sausage is made (eg the level of histrionics in this thread)but nobody but the very foolish would argue with the outputs of this very unique social process. And while it may provide some small measure of comfort to insist that this is a problem with just the "social" sciences, this comfort is illusory. Even physics, as it has become increasingly expensive and abstract, has entered this domain. But I don't foresee people abandoning physics because of non-reproducibility as long as useful new knowledge is produced. To that extent all the concern over failed replications is just that, hand-wringing. The underlying problem which Mitchell hints at is that even if you replicated every experiment a million times this would not necessarily prove anything unless you have somehow managed to find "flawless experimenters." The logical conclusion is that even successful replications produce little new knowledge and there are indeed experiments that have been successfully replicated... until they're not. A person with a grade school level of understanding of science might declare the whole enterprise flawed and call for it abandoned but I think most adults would recognize the flaws, work to improve them, and keep forward. In that light the "reproducibility crisis" is for the most part a kind of media phenomenon -- nothing in reality has changed (and it's certainly not the case that experiments in the past were reproducible and science needs to be made great again) -- it's just that more people are understanding what science really is and how it can be improved.

by lutusp 10y ago

therpe1 10y ago

That's because Jason Mitchell is largely right and I suspect most scientists would, if pressed, agree with him. The reality is that science is a largely social process. It "works" because institutions have come together to devise cultures and processes that produce useful new knowledge. Now it may be that many people would rather not know how the sausage is made (eg the level of histrionics in this thread)but nobody but the very foolish would argue with the outputs of this very unique social process. And while it may provide some small measure of comfort to insist that this is a problem with just the "social" sciences, this comfort is illusory. Even physics, as it has become increasingly expensive and abstract, has entered this domain. But I don't foresee people abandoning physics because of non-reproducibility as long as useful new knowledge is produced. To that extent all the concern over failed replications is just that, hand-wringing. The underlying problem which Mitchell hints at is that even if you replicated every experiment a million times this would not necessarily prove anything unless you have somehow managed to find "flawless experimenters." The logical conclusion is that even successful replications produce little new knowledge and there are indeed experiments that have been successfully replicated... until they're not. A person with a grade school level of understanding of science might declare the whole enterprise flawed and call for it abandoned but I think most adults would recognize the flaws, work to improve them, and keep forward. In that light the "reproducibility crisis" is for the most part a kind of media phenomenon -- nothing in reality has changed (and it's certainly not the case that experiments in the past were reproducible and science needs to be made great again) -- it's just that more people are understanding what science really is and how it can be improved.