4 ms·

I assume Science knows what it's talking about, so I'm confused on a couple of points. Perhaps some practicing scientists could clear them up for everyone: For

I assume Science knows what it's talking about, so I'm confused on a couple of points. Perhaps some practicing scientists could clear them up for everyone: For any arbitrary paper, what is your assumption about its accuracy? How much do you rely on it? Can you put a number to it? The null hypothesis that research papers, especially in very difficult fields like social science, are near-infallible seems to be the error, AFAICT. It seems like something people outside science would assume, but I see scientists who are surprised by a 62% replication rate. I'm not surprised or concerned, but maybe I should be. Why do they call increasing the number of participants, "generous"? Doesn't that increase accuracy, and isn't accuracy the whole point of the replication study? Generosity implies some sort of favor, something above and beyond, while in this case it seems necessary based on the fact that the results changed - it would be failure of the replication study to not increase the number of participants.

TangoTrotFox 8y ago
When you repeat things over and over, even with an increasingly large sample, you artificially increase the chances of replicating something purely by chance. When papers are published they tend to publish ranges for the expected strength of their findings. What replication studies have found is that these ranges are not representative of reality in most cases. It's not about 'these papers are not infallible' but 'these papers seem to have misrepresented the strength of their findings.' And like the article mentions, even with this rather generous method of trying to 'replicate' studies, they found huge chunks of papers could not be replicated to anything like the strength originally suggested. The unstated implication here is that it seems a very large chunk of social science researchers are faking their numbers, likely through p-hacking, to create attention (and journal) grabbing headlines that are, for lack of a better word, simply fake.
BeetleB 8y ago
>For any arbitrary paper, what is your assumption about its accuracy? From what I've seen in academia, it varies depending on a number of variables, like who wrote the paper and whether their results help or hurt my research. Scientists are human, after all. And in my discipline, it was rare that the paper would have enough concrete details to replicate, so this way of thinking was the norm.
Fomite 8y ago
"For any arbitrary paper, what is your assumption about its accuracy? How much do you rely on it? Can you put a number to it? The null hypothesis that research papers, especially in very difficult fields like social science, are near-infallible seems to be the error, AFAICT. It seems like something people outside science would assume, but I see scientists who are surprised by a 62% replication rate. I'm not surprised or concerned, but maybe I should be." There is no such thing as "an arbitrary paper". It will depend very much on the study in question - including soft factors like who wrote it, but also the study design, sample size, if I think they approached the statistics appropriately, etc. "Why do they call increasing the number of participants, "generous"? Doesn't that increase accuracy, and isn't accuracy the whole point of the replication study? Generosity implies some sort of favor, something above and beyond, while in this case it seems necessary based on the fact that the results changed - it would be failure of the replication study to not increase the number of participants." The reason it might be thought of as generous is the criteria they determine for saying something is replicated is a significant effect in the same direction as the original study (for the record, I hate this criterion). A larger study is more likely to find a significant result if indeed there is one there, so they're giving studies that report an effect a very strong chance of seeing that replicated if indeed it was real.

TangoTrotFox 8y ago

When you repeat things over and over, even with an increasingly large sample, you artificially increase the chances of replicating something purely by chance. When papers are published they tend to publish ranges for the expected strength of their findings. What replication studies have found is that these ranges are not representative of reality in most cases. It's not about 'these papers are not infallible' but 'these papers seem to have misrepresented the strength of their findings.' And like the article mentions, even with this rather generous method of trying to 'replicate' studies, they found huge chunks of papers could not be replicated to anything like the strength originally suggested. The unstated implication here is that it seems a very large chunk of social science researchers are faking their numbers, likely through p-hacking, to create attention (and journal) grabbing headlines that are, for lack of a better word, simply fake.

BeetleB 8y ago

>For any arbitrary paper, what is your assumption about its accuracy? From what I've seen in academia, it varies depending on a number of variables, like who wrote the paper and whether their results help or hurt my research. Scientists are human, after all. And in my discipline, it was rare that the paper would have enough concrete details to replicate, so this way of thinking was the norm.

Fomite 8y ago

"For any arbitrary paper, what is your assumption about its accuracy? How much do you rely on it? Can you put a number to it? The null hypothesis that research papers, especially in very difficult fields like social science, are near-infallible seems to be the error, AFAICT. It seems like something people outside science would assume, but I see scientists who are surprised by a 62% replication rate. I'm not surprised or concerned, but maybe I should be." There is no such thing as "an arbitrary paper". It will depend very much on the study in question - including soft factors like who wrote it, but also the study design, sample size, if I think they approached the statistics appropriately, etc. "Why do they call increasing the number of participants, "generous"? Doesn't that increase accuracy, and isn't accuracy the whole point of the replication study? Generosity implies some sort of favor, something above and beyond, while in this case it seems necessary based on the fact that the results changed - it would be failure of the replication study to not increase the number of participants." The reason it might be thought of as generous is the criteria they determine for saying something is replicated is a significant effect in the same direction as the original study (for the record, I hate this criterion). A larger study is more likely to find a significant result if indeed there is one there, so they're giving studies that report an effect a very strong chance of seeing that replicated if indeed it was real.