3 ms·

This is incorrect, right? "If all 100 of the original studies examined by OSC had reported true effects, then sampling error alone should cause 5% of the repli

This is incorrect, right? "If all 100 of the original studies examined by OSC had reported true effects, then sampling error alone should cause 5% of the replication studies to “fail” by producing results that fall outside the 95% confidence interval of the original study" It'd have to be a pretty poor(read corrupt) field of study of all the random-ish studies chosen at close to 95%? If the original studies are legit, than many/most of them would be at much higher confidence intervals? Only scam fields would they all be around 95%? https://xkcd.com/882/ https://xkcd.com/882/ - this is more about how bad science creeps in legitimate fields, the authors would be implying all the studies in this fields are this bad. This would be my reading, not sure....

maxander 11y ago
You aren't by any chance a physicist or something, are you? :) 95% confidence is just about the lowest that a scientist in any field would think about publishing with, but in many fields- especially those whose subjects are the delicate and high-variance animals known as humans- statistical confidence is expensive. It wouldn't surprise me that lots of papers are put out that just barely reach the lowest standards of evidence, because grant-funded researchers couldn't afford the 1000 more test subjects it would take to get another sigma. Which isn't great, but strictly speaking its not doing society a disservice, either- even uncertain knowledge decreases the entropy of our vast ignorance. It does, however, add an important dimension to how these studies should be interpreted- that small chance that any given study is incorrect matters, particularly when there's such a mind-bogglingly huge amount of research being done in the modern day. Its entirely unsurprising to read about a study claiming with p < 0.01 that a glass of wine a day will make your hair fall out, when you consider that there's been many thousands of papers published about things like wine. Confidence- and p-values alone aren't going to save us there.
- aaron695 11y ago
  Yes, I see you're correct. Most studies would chose a 95% confidence interval so their statement stands.
vacri 11y ago
The article gives a clear example of why it is difficult to replicate psychology-based studies. Iron is iron everywhere, and behaves similarly. Put iron at STP, and it will behave the same in Massachusetts and Amsterdam. The same is not true of humans - the study they give as an example is "students opinions of affirmative action policies", a clearly political set of questions, taken of people who are not only not in the same political environment, but speak different native languages! Not being able to reproduce a political questionnaire done of Harvard students on U of A students should surprise exactly zero people, and should never have been used for verifying reproducibility.
- harry8 11y ago
  I'm not particularly interested in Harvard students. I'm pretty sure they're not super interesting to anyone outside of the Harvard diaspora and their friends. I'll bet 100% the paper was generalising from Harvard students to some wider population, or why would you publish it. If the paper says "this thing we're publishing is only applies to Harvard students and has zero wider significance." I'll eat humble pie and apologise. Psychology has a big issue of claiming things as fact that are no such thing. It has had this issue since forever. The history of psychology research is truly ethically hideous both in terms of what it did to people in experiments and what it claimed to have found that was then used to maim people in the general population. Research lobotomy if you have any doubts at all about that statement. Some fraud got a Nobel prize for research that he used to advocate "safe lobotomy with no adverse affects." (But maybe he fooled himself and it wasn't blatant fraud. Bah.) These are the screams from the psychology establishment as they get called out for claiming so much that is just totally false. It's really hard to know something. There is an immense amount of work you have to do and you all you might find after decades is "dead end". Not in Psych research papers and they absolutely deserve the contempt they get for it.
  - vacri 11y ago
    > The history of psychology research ... Research lobotomy... So, you've pretty much publicly announced that you're heavily clueless about what psychology is, and are buying into that armchair critic opinion that it's a discredited arm of psychiatry. Discussions on HN constantly revolve around psychology. How to negotiate for a better salary? Psychology. A/B testing? Psychology. How to give or take an interview effectively? Psychology. UX design and testing? Psychology. How to motivate yourself or others, or avoid procrastination? Psychology. Game theory? Psychology. And this is just the common discussions on the rather niche forum of HN. As I write this comment, the #1 article is "Dsxyliea", which is... psychology. When you abuse psychology because of a brief, discredited and niche episode in its history, you're saying the equivalent of "web browsers are shit software, because IE on XP can't do SNI".

maxander 11y ago

You aren't by any chance a physicist or something, are you? :) 95% confidence is just about the lowest that a scientist in any field would think about publishing with, but in many fields- especially those whose subjects are the delicate and high-variance animals known as humans- statistical confidence is expensive. It wouldn't surprise me that lots of papers are put out that just barely reach the lowest standards of evidence, because grant-funded researchers couldn't afford the 1000 more test subjects it would take to get another sigma. Which isn't great, but strictly speaking its not doing society a disservice, either- even uncertain knowledge decreases the entropy of our vast ignorance. It does, however, add an important dimension to how these studies should be interpreted- that small chance that any given study is incorrect matters, particularly when there's such a mind-bogglingly huge amount of research being done in the modern day. Its entirely unsurprising to read about a study claiming with p < 0.01 that a glass of wine a day will make your hair fall out, when you consider that there's been many thousands of papers published about things like wine. Confidence- and p-values alone aren't going to save us there.

aaron695 11y ago
Yes, I see you're correct. Most studies would chose a 95% confidence interval so their statement stands.

vacri 11y ago

The article gives a clear example of why it is difficult to replicate psychology-based studies. Iron is iron everywhere, and behaves similarly. Put iron at STP, and it will behave the same in Massachusetts and Amsterdam. The same is not true of humans - the study they give as an example is "students opinions of affirmative action policies", a clearly political set of questions, taken of people who are not only not in the same political environment, but speak different native languages! Not being able to reproduce a political questionnaire done of Harvard students on U of A students should surprise exactly zero people, and should never have been used for verifying reproducibility.

harry8 11y ago
I'm not particularly interested in Harvard students. I'm pretty sure they're not super interesting to anyone outside of the Harvard diaspora and their friends. I'll bet 100% the paper was generalising from Harvard students to some wider population, or why would you publish it. If the paper says "this thing we're publishing is only applies to Harvard students and has zero wider significance." I'll eat humble pie and apologise. Psychology has a big issue of claiming things as fact that are no such thing. It has had this issue since forever. The history of psychology research is truly ethically hideous both in terms of what it did to people in experiments and what it claimed to have found that was then used to maim people in the general population. Research lobotomy if you have any doubts at all about that statement. Some fraud got a Nobel prize for research that he used to advocate "safe lobotomy with no adverse affects." (But maybe he fooled himself and it wasn't blatant fraud. Bah.) These are the screams from the psychology establishment as they get called out for claiming so much that is just totally false. It's really hard to know something. There is an immense amount of work you have to do and you all you might find after decades is "dead end". Not in Psych research papers and they absolutely deserve the contempt they get for it.
- vacri 11y ago
  > The history of psychology research ... Research lobotomy... So, you've pretty much publicly announced that you're heavily clueless about what psychology is, and are buying into that armchair critic opinion that it's a discredited arm of psychiatry. Discussions on HN constantly revolve around psychology. How to negotiate for a better salary? Psychology. A/B testing? Psychology. How to give or take an interview effectively? Psychology. UX design and testing? Psychology. How to motivate yourself or others, or avoid procrastination? Psychology. Game theory? Psychology. And this is just the common discussions on the rather niche forum of HN. As I write this comment, the #1 article is "Dsxyliea", which is... psychology. When you abuse psychology because of a brief, discredited and niche episode in its history, you're saying the equivalent of "web browsers are shit software, because IE on XP can't do SNI".