13 ms·

Is Psychology About to Come Undone?

211 points by yread 15y ago

alttag 15y ago
The challenge of publishing "novel" work exists in many fields, not just psychology, and the debate about potential false positives is ongoing. It's great to see this reproducibility work being done. I'd like to see it in my field.
deleted 15y ago
[deleted]
bane 15y ago
It seems that psychology has always been about the atypical rather than the typical. In most other sciences, the goal is to hopefully build a useful model of the typical, and observations outside that model result in alterations to the model. In psychology the idea seems to be to produce the inverse of that, yet there doesn't appear to be any particular end goal of producing the typical model, only in defining the atypical cases to the nth-degree. The result? We have no better idea what the typical mind is like than we did a hundred years ago, but we have exhaustive lists of subtly different atypical models so encompassing that almost anybody could be recognized as having a psychological problem of some sort! Describing a new disease (atypical model) is one of the only ways to get recognized in the field. But all the easy cases have been taken, so bizarre models seem to make the publishing rounds much more readily than new subtle delineations on previously recognized diseases. In treatment this turns into quack and fad medicine like "I prescribe shared strip club night for marriage problems, with the idea that it forces couples to talk to each other about their sexual problems blah blah blah" or "Anger therapy" or "Primal Therapy" or other such nonsense. I'd say that with the tools we have available today, that psychology is due for an Einstein level revolution, but I'm not sure that the field, internally, is ready for this.
- radu_floricica 15y ago
  It happened already 50 years ago, it just doesn't get any press. Pretty much everything I've been reading in the past few years (Kahneman, Baumeister, Baron, Stanovich, Wilson for example) will pass this with flying colors. It is true however that psychology requires a bit more intellectual honesty then other fields. Mistakes are more subtle and easier to cover up, samples are smaller and there's always the excuse of different cultures. I'm really curious what this project will uncover.
  - bane 15y ago
    I hope so, I've seen some work on building typical models of the mind with averages of cat scans etc. "and there's always the excuse of different cultures" That's a good point. Related to this are the fascinating Culture-bound syndromes. http://en.wikipedia.org/wiki/Culture-bound_syndrome http://en.wikipedia.org/wiki/Culture-bound_syndrome
  - neilk 15y ago
    Kahneman's getting a little more press lately, with a bestseller on the shelves. His story is interesting - he was given a lot of responsibility at a young age, because he was doing officer evaluation for the newly-formed Israeli army. He has to produce real, repeatable results, and has a wealth of data to test his earlier conclusions, and perhaps the sheer newness of the entire enterprise makes it safer for him to be honest. In a few years he realizes the vacuity of his own psychological results, and by implication, many of the methods he's been taught. That's what kicks off his whole program of investigating sources of bias and error. Would this have ever happened in a standard academic setting?
    - radu_floricica 15y ago
      Quite possibly. Roy Baumeister has possibly the best thinking process I've ever met. He had the gall to approach a subject like this, and do it with flawless rigor: http://www.amazon.com/Meanings-Life-Roy-F-Baumeister/dp/0898625319 http://www.amazon.com/Meanings-Life-Roy-F-Baumeister/dp/0898... Too bad his latest book (Willpower) is coauthored with a professional writer... I miss his style.
- DanBC 15y ago
  > that psychology is due for an Einstein level revolution The Behaviour Therapy model was that revolution. The old model had many sessions trying to uncover past trauma before allowing the patient to move on. People with a phobia of something spent many weeks talking about their childhood and their adult life. Now a severe phobia can be cured for most people in about an hour. And it stays gone. I'm glad that psychology studies are being given more rigorous scrutiny. There's been too much sloppy science in the field for much too long.
  - evilduck 15y ago
    Now a severe phobia can be cured for most people in about an hour. And it stays gone. Details or a citation? I'm genuinely curious.
    - mattgreenrocks 15y ago
      Same here. I'd love to read about this.
      - disgruntledphd2 15y ago
        It appears that he is referring to cognitive-behavioural therapy, or CBT. The Wikipedia page is pretty good. That being said, it works very well for phobias, but in other areas its about as good as all the other methods of therapy, which implies that its much more down to the meanings that a patient takes from the experience rather than the treatment itself.
      - DanBC 15y ago
        I am talking about "Systemic Desensitization" as developed by Wolpe. That helped form CBT. The evidence base for SD and for CBT is pretty good. (It's a good treatment for depression, if applied by a skilled practitioner.) Here's one link to a BBC Radio 4 programme about treatment of phobia using SD. (Might not be available outside UK, but there's probably some way of getting it.) (http://www.bbc.co.uk/programmes/b01g5yy1 http://www.bbc.co.uk/programmes/b01g5yy1)
- pron 15y ago
  You're referring to clinical psychology. However, most psychological research (particularly that published in the journals chosen for review) does try to build a model of the typical mind.
  - bane 15y ago
    Too right, I should have made that distinction.
- wch 15y ago
  Psychology != Clinical Psychology, which is what you're referring to. None of the journals in this project are clinical psych journals.
- goldmab 15y ago
  We have no better idea what the typical mind is like than we did a hundred years ago I can't take this statement seriously. Have you ever even taken an intro cognition or perception class? Almost everything is about how minds typically work, and all of the findings are less than one hundred years old.
  - bane 15y ago
    Many years ago. Perhaps the field has changed or I had a lousy course. But most of it seemed to be about describing the edge cases where our brains produce false models or find wrong patterns in the world.
    - goldmab 15y ago
      Of course brains produce false models and find wrong patterns. Modeling and pattern-matching is what brains do. So they're going to get it wrong a lot. The interesting thing isn't how funny it is that the mind screws up sometimes, it's knowing the actual mechanisms of perception, cognition, memory, behavior, etc. Edge cases and failures are only the beginning of understanding, because they hint at how things work. For example, visual perception is heavily based on detecting edges. So there are a set of optical illusions where you fail to accurately perceive the colors or shades of different areas (like the chessboard illusion), because the relative shading of adjacent areas is more important for producing edges and shapes in your mind.
jerf 15y ago
If psychology is "under attack", or for that matter any scientific field is under attack, the solution is to do the hard work to prove that your work is meaningful and accurate, and if you discover it isn't, to fix those problems. "People" are gullible and easy to fool in the short term, but I think it is commonly underestimated how smart "people" can be in the long term. Yes, you could throw up a smokescreen and dodge out of the spotlight cast on your bad science today, but that will only be a momentary reprieve of the pain. In the long term you'll still be under attack, and given that you will have been witnessed using smokescreens and obfuscations, you'll probably be on the losing side of that attack. In the short term, the pain of revealing just how much flimflam there is might hurt, but the result will be a discipline that in 5 or 10 years is no longer under serious attack, because "people" will notice that honesty and react to it. If science is under attack, it is only because "people" are noticing that a lot of it is bunk... and the problem is people are right. We've seen that in a number of studies in a number of fields lately. The only answer that's going to truly restore confidence and respect is to eliminate the bunk. Politics won't work. Pay the piper now, or pay the piper more later. How often those are the choices.... and of all the people who should understand the psychological appeal of paying the piper later and taking the easy road today, you'd think it would be the psychologists...
- roc 15y ago
  I take a deep exception to the idea that a group of scientists attempting to reproduce the published work of their peers is, in any way, "an attack". Casting things that way immediately exposes a speaker as either not understanding science, and/or not having good science as their primary goal. Reproducing results is science. Perhaps the most important kind.
  - jerf 15y ago
    That is a reaction to the statement from the original article: "Nosek told Science that a senior colleague warned him not to take this on “because psychology is under threat and this could make us look bad.” In a Google discussion group, one of the researchers involved in the project wrote that it was important to stay “on message” and portray the effort to the news media as “protecting our science, not tearing it down.”" I see they say threat in that text, not attack, which appears elsewhere. Might I point out that if you carefully read my post it is vehemently agreeing that replication is necessary, and from there it should seem logical that I would not consider something necessary an "attack".
pron 15y ago
Small sample sizes and publication bias are a lethal combination in any field. Suppose scientists wanted to test the hypothesis that a fair coin always comes up heads. Due to budgetary issues, scientists are only able to toss the coin 5 times. 40 groups of researchers conduct the experiment in universities around the world. One of them is quite likely to get 5 heads. Guess which result is likely to be published (or which group is going to even attempt to publish its result). Moreover, this result is statistically significant according to the well-accepted peer-reviewed journal standard of p < 0.05. I suggest people read John Ioannidis's paper: Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...), and the Atlantic story about Ioannidis and his work: Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...)
- Alex3917 15y ago
  Here is a more complete reading list: Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/print/2010/11/lies-damne...) The Truth Wears Off (http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...) Grant System Leads Cancer Researchers to Play It Safe (http://www.nytimes.com/2009/06/28/health/research/28cancer.html?pagewanted=all http://www.nytimes.com/2009/06/28/health/research/28cancer.h...) Placebos Are Getting More Effective. Drugmakers Are Desperate to Know Why. (http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo_effect?currentPage=all http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo...) The Cult of Pharmacology: How America Became the World's Most Troubled Drug Culture (http://www.erowid.org/library/review/review.php?p=269 http://www.erowid.org/library/review/review.php?p=269) The Mouse Trap: The dangers of using one lab animal to study every disease (http://www.slate.com/articles/health_and_science/the_mouse_trap/2011/11/lab_mice_are_they_limiting_our_understanding_of_human_disease_.html http://www.slate.com/articles/health_and_science/the_mouse_t...) Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...) Mistakes in Scientific Studies Surge (http://online.wsj.com/article/SB10001424052702303627104576411850666582080.html http://online.wsj.com/article/SB1000142405270230362710457641...) Trials and Errors: Why Science Is Failing Us (http://www.wired.com/magazine/2011/12/ff_causation/all/1 http://www.wired.com/magazine/2011/12/ff_causation/all/1) In cancer science, many "discoveries" don't hold up (http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-...) Lullaby for a Mouse: Anthropological Observations of an Animal Model of Psychosis (http://vimeo.com/33803995 http://vimeo.com/33803995) Inside the Battle to Define Mental Illness (http://www.wired.com/magazine/2010/12/ff_dsmv/all/1 http://www.wired.com/magazine/2010/12/ff_dsmv/all/1) There are of course quite a few relevant books as well, but that's probably enough to keep most people busy for a few days. Also, most of these articles are about drug research, medicine, or psychology, but it applies to basically every field.
Alex3917 15y ago
"Yet when Stuart Ritchie, a doctoral student in psychology at the University of Edinburgh, and two colleagues failed to replicate his findings, they had a heck of a time getting the results into print" Maybe because they were a bunch of hacks. They didn't actually replicate Bem's methodology, so whether or not they got the same results is irrelevant. The journals were right for rejecting their work.
- Estragon 15y ago
  Got a cite to back that up? Because Ritchie claims that no serious methodological issues were raised during the review process, other than the ridiculous "You can't get positive results from an ESP test unless you believe in it."
  - Alex3917 15y ago
    I don't remember the exact article that discussed it, but I'm sure you could find it by searching HN. Other than the issue you mentioned, the other issue was that the original study was done in person using Ivy league students, whereas the replication was done over the Internet using a non-comparable demographic. Regardless of whether or not one thinks it's ridiculous that these issues could have any effect, it's intellectually dishonest for them to say that they've replicated Bem's methodology and failed to reproduce the results when in fact they haven't actually done so. (The researcher-belief issue matters since we already know that the how well a drug work depends on how much the person administering the drug believes it will work, which is why for well-designed drug trials the investigators aren't the ones who administer the drug. So it's not like the replication was being dinged for not subscribing to some exotic new methodology, rather they were being shot down for not following existing best practices. That said, I have no idea whether there were videos how much the researcher's beliefs would have actually been apparent, so I don't really know whether or not it was reasonable for the journal to reject it based on this one point.)
    - gergles 15y ago
      > we already know that the how well a drug work depends on how much the person administering the drug believes it will work hahahahahahaahahahahahahahahah what? No, no we do not. If you have a citation that suggests otherwise, I would be thrilled to read it. The reason that we blind is because there ARE effects based on perception if the person administering the drug knows its experimental. There have been studies indicating that the administrator would say things like "this really should work", for example. That invokes the placebo effect response in the patients (which is also well documented,) but the concept that "we know how well a drug works depends on how much the person administering it believes in it" is completely incorrect.
BenoitEssiambre 15y ago
God I hope this becomes a trend in other fields as well. Spending time in universities has made me very cynical of the research that comes out of them. There is just too much incentives for profs to ignore biases in their research. I've seen it happen many times. For them, it is the difference between becoming a prof or a lab tech and it is literally worth millions. You can not trust any research done where the career of the researcher depends on them finding results. Universities, by hiring based on research credentials that currently translate roughly in the amount of positive results a person has generated, completely render worthless the research going on in their departments. Aggravating the situation is the fact that the peer review system is too incestuous to be relied on especially when the peers are probably also 'bias ignorers' with incentives to keep the flaws in the system. I strongly believe that for hiring purposes, the research skills of profs should be evaluated on criterion that are incidental to research. Mostly math, probability, statistics (yes even for psychology) then methodology skills and also maybe leadership, communication skills and dedication to science (last only because it is difficult to measure). update:typos
- viggity 15y ago
  "I take the hard view that science involves the creation of testable hypotheses" - Michael Crichton I'm not trying to be a troll. Really, I promise. But I really hope that the next field they do this in, is in Climate Science, specifically articles on climate change. It seems like bulk of their work is based on computer models and I just don't know how you're supposed to replicate any of that. How do you isolate the millions of variables to show cause and effect with a closed source computer model. That is my biggest problem with climate science.
  - roc 15y ago
    > "It seems like bulk of their work is based on computer models" Then it seems that you are confused. Climate science is about gathering data, putting forth a testable hypothesis and seeing how future data fits the prediction. Yes, they use computer models. In the same way that you or I might gather data from a wind tunnel test and make a computer model of wind flow from that data to extrapolate a theory and create new testable hypotheses of a more-aerodynamic shape. But so long as new data is used to test those hypotheses, the use of a computer model is irrelevant. Science is being done. Testable hypotheses are being created and new data is being compared to those predictions. As long as we aren't running around half-cocked, believing the predictions of hypotheses that have not yet been tested, there's no concern. And the core hypotheses under the umbrella of Climate science have been tested. Some disproved by data, leading to changes in thinking, further research and new testable theories. Some supported by data. None of the 'alternate' hypotheses can say the same. (No, it's not the solar cycle. No, it's not distorted temperature readings from data stations in urban areas.) The very reason that there is a general consensus around the so called 'tent-pole' theories in Climate Science is precisely because they have been tested and have been shown to make accurate predictions and continue to do so. Some of them, for decades now.
6ren 15y ago
reminds me of Feynman's cargo cult science's rat mazes http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm
amcintyre 15y ago
"If you’re a psychologist, the news has to make you a little nervous—particularly if you’re a psychologist who published an article in 2008 in any of these three journals...<snip>...Because, if you did, someone is going to check your work.' Can somebody in the field comment on this statement? Is it really so out of the ordinary to attempt to reproduce work in psychology that this kind of statement is warranted?
- luriel 15y ago
  Yea, I would think that any sane and honest scientist would be delighted that somebody else is going to test their work. But after things like this: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... I'm starting to wonder how many scientists are really more dedicated to search for the truth than search for grants and tenure.
  - eli_gottlieb 15y ago
    I'm starting to wonder how many scientists are really more dedicated to search for the truth than search for grants and tenure. Scientists would love to do proper science. Just please, please, give us a way to not starve from doing it!
- nchlswu 15y ago
  I only have undergrad research experience. In general, I haven't really seen focus on replicating previous results - but it depends on the work and the researcher. Most of what I've seen is a researchers who extend their own work. in doing so, they replicate their previous research and reveal other information. In this case, replication is definitely a goal, but it's focused on a researcher's own work and a byproduct of exploring a particular research problem. I've witnessed people discussing replications and discussing that they were unable to replicate X's results. Here, there were clear are efforts made to replicate results, but I've only witnessed this in an informal setting, so the results were never published. I've never seen a big push to publish results that discredited a result due to an inability to replicate results I suspect that there is little effort made after the fact unless the theory was particularly 'significant,' or if a researcher can offer a competing account. The one time I learned about replication was when two competing researchers were offering competing accounts and discrediting anothers' results. However, in these cases, there were flaws in each others' research Obviously, in a classroom setting, we have seen many cases were replication has occurred, which makes a particular theory very strong. It is rare that the published results are straight-replications, though. They are often extensions.
hluska 15y ago
I've always had problems with fields like psychology. Part of the field revolves around hard science - neuroscience researchers (as one example) use tools like fMRIs, etc to capture observable phenomenon. On the other hand, there is the art of psychology - social/developmental psychologists use dubious tools like surveys and interviews to try and prove a thesis. I think that the average Hacker News reader could write a survey that would 'prove' that gravity doesn't exist. What kind of experimental integrity do surveys have? And then there are participant pools. These pools are primarily composed of undergraduates (who often earn bonus marks/money for participating). Add in a strong publish or perish mentality and you can see some serious problems. A whole lot of researchers are using dubious methods on a set of participants that do not adequately reflect humanity as a whole.
- riffraff 15y ago
  Add in that a lot of the psychology tradition is based on, let's say, "less than strictly scientific" traditions and personality cult.
- disgruntledphd2 15y ago
  There are some issues I (as a psychologist) would have with your assumptions. Firstly, fMRI and neuroscience research is probably the single biggest source of errors and shoddy research in psychology. Its a combination of really small sample sizes, poor statistical tests (brain regions are independent of one another, really?) and huge amounts of data dredging to find significance. See for a roundup: http://escholarship.org/uc/item/51d4r5tn http://escholarship.org/uc/item/51d4r5tn Interestingly enough, most of the observable pheonomena you note in neuroscience are linked to traits that people think about (optimism, personality etc) through surveys, so even if their methods were perfect, the results still wouldn't be. On your survey point, I would agree that the participant pools are quite limited and generalizability is quite low, but your example is ludicrous. No one does surveys of gravity in psychology, surveys are carried out to investigate the manner in which people conceptualise their experience. For a really great roundup of the problems with typical social science participant pools, see: Heinrich et al: humancond.org/_media/papers/weirdest_people.pdf Indeed, the entire issue of BBS that that article appears in is well worth a look for a deeper understanding of these problems. These surveys should then be calibrated against behavioural outcomes, but this does not happen often enough, which is a major issue in my view. Thats the major problem with surveys, well that and some poor methods accepted too uncritically within the field (factor analysis). To summate, psychology has many, many problems and I fully support this reproducibility effort (and it will expose a lot of findings as non-replicable). But don't single out surveys for derision, neuroscience deserves as much if not more of your scorn.
iandanforth 15y ago
In 2003 I did the work that lead to this paper (http://bit.ly/I1qvuI http://bit.ly/I1qvuI). Could it be replicated? Probably not. Let me outline a few challenges that don't have to do with a result being true or not. 1. The passage of time Unlike physics and biology, culture has a huge impact on the field of psychology, and cultures change. Additionally, my work was about addiction to online games. I have no idea if the same type of people exist today that did then. I do know that the game I was studying (Asheron's Call) does not exist in anywhere near the same form as it did then. 2. Copyright While I was lucky enough to find psychological measures (aka tests) that were in the public domain, most are not. To replicate a result you may have to pay to access the tests and then most likely cannot re-publish them. At the very least this makes replicating work inconvenient and expensive. 3. Data Because psychology is largely driven by statistics, to replicate a result you should really start by re-analyzing the data. The reality? This data simply isn't available. It is not published alongside the results, and I don't have it anymore. I might be able to find the cleaned data, but that's not important. Error and bias can just as easily be introduced during data cleaning. For example I removed many data points that appeared as outliers, but perhaps these should have been included. No one will ever know. I hope barriers such as these are addressed in this investigation. It would be disappointing to impune the work of scientists when it is the process that could really use reform.
- k4st 15y ago
  Note: I am not a psychologist. What do you mean by "some tests are copyrighted"? Are you talking about a statistical method being copyrighted, an implementation thereof, or something else entirely?
  - ttt_ 15y ago
    He's refering to this (example): https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limitations https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limi... The BDI is copyrighted, a fee must be paid for each copy used, and photocopying it is a violation of copyright.
    - 15y ago
developer9 15y ago
I agree with others that this is a good thing. Science is only proven until it is disproven. For this reason, I think we need to hold other complex sciences like meteorology to a higher standard. I want to see how every prediction changes over time and how wrong it was per measurement later. If we had a map of places whose predictions were more wrong than others and this were publicized, there would be more impetus to fix these issues and improve the science. On the other hand, people need to take fringe science and religion more seriously and stop trying to debunk it. We can learn about this universe from every belief system.
jamaicahest 15y ago
I personally hope the software world picks this up, so we might get some actual data on things like TDD, agile, waterfall, instead of peoples' anectdotes and experiences on past projects and whatever Martin Fowler thinks up (no offense, I love Fowler, but he would benefit from proving some of his assumptions)
- mangodrunk 15y ago
  I agree. It seems we have a problem where things that are most likely just opinion or preference are treated as fact and to worsen the situation there is a disregard for evidence or the need of it. For example, take the argument why indexes should start with zero. [0] Some people will claim that it's easier to learn without providing evidence or that those who use such a system produce less bugs, again without evidence. [0] http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EW...
nradov 15y ago
A cynic would say that psychology can't come undone since it's never really been "done" in the first place. We know so little about the fundamental nature of human thought that the whole field is basically a joke. Their journal articles are built on a foundation of sand. I expect that 200 years in the future historians will look back at 2012 psychology as a pseudoscience, little better than phrenology.
montecarl 15y ago
I am glad someone is doing this. I recently tried to implement an algorithm described in a paper in a well known chemistry journal and discovered that the paper's results were incorrect. It turns out that the author's program (which was not made available with the paper) didn't correspond with what was described in the paper. That is to say, when I used the described algorithm, I got very different results than those in the paper. Now I'm not sure what to do. I want to correct the paper, but I don't want to openly criticize the work of others.
- corford 15y ago
  If you are prepared to back your claim up in a way that can be independently verified, why don't you want to criticize the original authors work? Surely correct science is more important than sparing someone's blushes?
  - montecarl 15y ago
    After reading your comment, I can't come up with any reason other than the blushes. Perhaps I'll reconsider.
    - otoburb 15y ago
      How about reaching out to the original author and outlining the problems that you've been encountering? This way, you can at least give them a chance to respond privately or correct potential misunderstandings with the paper (perhaps publicly). Similar analogy would be approaching a vendor regarding a security/vulnerability disclosure.
      - montecarl 15y ago
        I already have. They recently gave me the source code and I pointed out the problems. They agree that there are bugs in their code. I just haven't approached them about publishing a comment on their paper because I feel bad about it.
- nchlswu 15y ago
radarsat1 15y ago
It's funny because, coming from an "interdisciplinary" research area, many of us cower in fear of psychology journals, completely daunted by the rigorous level of (perceived?) statistical knowledge required to be accepted. Psychology is viewed by many as one of the most hard-core sciences, just because it has taken such a hard-line attitude toward statistical work, out of shear necessity, due to past problems arising from a history of more philosophical approaches to the subject. Other sciences (e.g. HCI) are often somehow "softer" just because they take pains to avoid the kind of criticism that can come from the kind of complex statistical interpretations we see in pure psychology journals. Although, I suppose I could be mixing up psychology with what we call "psychometrics", which is the kind of psychology that I'm more familiar with due to my research area, which involves perception of virtual reality. By the way I agree with others that this is not an "attack" on psychology, it is simply science. Verification of results can invalidate claims, but it can also easily provide further evidence for claims. Nothing bad can come of this initiative, if it's carried out properly.
- disgruntledphd2 15y ago
  You are definitely mixing up psychology with psychometrics. Psychometrics is awesomely thorough and aware of statistical models and the limits thereof. Psychology, on the other hand relies on SPSS and what other papers have done to determine their methods.
Estebon 15y ago
I have found Hacker New to be full of some of the smartest people I've encountered on the internet. However, every time a discussion pops up where a social or soft science is involved, the discussion becomes mired by arrogance, bias, and general small mindedness about what constitutes "real science." I see comment after comment of where the author implies or outright says that psychology is pseudoscience or close to it. The challenge in designing, implement, and supporting a study in psychology or similar science is staggering. That's no excuse for questionable statistics interpretation or outright manipulations, but to write off an entire field of study because it doesn't have the convenient quantitative measuring capabilities that other sciences have is ridiculous. I have noticed a pattern as of late, or maybe its just the articles I have been reading on HN, but there seems to be a lot more people making dubious claims, commenting on things they no nothing about, and just genuinely turning this place into a cesspool like the rest of the internet. What is so hard about not posting an opinion on subjects that you don't have knowledge of or any proof to back up your claims? If you haven't studied psychology, have even a vague idea of what it's about, or what its past or present state is then why muddle up the conversation with you BS conjecture? I'm sure a legitimate licensed clinician would have a field day with some of the posters on this site.
- disgruntledphd2 15y ago
  While I agree with much of the substance of your post, the fact remains that psychology is a mess. A friend of mine once told me that doing a doctorate in a subject only really gave you the ability to see how your field is wrong and all the problems with it, and I would agree. The problems with psychology, from someone who's been at it for a while: 1) lack of focus on replication 2) in survey studies, failure to correlate psychometric measures with behaviour or other kinds of measures (surveys versus reaction time measures versus physiological measures). 3) An unfortunate lack of understanding of the assumptions behind the statistical procedures used routinely within the field. 4) Misplaced emphasis on theory at the expense of prediction. A relatively well known psychologist, quite statistical aware, posted on stats.stackexchange.com that the goal of psychology was theoretical understanding, not prediction. For the life of me, I can't see how one can develop good theories without prediction, but it appears to be a dirty word within much of psychology. That being said, people are hard. They change their behaviour based on what they think you are trying to do, they tell you what you want to hear, and even when they tell the truth as they see it, they may well be mistaken. So while its not right to write off an entire field for some errors (in fact, my results are all perfect) its also useless to deny the problem and pull a Freud by saying that everyone who disagrees with your methods has some kind of psychological disorder.
deleted 15y ago
[deleted]
pchivers 15y ago
Richard Feynman on reproducibility in science: "When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens." (from "Cargo Cult Science")
- nerdfiles 15y ago
  http://webjournal.nerdfiles.net/2012/04/03/this/ http://webjournal.nerdfiles.net/2012/04/03/this/ Notice the date. That was the date. What is interesting is that they are engaging in an activity that wishes to avoid reproducibility. They allow "facts" and "ideas" to swap. Articles like http://ir.canterbury.ac.nz/handle/10092/5828 http://ir.canterbury.ac.nz/handle/10092/5828 exist. There's the Philosophical Foundations of Neuroscience. All that theorizing needs to record facts. To do that, we need to be able to follow what the hell is going on without blatant wankery like with Anthony Crick or early John Searle's "foot in the mind" rubbish or Dennett's everything's-a-scientist metaphysics. Philosophers got the descriptivist bug with Experimental Philosophy. Linguistics has always been largely descriptivist. Now psychology is to do the same. Who cares that they may have false theories, let's see if they've described anything. As Fraiser obviously demonstrates, psychologists have had their Golden Age. Psychology needs management.
tokenadult 15y ago
Free full text of the 1991 article by David Lykken, "What's wrong with Psychology, anyway?," can be downloaded from an online reference link. http://cogprints.org/371/ http://cogprints.org/371/ The article shows how many of these issues have long been on the radar screen of psychologists at the minority of universities (for example, the University of Minnesota) where the psychology departments train graduate students in the general scientific method. There are some very amusing, thought-provoking, and shocking examples in the article.
ams6110 15y ago
It would not surprise me in the slightest to find that a large amount of published research findings are pure bullshit.
rsanchez1 15y ago
Sociologists everywhere are panicking. On the other hand, they'll probably expose a whole bunch of "decided" research fields that new psychologists can revisit.
Mohnican 15y ago
What an over sensationalized headline. I realize it's the same as the source but I thought HN was better than this.

211 points by yread 15y ago

The challenge of publishing "novel" work exists in many fields, not just psychology, and the debate about potential false positives is ongoing. It's great to see this reproducibility work being done. I'd like to see it in my field.

deleted 15y ago

[deleted]

bane 15y ago

It seems that psychology has always been about the atypical rather than the typical. In most other sciences, the goal is to hopefully build a useful model of the typical, and observations outside that model result in alterations to the model. In psychology the idea seems to be to produce the inverse of that, yet there doesn't appear to be any particular end goal of producing the typical model, only in defining the atypical cases to the nth-degree. The result? We have no better idea what the typical mind is like than we did a hundred years ago, but we have exhaustive lists of subtly different atypical models so encompassing that almost anybody could be recognized as having a psychological problem of some sort! Describing a new disease (atypical model) is one of the only ways to get recognized in the field. But all the easy cases have been taken, so bizarre models seem to make the publishing rounds much more readily than new subtle delineations on previously recognized diseases. In treatment this turns into quack and fad medicine like "I prescribe shared strip club night for marriage problems, with the idea that it forces couples to talk to each other about their sexual problems blah blah blah" or "Anger therapy" or "Primal Therapy" or other such nonsense. I'd say that with the tools we have available today, that psychology is due for an Einstein level revolution, but I'm not sure that the field, internally, is ready for this.

radu_floricica 15y ago
It happened already 50 years ago, it just doesn't get any press. Pretty much everything I've been reading in the past few years (Kahneman, Baumeister, Baron, Stanovich, Wilson for example) will pass this with flying colors. It is true however that psychology requires a bit more intellectual honesty then other fields. Mistakes are more subtle and easier to cover up, samples are smaller and there's always the excuse of different cultures. I'm really curious what this project will uncover.
- bane 15y ago
  I hope so, I've seen some work on building typical models of the mind with averages of cat scans etc. "and there's always the excuse of different cultures" That's a good point. Related to this are the fascinating Culture-bound syndromes. http://en.wikipedia.org/wiki/Culture-bound_syndrome http://en.wikipedia.org/wiki/Culture-bound_syndrome
- neilk 15y ago
  Kahneman's getting a little more press lately, with a bestseller on the shelves. His story is interesting - he was given a lot of responsibility at a young age, because he was doing officer evaluation for the newly-formed Israeli army. He has to produce real, repeatable results, and has a wealth of data to test his earlier conclusions, and perhaps the sheer newness of the entire enterprise makes it safer for him to be honest. In a few years he realizes the vacuity of his own psychological results, and by implication, many of the methods he's been taught. That's what kicks off his whole program of investigating sources of bias and error. Would this have ever happened in a standard academic setting?
  - radu_floricica 15y ago
    Quite possibly. Roy Baumeister has possibly the best thinking process I've ever met. He had the gall to approach a subject like this, and do it with flawless rigor: http://www.amazon.com/Meanings-Life-Roy-F-Baumeister/dp/0898625319 http://www.amazon.com/Meanings-Life-Roy-F-Baumeister/dp/0898... Too bad his latest book (Willpower) is coauthored with a professional writer... I miss his style.
DanBC 15y ago
> that psychology is due for an Einstein level revolution The Behaviour Therapy model was that revolution. The old model had many sessions trying to uncover past trauma before allowing the patient to move on. People with a phobia of something spent many weeks talking about their childhood and their adult life. Now a severe phobia can be cured for most people in about an hour. And it stays gone. I'm glad that psychology studies are being given more rigorous scrutiny. There's been too much sloppy science in the field for much too long.
- evilduck 15y ago
  Now a severe phobia can be cured for most people in about an hour. And it stays gone. Details or a citation? I'm genuinely curious.
  - mattgreenrocks 15y ago
    Same here. I'd love to read about this.
    - disgruntledphd2 15y ago
      It appears that he is referring to cognitive-behavioural therapy, or CBT. The Wikipedia page is pretty good. That being said, it works very well for phobias, but in other areas its about as good as all the other methods of therapy, which implies that its much more down to the meanings that a patient takes from the experience rather than the treatment itself.
    - DanBC 15y ago
      I am talking about "Systemic Desensitization" as developed by Wolpe. That helped form CBT. The evidence base for SD and for CBT is pretty good. (It's a good treatment for depression, if applied by a skilled practitioner.) Here's one link to a BBC Radio 4 programme about treatment of phobia using SD. (Might not be available outside UK, but there's probably some way of getting it.) (http://www.bbc.co.uk/programmes/b01g5yy1 http://www.bbc.co.uk/programmes/b01g5yy1)
pron 15y ago
You're referring to clinical psychology. However, most psychological research (particularly that published in the journals chosen for review) does try to build a model of the typical mind.
- bane 15y ago
  Too right, I should have made that distinction.
wch 15y ago
Psychology != Clinical Psychology, which is what you're referring to. None of the journals in this project are clinical psych journals.
goldmab 15y ago
We have no better idea what the typical mind is like than we did a hundred years ago I can't take this statement seriously. Have you ever even taken an intro cognition or perception class? Almost everything is about how minds typically work, and all of the findings are less than one hundred years old.
- bane 15y ago
  Many years ago. Perhaps the field has changed or I had a lousy course. But most of it seemed to be about describing the edge cases where our brains produce false models or find wrong patterns in the world.
  - goldmab 15y ago
    Of course brains produce false models and find wrong patterns. Modeling and pattern-matching is what brains do. So they're going to get it wrong a lot. The interesting thing isn't how funny it is that the mind screws up sometimes, it's knowing the actual mechanisms of perception, cognition, memory, behavior, etc. Edge cases and failures are only the beginning of understanding, because they hint at how things work. For example, visual perception is heavily based on detecting edges. So there are a set of optical illusions where you fail to accurately perceive the colors or shades of different areas (like the chessboard illusion), because the relative shading of adjacent areas is more important for producing edges and shapes in your mind.

jerf 15y ago

If psychology is "under attack", or for that matter any scientific field is under attack, the solution is to do the hard work to prove that your work is meaningful and accurate, and if you discover it isn't, to fix those problems. "People" are gullible and easy to fool in the short term, but I think it is commonly underestimated how smart "people" can be in the long term. Yes, you could throw up a smokescreen and dodge out of the spotlight cast on your bad science today, but that will only be a momentary reprieve of the pain. In the long term you'll still be under attack, and given that you will have been witnessed using smokescreens and obfuscations, you'll probably be on the losing side of that attack. In the short term, the pain of revealing just how much flimflam there is might hurt, but the result will be a discipline that in 5 or 10 years is no longer under serious attack, because "people" will notice that honesty and react to it. If science is under attack, it is only because "people" are noticing that a lot of it is bunk... and the problem is people are right. We've seen that in a number of studies in a number of fields lately. The only answer that's going to truly restore confidence and respect is to eliminate the bunk. Politics won't work. Pay the piper now, or pay the piper more later. How often those are the choices.... and of all the people who should understand the psychological appeal of paying the piper later and taking the easy road today, you'd think it would be the psychologists...

roc 15y ago
I take a deep exception to the idea that a group of scientists attempting to reproduce the published work of their peers is, in any way, "an attack". Casting things that way immediately exposes a speaker as either not understanding science, and/or not having good science as their primary goal. Reproducing results is science. Perhaps the most important kind.
- jerf 15y ago
  That is a reaction to the statement from the original article: "Nosek told Science that a senior colleague warned him not to take this on “because psychology is under threat and this could make us look bad.” In a Google discussion group, one of the researchers involved in the project wrote that it was important to stay “on message” and portray the effort to the news media as “protecting our science, not tearing it down.”" I see they say threat in that text, not attack, which appears elsewhere. Might I point out that if you carefully read my post it is vehemently agreeing that replication is necessary, and from there it should seem logical that I would not consider something necessary an "attack".

pron 15y ago

Small sample sizes and publication bias are a lethal combination in any field. Suppose scientists wanted to test the hypothesis that a fair coin always comes up heads. Due to budgetary issues, scientists are only able to toss the coin 5 times. 40 groups of researchers conduct the experiment in universities around the world. One of them is quite likely to get 5 heads. Guess which result is likely to be published (or which group is going to even attempt to publish its result). Moreover, this result is statistically significant according to the well-accepted peer-reviewed journal standard of p < 0.05. I suggest people read John Ioannidis's paper: Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...), and the Atlantic story about Ioannidis and his work: Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...)

Alex3917 15y ago
Here is a more complete reading list: Lies, Damned Lies, and Medical Science (http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-and-medical-science/8269/ http://www.theatlantic.com/magazine/print/2010/11/lies-damne...) The Truth Wears Off (http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...) Grant System Leads Cancer Researchers to Play It Safe (http://www.nytimes.com/2009/06/28/health/research/28cancer.html?pagewanted=all http://www.nytimes.com/2009/06/28/health/research/28cancer.h...) Placebos Are Getting More Effective. Drugmakers Are Desperate to Know Why. (http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo_effect?currentPage=all http://www.wired.com/medtech/drugs/magazine/17-09/ff_placebo...) The Cult of Pharmacology: How America Became the World's Most Troubled Drug Culture (http://www.erowid.org/library/review/review.php?p=269 http://www.erowid.org/library/review/review.php?p=269) The Mouse Trap: The dangers of using one lab animal to study every disease (http://www.slate.com/articles/health_and_science/the_mouse_trap/2011/11/lab_mice_are_they_limiting_our_understanding_of_human_disease_.html http://www.slate.com/articles/health_and_science/the_mouse_t...) Why Most Published Research Findings Are False (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal...) Mistakes in Scientific Studies Surge (http://online.wsj.com/article/SB10001424052702303627104576411850666582080.html http://online.wsj.com/article/SB1000142405270230362710457641...) Trials and Errors: Why Science Is Failing Us (http://www.wired.com/magazine/2011/12/ff_causation/all/1 http://www.wired.com/magazine/2011/12/ff_causation/all/1) In cancer science, many "discoveries" don't hold up (http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328 http://www.reuters.com/article/2012/03/28/us-science-cancer-...) Lullaby for a Mouse: Anthropological Observations of an Animal Model of Psychosis (http://vimeo.com/33803995 http://vimeo.com/33803995) Inside the Battle to Define Mental Illness (http://www.wired.com/magazine/2010/12/ff_dsmv/all/1 http://www.wired.com/magazine/2010/12/ff_dsmv/all/1) There are of course quite a few relevant books as well, but that's probably enough to keep most people busy for a few days. Also, most of these articles are about drug research, medicine, or psychology, but it applies to basically every field.

Alex3917 15y ago

"Yet when Stuart Ritchie, a doctoral student in psychology at the University of Edinburgh, and two colleagues failed to replicate his findings, they had a heck of a time getting the results into print" Maybe because they were a bunch of hacks. They didn't actually replicate Bem's methodology, so whether or not they got the same results is irrelevant. The journals were right for rejecting their work.

Estragon 15y ago
Got a cite to back that up? Because Ritchie claims that no serious methodological issues were raised during the review process, other than the ridiculous "You can't get positive results from an ESP test unless you believe in it."
- Alex3917 15y ago
  I don't remember the exact article that discussed it, but I'm sure you could find it by searching HN. Other than the issue you mentioned, the other issue was that the original study was done in person using Ivy league students, whereas the replication was done over the Internet using a non-comparable demographic. Regardless of whether or not one thinks it's ridiculous that these issues could have any effect, it's intellectually dishonest for them to say that they've replicated Bem's methodology and failed to reproduce the results when in fact they haven't actually done so. (The researcher-belief issue matters since we already know that the how well a drug work depends on how much the person administering the drug believes it will work, which is why for well-designed drug trials the investigators aren't the ones who administer the drug. So it's not like the replication was being dinged for not subscribing to some exotic new methodology, rather they were being shot down for not following existing best practices. That said, I have no idea whether there were videos how much the researcher's beliefs would have actually been apparent, so I don't really know whether or not it was reasonable for the journal to reject it based on this one point.)
  - gergles 15y ago
    > we already know that the how well a drug work depends on how much the person administering the drug believes it will work hahahahahahaahahahahahahahahah what? No, no we do not. If you have a citation that suggests otherwise, I would be thrilled to read it. The reason that we blind is because there ARE effects based on perception if the person administering the drug knows its experimental. There have been studies indicating that the administrator would say things like "this really should work", for example. That invokes the placebo effect response in the patients (which is also well documented,) but the concept that "we know how well a drug works depends on how much the person administering it believes in it" is completely incorrect.

BenoitEssiambre 15y ago

God I hope this becomes a trend in other fields as well. Spending time in universities has made me very cynical of the research that comes out of them. There is just too much incentives for profs to ignore biases in their research. I've seen it happen many times. For them, it is the difference between becoming a prof or a lab tech and it is literally worth millions. You can not trust any research done where the career of the researcher depends on them finding results. Universities, by hiring based on research credentials that currently translate roughly in the amount of positive results a person has generated, completely render worthless the research going on in their departments. Aggravating the situation is the fact that the peer review system is too incestuous to be relied on especially when the peers are probably also 'bias ignorers' with incentives to keep the flaws in the system. I strongly believe that for hiring purposes, the research skills of profs should be evaluated on criterion that are incidental to research. Mostly math, probability, statistics (yes even for psychology) then methodology skills and also maybe leadership, communication skills and dedication to science (last only because it is difficult to measure). update:typos

viggity 15y ago
"I take the hard view that science involves the creation of testable hypotheses" - Michael Crichton I'm not trying to be a troll. Really, I promise. But I really hope that the next field they do this in, is in Climate Science, specifically articles on climate change. It seems like bulk of their work is based on computer models and I just don't know how you're supposed to replicate any of that. How do you isolate the millions of variables to show cause and effect with a closed source computer model. That is my biggest problem with climate science.
- roc 15y ago
  > "It seems like bulk of their work is based on computer models" Then it seems that you are confused. Climate science is about gathering data, putting forth a testable hypothesis and seeing how future data fits the prediction. Yes, they use computer models. In the same way that you or I might gather data from a wind tunnel test and make a computer model of wind flow from that data to extrapolate a theory and create new testable hypotheses of a more-aerodynamic shape. But so long as new data is used to test those hypotheses, the use of a computer model is irrelevant. Science is being done. Testable hypotheses are being created and new data is being compared to those predictions. As long as we aren't running around half-cocked, believing the predictions of hypotheses that have not yet been tested, there's no concern. And the core hypotheses under the umbrella of Climate science have been tested. Some disproved by data, leading to changes in thinking, further research and new testable theories. Some supported by data. None of the 'alternate' hypotheses can say the same. (No, it's not the solar cycle. No, it's not distorted temperature readings from data stations in urban areas.) The very reason that there is a general consensus around the so called 'tent-pole' theories in Climate Science is precisely because they have been tested and have been shown to make accurate predictions and continue to do so. Some of them, for decades now.

6ren 15y ago

reminds me of Feynman's cargo cult science's rat mazes http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm

amcintyre 15y ago

"If you’re a psychologist, the news has to make you a little nervous—particularly if you’re a psychologist who published an article in 2008 in any of these three journals...<snip>...Because, if you did, someone is going to check your work.' Can somebody in the field comment on this statement? Is it really so out of the ordinary to attempt to reproduce work in psychology that this kind of statement is warranted?

luriel 15y ago
Yea, I would think that any sane and honest scientist would be delighted that somebody else is going to test their work. But after things like this: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... I'm starting to wonder how many scientists are really more dedicated to search for the truth than search for grants and tenure.
- eli_gottlieb 15y ago
  I'm starting to wonder how many scientists are really more dedicated to search for the truth than search for grants and tenure. Scientists would love to do proper science. Just please, please, give us a way to not starve from doing it!
nchlswu 15y ago
I only have undergrad research experience. In general, I haven't really seen focus on replicating previous results - but it depends on the work and the researcher. Most of what I've seen is a researchers who extend their own work. in doing so, they replicate their previous research and reveal other information. In this case, replication is definitely a goal, but it's focused on a researcher's own work and a byproduct of exploring a particular research problem. I've witnessed people discussing replications and discussing that they were unable to replicate X's results. Here, there were clear are efforts made to replicate results, but I've only witnessed this in an informal setting, so the results were never published. I've never seen a big push to publish results that discredited a result due to an inability to replicate results I suspect that there is little effort made after the fact unless the theory was particularly 'significant,' or if a researcher can offer a competing account. The one time I learned about replication was when two competing researchers were offering competing accounts and discrediting anothers' results. However, in these cases, there were flaws in each others' research Obviously, in a classroom setting, we have seen many cases were replication has occurred, which makes a particular theory very strong. It is rare that the published results are straight-replications, though. They are often extensions.

hluska 15y ago

I've always had problems with fields like psychology. Part of the field revolves around hard science - neuroscience researchers (as one example) use tools like fMRIs, etc to capture observable phenomenon. On the other hand, there is the art of psychology - social/developmental psychologists use dubious tools like surveys and interviews to try and prove a thesis. I think that the average Hacker News reader could write a survey that would 'prove' that gravity doesn't exist. What kind of experimental integrity do surveys have? And then there are participant pools. These pools are primarily composed of undergraduates (who often earn bonus marks/money for participating). Add in a strong publish or perish mentality and you can see some serious problems. A whole lot of researchers are using dubious methods on a set of participants that do not adequately reflect humanity as a whole.

riffraff 15y ago
Add in that a lot of the psychology tradition is based on, let's say, "less than strictly scientific" traditions and personality cult.
disgruntledphd2 15y ago
There are some issues I (as a psychologist) would have with your assumptions. Firstly, fMRI and neuroscience research is probably the single biggest source of errors and shoddy research in psychology. Its a combination of really small sample sizes, poor statistical tests (brain regions are independent of one another, really?) and huge amounts of data dredging to find significance. See for a roundup: http://escholarship.org/uc/item/51d4r5tn http://escholarship.org/uc/item/51d4r5tn Interestingly enough, most of the observable pheonomena you note in neuroscience are linked to traits that people think about (optimism, personality etc) through surveys, so even if their methods were perfect, the results still wouldn't be. On your survey point, I would agree that the participant pools are quite limited and generalizability is quite low, but your example is ludicrous. No one does surveys of gravity in psychology, surveys are carried out to investigate the manner in which people conceptualise their experience. For a really great roundup of the problems with typical social science participant pools, see: Heinrich et al: humancond.org/_media/papers/weirdest_people.pdf Indeed, the entire issue of BBS that that article appears in is well worth a look for a deeper understanding of these problems. These surveys should then be calibrated against behavioural outcomes, but this does not happen often enough, which is a major issue in my view. Thats the major problem with surveys, well that and some poor methods accepted too uncritically within the field (factor analysis). To summate, psychology has many, many problems and I fully support this reproducibility effort (and it will expose a lot of findings as non-replicable). But don't single out surveys for derision, neuroscience deserves as much if not more of your scorn.

iandanforth 15y ago

In 2003 I did the work that lead to this paper (http://bit.ly/I1qvuI http://bit.ly/I1qvuI). Could it be replicated? Probably not. Let me outline a few challenges that don't have to do with a result being true or not. 1. The passage of time Unlike physics and biology, culture has a huge impact on the field of psychology, and cultures change. Additionally, my work was about addiction to online games. I have no idea if the same type of people exist today that did then. I do know that the game I was studying (Asheron's Call) does not exist in anywhere near the same form as it did then. 2. Copyright While I was lucky enough to find psychological measures (aka tests) that were in the public domain, most are not. To replicate a result you may have to pay to access the tests and then most likely cannot re-publish them. At the very least this makes replicating work inconvenient and expensive. 3. Data Because psychology is largely driven by statistics, to replicate a result you should really start by re-analyzing the data. The reality? This data simply isn't available. It is not published alongside the results, and I don't have it anymore. I might be able to find the cleaned data, but that's not important. Error and bias can just as easily be introduced during data cleaning. For example I removed many data points that appeared as outliers, but perhaps these should have been included. No one will ever know. I hope barriers such as these are addressed in this investigation. It would be disappointing to impune the work of scientists when it is the process that could really use reform.

k4st 15y ago
Note: I am not a psychologist. What do you mean by "some tests are copyrighted"? Are you talking about a statistical method being copyrighted, an implementation thereof, or something else entirely?
- ttt_ 15y ago
  He's refering to this (example): https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limitations https://en.wikipedia.org/wiki/Beck_Depression_Inventory#Limi... The BDI is copyrighted, a fee must be paid for each copy used, and photocopying it is a violation of copyright.
  - 15y ago

developer9 15y ago

I agree with others that this is a good thing. Science is only proven until it is disproven. For this reason, I think we need to hold other complex sciences like meteorology to a higher standard. I want to see how every prediction changes over time and how wrong it was per measurement later. If we had a map of places whose predictions were more wrong than others and this were publicized, there would be more impetus to fix these issues and improve the science. On the other hand, people need to take fringe science and religion more seriously and stop trying to debunk it. We can learn about this universe from every belief system.

jamaicahest 15y ago

I personally hope the software world picks this up, so we might get some actual data on things like TDD, agile, waterfall, instead of peoples' anectdotes and experiences on past projects and whatever Martin Fowler thinks up (no offense, I love Fowler, but he would benefit from proving some of his assumptions)

mangodrunk 15y ago
I agree. It seems we have a problem where things that are most likely just opinion or preference are treated as fact and to worsen the situation there is a disregard for evidence or the need of it. For example, take the argument why indexes should start with zero. [0] Some people will claim that it's easier to learn without providing evidence or that those who use such a system produce less bugs, again without evidence. [0] http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EW...

nradov 15y ago

A cynic would say that psychology can't come undone since it's never really been "done" in the first place. We know so little about the fundamental nature of human thought that the whole field is basically a joke. Their journal articles are built on a foundation of sand. I expect that 200 years in the future historians will look back at 2012 psychology as a pseudoscience, little better than phrenology.

montecarl 15y ago

I am glad someone is doing this. I recently tried to implement an algorithm described in a paper in a well known chemistry journal and discovered that the paper's results were incorrect. It turns out that the author's program (which was not made available with the paper) didn't correspond with what was described in the paper. That is to say, when I used the described algorithm, I got very different results than those in the paper. Now I'm not sure what to do. I want to correct the paper, but I don't want to openly criticize the work of others.

corford 15y ago
If you are prepared to back your claim up in a way that can be independently verified, why don't you want to criticize the original authors work? Surely correct science is more important than sparing someone's blushes?
- montecarl 15y ago
  After reading your comment, I can't come up with any reason other than the blushes. Perhaps I'll reconsider.
  - otoburb 15y ago
    How about reaching out to the original author and outlining the problems that you've been encountering? This way, you can at least give them a chance to respond privately or correct potential misunderstandings with the paper (perhaps publicly). Similar analogy would be approaching a vendor regarding a security/vulnerability disclosure.
    - montecarl 15y ago
      I already have. They recently gave me the source code and I pointed out the problems. They agree that there are bugs in their code. I just haven't approached them about publishing a comment on their paper because I feel bad about it.
nchlswu 15y ago

radarsat1 15y ago

It's funny because, coming from an "interdisciplinary" research area, many of us cower in fear of psychology journals, completely daunted by the rigorous level of (perceived?) statistical knowledge required to be accepted. Psychology is viewed by many as one of the most hard-core sciences, just because it has taken such a hard-line attitude toward statistical work, out of shear necessity, due to past problems arising from a history of more philosophical approaches to the subject. Other sciences (e.g. HCI) are often somehow "softer" just because they take pains to avoid the kind of criticism that can come from the kind of complex statistical interpretations we see in pure psychology journals. Although, I suppose I could be mixing up psychology with what we call "psychometrics", which is the kind of psychology that I'm more familiar with due to my research area, which involves perception of virtual reality. By the way I agree with others that this is not an "attack" on psychology, it is simply science. Verification of results can invalidate claims, but it can also easily provide further evidence for claims. Nothing bad can come of this initiative, if it's carried out properly.

disgruntledphd2 15y ago
You are definitely mixing up psychology with psychometrics. Psychometrics is awesomely thorough and aware of statistical models and the limits thereof. Psychology, on the other hand relies on SPSS and what other papers have done to determine their methods.

Estebon 15y ago

I have found Hacker New to be full of some of the smartest people I've encountered on the internet. However, every time a discussion pops up where a social or soft science is involved, the discussion becomes mired by arrogance, bias, and general small mindedness about what constitutes "real science." I see comment after comment of where the author implies or outright says that psychology is pseudoscience or close to it. The challenge in designing, implement, and supporting a study in psychology or similar science is staggering. That's no excuse for questionable statistics interpretation or outright manipulations, but to write off an entire field of study because it doesn't have the convenient quantitative measuring capabilities that other sciences have is ridiculous. I have noticed a pattern as of late, or maybe its just the articles I have been reading on HN, but there seems to be a lot more people making dubious claims, commenting on things they no nothing about, and just genuinely turning this place into a cesspool like the rest of the internet. What is so hard about not posting an opinion on subjects that you don't have knowledge of or any proof to back up your claims? If you haven't studied psychology, have even a vague idea of what it's about, or what its past or present state is then why muddle up the conversation with you BS conjecture? I'm sure a legitimate licensed clinician would have a field day with some of the posters on this site.

disgruntledphd2 15y ago
While I agree with much of the substance of your post, the fact remains that psychology is a mess. A friend of mine once told me that doing a doctorate in a subject only really gave you the ability to see how your field is wrong and all the problems with it, and I would agree. The problems with psychology, from someone who's been at it for a while: 1) lack of focus on replication 2) in survey studies, failure to correlate psychometric measures with behaviour or other kinds of measures (surveys versus reaction time measures versus physiological measures). 3) An unfortunate lack of understanding of the assumptions behind the statistical procedures used routinely within the field. 4) Misplaced emphasis on theory at the expense of prediction. A relatively well known psychologist, quite statistical aware, posted on stats.stackexchange.com that the goal of psychology was theoretical understanding, not prediction. For the life of me, I can't see how one can develop good theories without prediction, but it appears to be a dirty word within much of psychology. That being said, people are hard. They change their behaviour based on what they think you are trying to do, they tell you what you want to hear, and even when they tell the truth as they see it, they may well be mistaken. So while its not right to write off an entire field for some errors (in fact, my results are all perfect) its also useless to deny the problem and pull a Freud by saying that everyone who disagrees with your methods has some kind of psychological disorder.

deleted 15y ago

[deleted]

pchivers 15y ago

Richard Feynman on reproducibility in science: "When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens." (from "Cargo Cult Science")

nerdfiles 15y ago
http://webjournal.nerdfiles.net/2012/04/03/this/ http://webjournal.nerdfiles.net/2012/04/03/this/ Notice the date. That was the date. What is interesting is that they are engaging in an activity that wishes to avoid reproducibility. They allow "facts" and "ideas" to swap. Articles like http://ir.canterbury.ac.nz/handle/10092/5828 http://ir.canterbury.ac.nz/handle/10092/5828 exist. There's the Philosophical Foundations of Neuroscience. All that theorizing needs to record facts. To do that, we need to be able to follow what the hell is going on without blatant wankery like with Anthony Crick or early John Searle's "foot in the mind" rubbish or Dennett's everything's-a-scientist metaphysics. Philosophers got the descriptivist bug with Experimental Philosophy. Linguistics has always been largely descriptivist. Now psychology is to do the same. Who cares that they may have false theories, let's see if they've described anything. As Fraiser obviously demonstrates, psychologists have had their Golden Age. Psychology needs management.

tokenadult 15y ago

Free full text of the 1991 article by David Lykken, "What's wrong with Psychology, anyway?," can be downloaded from an online reference link. http://cogprints.org/371/ http://cogprints.org/371/ The article shows how many of these issues have long been on the radar screen of psychologists at the minority of universities (for example, the University of Minnesota) where the psychology departments train graduate students in the general scientific method. There are some very amusing, thought-provoking, and shocking examples in the article.

ams6110 15y ago

It would not surprise me in the slightest to find that a large amount of published research findings are pure bullshit.

rsanchez1 15y ago

Sociologists everywhere are panicking. On the other hand, they'll probably expose a whole bunch of "decided" research fields that new psychologists can revisit.

Mohnican 15y ago

What an over sensationalized headline. I realize it's the same as the source but I thought HN was better than this.