8 ms·
Replication Controversy in Psychology
- danso 12y agoControversies like these make me glad I work in the field of programming...while arguments over what algorithm or frameworks work best are less sexy than what the sciences cover, at least with narrow claims, it's easier to concretely argue for one or the other...and then of course, if we're talking about open-source, then the errors of ambiguity are even scarcer. It seems like a large part of the problem, at least in these psychological experiments, is that there isn't a structured/uniform way to describe preconditions, methodology, assumptions, and measurement practices...OK, so you used 40 undergraduate students for your original experiment...how did you pick them out? How were their characteristics accounted for (e.g. age/gender/major/health/etc)? Did any of them ever see the Trainspotting scene before? Have any of them ever seen a movie of equal grossness? How many minutes did you wait after showing them the Trainspotting scene? In what order did you hand out the questionnaire? How did you seat the students? etc. etc. etc. etc. A totally honest researcher might have trouble enumerating and enforcing all of the different controls, never mind communicating them to other researchers. From the research papers I've read, the ability to communicate these facts doesn't seem to be of uniform quality.
- lliwta 12y agoCertain parts of software engineering research are moving in this direction, psychology as a the primary toolbox for research. Not sure how I feel about that. As far as science goes, Psychology isn't exactly the epitome of rigor. On the other hand, the problem really boils down to asking hard-to-investigate-scientifically questions, mostly about humans.
- agent00f 12y ago> Controversies like these make me glad I work in the field of programming...while arguments over what algorithm or frameworks work best are less sexy than what the sciences cover, at least with narrow claims, it's easier to concretely argue for one or the other Yeah, it's pretty ironic when the field with (Comp) Science in the name isn't really a science but math.
- gipp 12y agoIt is a science. A formal science. Same as math and logic, for example.
- kylebrown 12y agoHal Abelson disagrees with you. "Computer Science is a terrible name for this business. First of all, its not a science. It might engineering, or it might be art. But we'll actually see that computer -- so-called science -- actually has a lot in common with magic."[1] 1. https://www.youtube.com/watch?v=zQLUPjefuWA https://www.youtube.com/watch?v=zQLUPjefuWA
- cfallin 12y agoAdd to that list Fred Brooks (IBM System/360, Brooks' Law): http://www.cs.unc.edu/~brooks/Toolsmith-CACM.pdf http://www.cs.unc.edu/~brooks/Toolsmith-CACM.pdf Relevant quote: "... the scientist builds in order to study; the engineer studies in order to build. ... I submit that by any reasonable criterion the discipline we call 'computer science' is in fact not a science but a synthetic, an engineering, discipline. We are concerned with making things, be they computers, algorithms, or software systems."
- mikevm 12y agoRobert Sedgewick would disagree with Hal Abelson. Just because some people tend to use pure maths to analyze the run-time of algorithms, doesn't mean we can't apply the scientific method to the problem. Here's a nice presentation by Sedgewick where he shows how we can apply the scientific method to programming: http://www.cs.princeton.edu/~rs/talks/ScienceCS10.pdf http://www.cs.princeton.edu/~rs/talks/ScienceCS10.pdf And quote: > Algorithm designer who does not experiment gets lost in abstraction > Software developer who ignores cost risks catastrophic consequences
- jeffdavis 12y ago"...while arguments over what algorithm or frameworks work best are less sexy than what the sciences cover, at least with narrow claims, it's easier to concretely argue for one or the other..." Very few arguments related to programming are over algorithm choice, they are almost always over engineering concerns: readability, platform/language choice, frameworks, etc. Those, in turn, are often dependent on a mix of personal taste and predictions about the future (e.g. directions for the product architecture). So I don't think programmers are quite immune from non-scientific analysis. That's apparent from the longstanding programmer debates about basic things like compile-time type checking. Even narrowing the claims rarely seems to reach much consensus.
- idlewords 12y agoAn additional problem with research into things like social priming (and the attempts to replicate it) is that they all use a sample population of American undergraduates. This paper goes into interesting detail about why this is a terrible methodology: http://hci.ucsd.edu/102b/readings/WeirdestPeople.pdf http://hci.ucsd.edu/102b/readings/WeirdestPeople.pdf
- spindritf 12y agoUsing subjects from the same population should make replication easier. If the research was only valid for westerners, it would be a step up.
- autodidakto 12y agoThanks. I've been wondering for a while: How much do we know about human psychology is based on studies performed only on 20-year-olds?
- tedks 12y ago...well, quite a lot actually, because 20 year olds are more like other humans than different. Humans don't have life stages. We don't wrap ourselves in cocoons and change from grubs into butterflies from ages 16-20. For most of human existence, 20-25 year olds were the substantial power holders. There's a branch of psychology called developmental psychology. It has to do with how humans change over time. If developmental psychologists only studied undergraduates, we wouldn't have a good grasp of the area, but they don't. It's pretty easy to get kids and elderly people to study. It's even pretty easy to get adults to study. It's more expensive than undergraduates, but it's doable. Beyond that, what sort of differences do you expect to see in areas like, say, input attention, working memory, fluid intelligence, social behavior, perception, motor control, etc.? 18-22 year olds are pretty much normal adults in all of those regards. The thing that makes this really insignificant is that we know so little about how minds work. Psychology has only really existed since Skinner and behaviorism made it a real science. If I spent 10 seconds I could think of nice, clever-sounding retorts as to why the perception system of a 22 year old is vastly different from the perception system of a 32 year old, but in reality we know almost nothing and learning anything is good. As Heinlein said, it's wrong to think the world is a sphere, but it's much better than thinking the world is flat. These discussions about replication always sadden me, because they miss the point by so much. Psychology is one of the least respected and yet most vital sciences. The problem is that everyone is a lay expert, because everyone has a mind and thinks they understand how minds work. Psychology has taught us so many valuable, horrible and beautiful things about ourselves. If you're reasonably intelligent, you can come up with reasons not to believe anything. It's easy to discredit things. It's hard to build things. (It's also false that psychology studies are only performed on undergrads, usually undergrads are a good pilot testing base and then you move on to externally recruited populations, unless there's really no reason to do so based on the field.)
- ajarmst 12y agoIoannidis' "Why most published research findings are false" (http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...) is particularly relevant here, I think.
- ajarmst 12y agoNote, in particular, his "Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. "
- wyager 12y agoNow, I'm just a layman, so I could be completely off the mark about this, but this is how it seems to me. Doesn't every single generation of psychologists pretty much completely reject the findings of the previous generation? All of Freud's stuff is now considered total bullshit. All of the stuff psychologists said in the 50s about homosexuality being a mental illness is now considered bullshit. In other fields of science, like chemistry or physics, this has never happened since the introduction of the scientific method. Scientists have been wrong lots of times, but never has the majority of the body of scientific knowledge been thrown out. Usually the changes are something small, e.g. "Einstein was wrong about hidden variable theory in some respect." Even the most drastic changes in scientific thought are usually just the narrowing down from a number of existing theories to a smaller number of theories. In most fields of science, we seem to converge on the correct result over time, with very small deviations. Psychology doesn't seem to be converging at all. Am I wrong? If not, why is this?
- aridiculous 12y agoThe problem with your premise is that all of Freud's stuff isn't considered total bullshit. The mass media, science magazines, and materialists like to present it that way, but psychoanalytic and psychodynamic traditions (depth psychology) are still alive and well. It is, perhaps, not quite as prominent in experimental psychology (which I'm guessing is what you're talking about), but it's certainly still a force in clinical work. Humanistic, positive psychology, cognitive-behavioral, gestalt, and most other approaches were inspired by, informed by, and still use parts of depth psychology in their approaches. I think you're mixing up clinical psychology and experimental psychology. The aims for both will vary, and I tend to agree with you that presenting psychology as a hard science is ludicrous (but unfortunately the only way to get funding). You're also right about the political component of psychology (your homosexuality anecdote); psychology can be used as an instrument by the leaders of society for political, social, and economic gains and control. That doesn't mean it has no contribution to man's knowledge of the world. Just because the newer approaches appear to reject older theories does not mean the older theories aren't useful. Newtonian mechanics explains phenomena quite well for everyday living. Relativity was a revolution in the field but there's a still a place for Newtonian physics (i.e. you don't have to erase the classical theory). The way knowledge evolves in these long-standing fields isn't as mechanical as I think you're claiming it to be. Science is much more complex than that.
- ninguem2 12y agoIf the standard in Psychology research is the so-called p-value being less that 0.05, it means that they publish results whenever the chance of the findings being a coincidence is less than 5%. It stands to reason that 5% of the published results in Psychology will be coincidences and not real.
- masterjack 12y agoIt is even worse than that when negative results are not published. This is because a study can keep getting repeated by different parties until they get that p-value. In other words, if you never publish a negative or inconclusive result (or stop studying once a conclusive result is published) then given enough time you /will/ get a positive result for every single plausible hypothesis.
- beagle3 12y agoIt is actually much much worse. Let's do the math. 1,000 non-tenured professors do 20 experiments each. On average, each of them has one p<0.05 result that is actually a coincidence. They all rush through publication in an attempt to get tenure. Unless you have reason to believe some results are actually plausible, it's possible ALL published results are wrong. The 5% is the ratio between "experiments done" and "results that are not meaningful yet randomly deemed worthy of publication". It has no relation to the number of "results that are actually meaningful", which may outnumber the former 100:1 ... or, more likely, be outnumbered by them 100:1. I wish Neyman/Pearson theory was taught more widely, or at least its implication to the false-positive/false-negative tradeoff. Alas, that would mean a lot less published results and a lot less bragging/tenure rights.
- analog31 12y agoMy limited understanding is that the simplistic formula for computing p-value assumes certain things about the lack of hidden correlations and effects. If those things are present and correctly taken into account, then the correct p-value would be much higher. Arbitrarily multiplying reported p-values by 10 would not seem to be too far from the mark.
- capnrefsmmat 12y ago
- danielweber 12y agoI was very surprised to not see Jason Mitchell at Harvard Univeristy's attack on the concept of replication: http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm The thing is amazing to read, in the "someone who thinks he's a scientist actually wrote this?" sense, like this gem: Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way.
- jerf 12y agoThat statement reveals a stunningly large unconscious bias. Let us assume that Study A produces result X, and Study B failed to reproduce it. Let us take for the sake of argument an assumption that one of these studies is flawed. Which one is it? You can not tell. There is an old saying: "A man with a watch knows what time it is. A man with two watches is never sure." - http://en.wikipedia.org/wiki/Segal%27s_law http://en.wikipedia.org/wiki/Segal%27s_law
- lambdaphage 12y agoI remember reading that and reeling, shocked that anyone could write and sign their name to that. Whenever I am pessimistic about my chances in science, I remember that that guy has a job at Harvard.
- tedks 12y agoHe is a scientist; unless you're also a domain expert in his field with at least as much expertise as him, or a newcomer with radically impressive work, you can't just not call him a scientist because he disagrees with you. If a domain expert disagrees with you, it is probably because you are wrong. That's just the base rate and it's probably true here as well. Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. It's foolish to think any psych study has ever been replicated. The way psychologists deal with this is the concept of converging operations. This has to do more with being clever and considering implications of a thing you're trying to study with respect to other theories you have much more evidence for and in different situations. If you think stereotype threat is a thing, for example, it would make sense for stereotype boost to exist if the stereotype is positive. If a priming effect speeds up the recognition of "doctor" after a subject has been shown "nurse", maybe recognition for "warmth" will be slower after a subject has been shown "frozen". If people only encode based on schema variables, they'll be unable to remember information unrelated to a schema even if they're provided with congruent encoding cues. Converging operations is just studying the same concept from different angles and seeing if you keep seeing consistent results. If the results aren't consistent the hypothesis you're investigating will, in fact, die. Psychology is an advancing science as of right now. If people were just publishing random noise there'd be no reason for concepts to fall out of favor. I think people on hacker news tend to wildly underestimate the degree of skill involved in experimental psychology. So much of statistics, even, has come out of psychology because of the need to refine and clarify what effect you're seeing. Alternatively, if you think because of this existing academic psychology is lacking, you're in a fantastic place to be. Do literally anything to get some seed money, just working in a standard SDE job would be fine, and then run some small experiments (and replicate them, of course) on learning or addiction. Then write the next Farmville or build the next Facebook. If there's really that much low-hanging fruit, go out and pluck it. Put up or shut up.
- michaelfeathers 12y agoMaybe the answer is to perform distributed research by default. Design experiments and have them conducted by unrelated teams in different locations.
- couchand 12y agoThat's a great idea but you really only need half of it. Design the experiments in enough detail that unrelated teams could conduct them, then publish the results no matter the outcome. That's probably enough to catch most of the loose research out there.
- east2west 12y agoThis covers all issues surrounding reproducability pretty well. It is astounding that scientists want to hide behind their expertises when their results and conclusions are being tested. Biologists who are under a similar cloud tried the same tactics too. They have some secret sauce that make their lab and experiment special and unique, beyond the comprehension of mere mortals. Of course, as publically funded scientists they have a moral if not legal obligation to disclose their "secret sauce" if that is very important. Why publish a paper but leave the public in dark about the most important factor? At best it is an oversight, at worst it is dishonst. Experiment and reason underpin the modern science. Common sense says inreproducible experiments call their colusions into question. Scientific experise is no escape from reason. A fancy degree from a "top" university does not make infallible anyone. Well controlled experiments are the final arbitrator.
- MarkMc 12y agoAm I being naive to think that scientific papers should be written in a way such that people outside the area of expertise can reproduce it? You shouldn't need to be a psychologist to run a psychology experiment. You shouldn't need to be a physicist to run a physics experiment.
- rando289 12y agoYes. You are being naive. You could reproduce rote aspects of the experiment. Not even all of them, you don't know how to calibrate some expensive lab equipment, its not the paper's job to explain that. And lots of things like conclusions and analysis take expertise. It's not about "being", its about having expertise. Papers say because foo, we conclude bar. If you have almost no idea what foo is, you just can't write that. It's not the paper's job to give you an introduction to physics or anything.
- east2west 12y agoAs much as I hate that sceintific papers don't give good introductions to the public, specialists do have their values. An educated, intelligent reader has limits when it comes to frontline research. It is like asking someone flying a 787 jetliner without touching an airplane before. Prior knowledge and experience are important. What I absolutely reject is false exclusivity that some scientists hide behind. A professional statistician can spot statistical errors even if he or she never did any cancer study before. A rational person can discern the trustworth of a scientific report if enough details and background are well explained. I always find that popular press lacks a degree of healthy comptempt for specialists. It actually has improved somewhat recently. Still sometimes newspaper reports read like press release.
- plg 12y agoAs long as an academic researcher's personal career progress (and pocketbook) depend on a seemingly never-ending flow of "novel" and "significant" findings this will continue.
- misnome 12y agoI was astounded to read: > [Psychology] is actually leading the way in tackling a problem that is endemic throughout science [replication] "Throughout science" presumably is meant to apply to everything including physics, biology, etc. In fact, the article explicitly recognises "Failures to replicate" in several other fields (medicine, biology, observational astronomy) which seems to contradict itself - it's a big special thing when a psychology journal publishes replications, and they are leading the way! Except here is a load of uncontroversial non-replications from other fields. Perhaps I'm just being spoilt - being in High Energy Physics, almost everyone I work with has an understanding of the statistics involved, and extremely skeptical outlooks on everything that is anything less than perfectly rigorous - along with (usually) multiple separate experiments all measuring (or capable of measuring) the same thing. One big example I can think of was DAMA: http://en.wikipedia.org/wiki/DAMA/NaI http://en.wikipedia.org/wiki/DAMA/NaI - found evidence for dark matter, and a successive experiment by the same team http://en.wikipedia.org/wiki/DAMA/LIBRA http://en.wikipedia.org/wiki/DAMA/LIBRA has found the same result - but nobody "believes" the result, until it is measured by somebody else, even though nobody can find a cause for the false positive (assuming it is false). Finally, I link (again) the Feynman 1974 Caltech commencement address, where he actually talks about this reproducibility issue: http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html