22 ms·

Psychology’s Replication Crisis Is Real

490 points by zwieback 8y ago

openloop 8y ago
Psuedoscience at it’s finest! I love it! God bless!
SeanLuke 8y ago
> * A mention of the marshmallow test was removed from an early paragraph, since the circumstances there differ from those of other failed replications. I tried digging into the study webpage, but holy cow it's not organized in an easily digestable fashion. So does anyone know what happened with the marshmallow test replication?
- ddebernardy 8y ago
  It failed. https://www.theatlantic.com/family/archive/2018/06/marshmallow-test/561779/ https://www.theatlantic.com/family/archive/2018/06/marshmall...
  - SeanLuke 8y ago
    That much is obvious. But your link is to a different replication study. I want to know why there is an asterisk at the bottom of this article.
  - hn_throwaway_99 8y ago
    Reading your article, I wouldn't necessarily interpret it as "failed": > Instead, it suggests that the capacity to hold out for a second marshmallow is shaped in large part by a child’s social and economic background—and, in turn, that that background, not the ability to delay gratification, is what’s behind kids’ long-term success. I.e. it would still seem to say that the capability to delay gratification is tied to economic success, except that's really a confounding variable. It's also very possible that the ability to delay gratification that one gets from being more affluent is also a factor in future success.
    - taeric 8y ago
      Both of those, though, spell failure for the original experiment. It claimed that the predictor was delayed gratification. That is false as false as the predictor of future success being how successful you are in 10 years. :)
    - eezurr 8y ago
      I'm guilty of not reading the article, but I find pleasure in poking apart statistical analysis. There are so many things to consider, like average sugar intake of affluent/non-affluent families at the time of the first test and now. If both classes consume candy, perhaps the affluent have access to tastier candy (maybe important because the wealth distribution in the early 90s was more equal), so marshmallows are not tasty to them. (Note: I haven't had a marshmallow in a long time, but they are no where near as tasty as candy bought off the check out shelves). If present affluent families are educated about sugar (which was less so the case in the early 90s), they would probably feed their children less junk food, so the desire of wanting more of it is not as hard wired in their neurons, or it would taste overly sweet to them. Maybe less affluent families give their children sweets all the time because it shuts them up. Maybe the recipe of marshmallows has changed between then and now (see point 1).
- ineedasername 8y ago
  I think it's the Stanford marshmallow experiment in delayed gratification: children could take a single marshmallow immediately or wait a certain amount of time and get an additional reward. The results were tracked longitudinally for the participants, with positive life outcomes linked to those who were able to delay gratification as children.
  - thatcat 8y ago
    I thought it was later found that the delayed gratification simply correlated with parental wealth and that is why it linked to positive outcomes later in life.
- nradov 8y ago
  It would be interesting to try and replicate the experiment with other treats. As a child I sort of liked marshmallows, but not a lot. So if I had been an experimental subject, I might have taken the first marshmallow not due to a lack of willpower but just because I didn't care much about marshmallows.
lifeisstillgood 8y ago
It's positive news mostly - they took 15,000 volunteers and ran 28 "big" studies across them, working with original teams to get the nuances right. So they had larger samples than originals and ran experiments multiple times across multiple samples. Where a study failed in one it mostly failed everywhere, and where it worked it mostly worked in all the experiments. So, yes, priming someone with the number 32 won't get them to bet it in the casino, but it does mean that social science can be experimental - you can build and run lab scale social science - a positive !
- ALittleLight 8y ago
  This is a really optimistic take. I agree with you, ultimately, this is positive because it shows that it's possible for psychology to be done and have widespread meaningful results. In the short term though this seems extremely negative as it indicates that the psychology field and researchers we have today are producing substandard results. Two parts of this article increase my negative feelings. First, they mention that these 28 studies were selected because they were well known and influential. If well known and influential studies only have a 50% reproducibility rate - what about the average study? Another point I find very troubling is that people betting online were able to predict, with high accuracy, which studies were accurate and which weren't. This tells me that the problem is an "Emperor's New Clothes" type of problem and not just a challenge with the fundamental difficulty of the subject since the subject is so yielding to the public's intuition. Instead of a problem with the fundamental difficulty, the problem seems to be with the psychology of the researchers - being unable to diagnose and prevent methodological flaws in their research. This class of failure is much less excusable to me.
  - lifeisstillgood 8y ago
    For me the unrepeatable experiments suggest something even more interesting - that unless the originals were flawed or fraud, then something about the people tested has changed. We are all more aware of conmen movies, of fake news, or have read popsci books and now know what to look out for.
aj7 8y ago
They succeed in replicating “half the time.” If they did a second replication trial, I wonder what the overlap of replicated studies with the first replication trial.
- ThrowMeDown01 8y ago
  They did? > And they repeated those experiments many times over, with volunteers from 36 different countries, to see if the studies would replicate in some cultures and contexts but not others. So different sets of people, and different researchers (the article doesn't say it, but I doubt the same researchers traveled across the globe repeatedly).
viburnum 8y ago
Is there a list of the successfully replicated experiments? I haven't been able to find it.
- clord 8y ago
  I'd hope for a journal the only publishes replicated results. I think there is limited funding for this right now.
seibelj 8y ago
Even in hard sciences, there are papers that everyone in the niche knows is false, but fear to speak out about or publish a paper refuting it for fear of their career being damaged. My wife is a biochemist, and during her PhD they were working on an area of research with a handful of labs publishing about it. One lab in particular was known for a decent amount of questionable publications, but the PI was a big deal and no one would officially question anything. So they would whisper amongst each other and just ignore that paper (and all the additional papers built on top of it) because they all knew it was bullshit.
- akvadrako 8y ago
  Not all "hard" sciences. I'm pretty sure there are no papers in physics that are well known to be wrong but people fear speaking out, though I'm open to being proven wrong.
  - qudat 8y ago
    Richard Feynman was laughed at when presenting new ideas in physics when he was an up-and-comer. The human ego tends to intervene when careers are on the line.
    - whatshisface 8y ago
      Laughing at true new ideas is actually the opposite behavior flaw of whispering about false ideas that are known to be false but are associated with too much social power.
      - nothrabannosir 8y ago
        It’s the same family: public judgement of a claim influenced by the person making it, rather than its contents. Point being: Feynman’s ideas should have been judged on their merit, rather than Richard’s career. Same for the papers published by the institute mentioned above.
        8y ago
ineedasername 8y ago
I don't see how people can doubt this. Positive results get published. Even without p hacking that means a significant number of experiments at the p <0.05 level are published when they're one of the 1 in 20 due to chance. Combine that with p hacking and small samples with small magnitude effects... well replication is going to be a problem. Part of the solution here might be more impetus and incentive to publish negative results.
- hn_throwaway_99 8y ago
  Agreed. I think all fields should require pre-registration, where studies are announced before they are run to help counter the positive-results bias. Still not a 100% fix (I could imagine a lot of studies that look like they'll show no effect then move into the "oops, nevermind, I knocked over the test tube" variety), but certainly better than what we have now
- mamon 8y ago
  I’ve always thought that p<0.05 is ridiculously low bar, missing at least one zero after the dot.
  - vharuck 8y ago
    It should be more than enough if the experiment's replicated rnough times.
    - ineedasername 8y ago
      Yes, and with larger samples. Too many small sample experiments which, at most, should be considered a "pilot" study, and when results are positive even with p < 0.01 they should still only be considered "suggestive" until replicated, preferably with larger samples.
    - IshKebab 8y ago
      That's equivalent to having a lower p threshold in the first place though.
aligshow 8y ago
Psychology is a political pressure group directed by the American Psychological Association (APA). https://www.washingtonpost.com/news/the-fix/wp/2016/08/07/the-american-psychiatric-association-reminds-its-doctors-no-psychoanalyzing-donald-trump/?utm_term=.e81c978e0ad3 https://www.washingtonpost.com/news/the-fix/wp/2016/08/07/th... The American Psychiatric Association issues a warning: No psychoanalyzing Donald Trump https://www.psychologytoday.com/us/blog/brainstorm/201806/apa-urges-trump-change-immigration-policy-0 https://www.psychologytoday.com/us/blog/brainstorm/201806/ap... The organization is highlighting the psychological danger of family separation. https://www.apa.org/news/press/releases/2018/09/detain-child-migrants.aspx https://www.apa.org/news/press/releases/2018/09/detain-child... Statement of APA President Regarding Administration's Proposal to Detain Child Migrants Longer Than Legally Allowed > “We are dismayed that the administration continues to place the mental and physical health of migrant children and their families in jeopardy. Holding children — even with their parents — in these federal facilities for longer than the 20 days allowed by current law is unacceptable. Research has shown that immigrant detainees are particularly vulnerable to psychological stress. Furthermore, the longer the detention period, the greater the risk of depression and other mental health symptoms for immigrants who were previously exposed to interpersonal trauma. I can't remember any time when the APA criticized Obama the way it is criticizing Trump.
mcguire 8y ago
"But skeptics have argued that the misleadingly named “crisis” has more mundane explanations. ... Third, people vary, and two groups of scientists might end up with very different results if they do the same experiment on two different groups of volunteers." Isn't that the definition of a failure to replicate?
- your-nanny 8y ago
  I would call it a failure of interpretation. at one extreme one can assume that your sample is unique, and results inapplicable to anyone else. At the other extreme is the assumption that your sample is sufficiently representative so that your results apply to everyone. obviously in most instances it is somewhere between these, and that it "failed" is the basis of new inquiry
- anvandare 8y ago
  Yes, but the difference is subtle. The theory might still be true after all, but only for a particular subset. "We've tested the theory that 'swans are white', and it turns out the swans in our test were all black". (The conclusion isn't that swans aren't white, it's that some swans are, while others are black.) (add.: the article mentions the 'WEIRD' category. Obviously the applicability/set range of a theory matters. If it only holds for "people who, between 2 and 5 pm on Sunday are walking around in Reading with a bowler hat.". Well, that's nice, but not very useful.)
  - ajuc 8y ago
    This is common enough that it seems psychology as a whole is more art than science. It's like alchemy - they noticed some patterns and have some success exploiting them, but the abstractions and explanations they provide are mostly bullshit, so the predictive power is basically 0.
    - anvandare 8y ago
      The problem is the Universe of which it speaks can not be reduced to a handful of beautiful and concise mathematical formulae. A group of humans (or even a single human) is more chaotic and complex in their behavior than a group of atoms ever could be. All Carbon-14 atoms are the same. No two humans, however, are.
Invictus0 8y ago
I'm not sure that there's necessarily a problem with psychology itself, but just that the number of participants in studies is consistently too small. Just like how physicists increasingly have to spend more money to get more accurate results (LIGO, LHC, redefinition of the kg, etc.), psychology needs to realize that 1000+ study participants is necessary to get valid results. Unlike physics, though, where it's easy to show that results are bad when the equipment is bad, the nature of psychology makes it easy to publish spurious results without enough support. When the psych community starts to demand higher N values, you expect fewer, lengthier, more costly, but more accurate results.
HaukeHi 8y ago
I've started a crowdfunding campaign that aims to tackle the replication crisis: Check out https://www.lets-fund.org https://www.lets-fund.org We're raising funds for Professor Chris Chambers at Cardiff University. Chambers is a leading proponent of a new and better way of doing and publishing research, called ‘Registered Reports’, where scientific papers are peer-reviewed before the results are known. This might: -make science more theory-driven, open and transparent -find methodological weaknesses prior to publication -get more papers published that fail to confirm the original hypothesis -increase the credibility of non-randomized natural experiments using observational data The funds will free up his time to focus on accelerating the widespread adoption of Registered Reports by leading scientific journals. This ‘meta-research’ project might be exceptionally high-impact because we can cause a paradigm shift in scientific culture, where Registered Reports become the gold standard for hypothesis-driven science.
jamp897 8y ago
Psychology in the west is really a form of cultural anthropology, in that it’s just measuring behaviors for a specific set of people but not the underline mind directly. So this would make sense IMO that you can’t repeat many of the studies because they’re not grounded in the fundamentals of the mind but a layer abstracted a way by cultures and habits. Akin to strudying a Toyota car and trying to generalize all of physics from it.
- pessimizer 8y ago
  I have a more cynical view: it's a field that almost exclusively deals in the creation and enforcement of behavioral norms; entirely political, verging on religious in its continual reification of theory-theory. I don't know what that has to do with the West, though. Behaviorism is the only real psychology.
  - jamp897 8y ago
    Exactly, the west part is that we tend to think that everything in the west is the whole world or whole field, when Asia is light years ahead in this case in that they focus on the underlining logic mathematically.
  - quickthrower2 8y ago
    What's your basis for this view? Any good reads you can recommend?
    - jamp897 8y ago
      The core issue is that you need to establish a norm, but without an absolute point of reference such as enlightenment it’s like trying to walk strait in a dessert which is not possible. But if you bring this point up of going in circles they act just like Plato describes in his Cave Alegory. So it’s not a new issue. I don’t know if anyone has written a book specifically about psychology being a faux science, like eugenics in a way, but someone should.
    - org3432 8y ago
MAXPOOL 8y ago
You would think that computer science can't have replication failures, but it can. And I'm talking about my own field: machine learning. There is so much hype that I suspect that people are pushing papers and trying to actively hide the irrelevance of the methods and algorithms they develop. Artificial intelligence faces reproducibility crisis http://science.sciencemag.org/content/359/6377/725 http://science.sciencemag.org/content/359/6377/725 Reproducibility in Machine Learning-Based Studies: An Example of Text Mining https://openreview.net/pdf?id=By4l2PbQ- https://openreview.net/pdf?id=By4l2PbQ- Missing data hinder replication of artificial intelligence studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-replication-artificial-intelligence-studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-r... >In a survey of 400 artificial intelligence papers presented at major conferences, just 6% included code for the papers' algorithms. Some 30% included test data, whereas 54% included pseudocode, a limited summary of an algorithm.
- softwaredoug 8y ago
  Yes. As a practitioner it’s very frustrating I think the situation is pretty analogous to other area with replication crises like psychology and nutrition: local variables dominate. Just like our own body’s there’s a lot that just depends on the domain or the specifics of the problem being solved... This has been my experience in a related field (information retrieval). There are trends and best practices, but the market and Impossible to recplicate overhyped academic research reinforce each other.
- androidgirl 8y ago
  Why isn't more research and research materials open source in the AI world? I don't really understand. If one doesn't have your training dataset or your code, how could they possibly replicate your results?
  - __s 8y ago
    So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved Or, more succinctly, code & data are left as an exercise for the reader
buka2009 8y ago
I WANT HACK ALL GAMES HELP PLS PLS
buka2009 8y ago
help I WANT HACK ALLGAMES
buka2009 8y ago
I VERY WANT HACK ALL GAMES PLSS HEELPP
niceworkbuddy 8y ago
We are seeing actual progress in science! This is good.
stuart78 8y ago
"Third, people vary, and two groups of scientists might end up with very different results if they do the same experiment on two different groups of volunteers." This seems to me to be such an important key to understanding this. Considering even myself, I'm not sure I would behave the same way as a subject from one study to the next (pretending there was a way to blind myself). And when you look across regional and global cultural variation, it must have an effect on participants. The results are less headline-getting if they seem to be overly qualified (e.g. we find that midwestern white women with the middle name Susan have a positive psychological response to salsa dancing) but a deeper appreciation for the complexity and diversity of individual psychology would seem superior to the current state of affairs.
deleted 8y ago
[deleted]
londonlover 8y ago
dang, what an absolute horrible time for this to occur as public faith in institutions is deteriorating.
- deleted 8y ago
  [deleted]
- pessimizer 8y ago
  Public faith in institutions is deteriorating because they do not do this. The more shit that gets thrown out, the better institutions look.
- buboard 8y ago
  maybe they are deteriorating because they 've been listening to bad science?
oldgradstudent 8y ago
There's no replication crisis. The replication failures are the only thing that is properly done. We have a pseudoscience crisis. And as others pointed out, it is not limited to psychology.
- tmalsburg2 8y ago
  There is no easy way to tell whether something is real or pseudoscience. A lot of things that we consider real today where perceived as pseudoscience in their time. Science is largly a gigantic collection of dead ends that look silly in hindsight and finding out what's good science and what's not is an integral part of the scientific process. The idea that there can be a science of just Newtons, Einsteins, and Feynmans is naive.
  - viridian 8y ago
    The problem is that replication studies are almost nonexistent for most discoveries. Science is to some extent a business, and very few people with a budget have any interest in allocating resources to replicating the work someone else did, for no to little gain.
  - oldgradstudent 8y ago
    If it's not reproducible it's not science. If you p-hacked your way into a statistically significant result, it is not science. You don't have to be Einstein. Even Einstein wasn't Einstein most of the time. But the results you do get must be reproducible.
forapurpose 8y ago
I haven't looked at this study, but prior replication studies have found that the effects were reproduced in almost every study, but that the effects were weaker in the replication than in the original study.[0] That raises different questions than 'psychology studies are all wrong', and I'm still waiting to see a serious discussion of them. So far, what I see is news (studies not replicated) and condemnation (like this article), from which I don't learn much. [0] That's not necessarily a condemnation of the researchers: Imagine a set of studies that, each time they are performed, have a 50-50 chance of producing the strong effects or weak effects. Which studies in that set are going to be the ones that you read about? It could be selection by randomness.
75dvtwin 8y ago
50% replication success rate. I suspect, the authors of the article are a bit too generous to call the original work 'sloppy'. And that the alternative would be: no experiments with conclusions' -- and that's, in their view, is worse. I disagree with that, I think 'being lied to', is worse than to 'not being told'. What if the original work was purposely fake just get a grant from some group (political or commercial)? If that's the case, than the field of psychology is a composed of big number of 'full-stack con-artists'.
deleted 8y ago
[deleted]
gboudrias 8y ago
As someone who is both a developper and a psychology student, your average coder knows about as much psychology as your average psychologist knows about coding. That is to say, not a whole lot. I am saddened but not surprised by psychology's replication crisis. But really it applies to all of academia, psychology is only in the spotlight because some of branches have less than stellar methodology in the first place. Social psychology gets criticized a lot, partly because it's one of the only branches rigorous enough to even be tested! Obviously I'm not in favour of this. But it's a very big machine that needs fixing, and I don't think it starts with tweaking the details; drastic changes are required if we are to gain back the lost credibility. To me, this applies both to social and hard sciences. Since this is a general problem with academia, either your field has already been called out or it has yet to be. I don't believe the current system offers an alternative (yet), though I'm cautiously optimistic about preregistration.
chiefalchemist 8y ago
It seems to me how the test subjects are selected likely matters. I presuming most studies advertise (in some form) for test subjects. That ad appeals to some but not to others, and that leaning could influence the results. In a way, the study starts there, not the lab. If the study were a landing page, with a call to action, the conversions are influenced by the ad. No doubt, 50% replication feels ugly. None the less, the fact is, the studies are slightly different. Maybe in theory that shouldn't matter, but perhaps the reality is it does. Again, per the PPC analogy.
jdoliner 8y ago
There's no such thing as a replication crisis in science, that's just a conflict-averse way of saying that some "scientists" aren't actually doing science. It's not limited to Psychology and this type of conflict-averse approach is a big part of the problem. Other scientists, the "good ones," if there are indeed any good ones left, need to realize that what's at stake is the trust of their profession as a whole. If you're not willing to expose false research in your field, then you're no longer working in science.
- dustycat 8y ago
  Science (academic work) is a career for many people, rather than a vocation. Rewards include high status, stable income, opportunities to travel, long vacations, etc. You can say it's not science but most "scientists" are just normal people with everyday priorities, rather than Einstein-like people who go on doing science work while working as patent clerks.
njarboe 8y ago
Richard Feynman had a speech warning about this problem back in 1974. Complete with a story about how a student was excited about doing some replication experiments on rats and was told "no, you cannot do that, because the experiment has already been done and you would be wasting time". Others have also been worried about replication problems from the 1960's on. Hopefully some of this worry sticks this time and we can get a better understanding of what is really true in these fields. Physics likes to have very well defined uncertainties on everything they observe and don't like to say something is "true" unless the fact is likely at 5 to 7 sigma. That seems like a good margin. In psychology, 2 sigma is the standard for publishing a result. [1]http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm
- cultus 8y ago
  P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong. This confusion, and ignorance of the prior probability of hypotheses is the reason why replication rates are so bad. In physics you can get away with not caring about the distinction, because of how accurate the measurements and precise the theories are. That doesn't fly in most other sciences, though.
  - jmcgough 8y ago
    > P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong As an undergrad, I even had tenured professors try to tell me that a p-value is the chance that a result is wrong. Most researchers in psych or bio sciences have a weak understanding of statistics, usually taking a single statistics class in undergrad, then a single one in grad.
beefman 8y ago
Paper is here: https://psyarxiv.com/9654g/ https://psyarxiv.com/9654g/
- entwife 8y ago
  Thanks. It's usually more informative to read the original source.
jcaldas 8y ago
It's somewhat heart-warming to read the comments here about machine learning. I did my PhD in machine learning from 2007 to 2012, and the main reason I left research was because of the widespread fraud. Most papers reported an improved performance over some other methods in very specific data sets, but source code was almost always not provided. Once, I dug so deeply into a very highly cited paper that I understood not only that the results were faked, but precisely the tricks that were used to fake them. I believe scientific fraud arises primarily from two causes: - Publish or perish. Everyone's desperate to publish. Some Principal Investigators have a new paper roughly every other week! - Careerism. For some highly ambitious people, publishing papers comes before everything else, even if that means committing fraud. This happens even with highly successful researchers, who have the occasional brilliant, highly cited paper, but who also publish a lot of incremental, dubious work. P.S. Mildly off-topic, but I love the Ethereum research community at https://ethresear.ch/ https://ethresear.ch/ , precisely because it is so open and transparent! I wish an equivalent community existed for machine learning.
- SquishyPanda23 8y ago
  > the main reason I left research was because of the widespread fraud. I'm super curious about this. I had a similar experience in another field. Is this something you witnessed going down in person, or did you just develop a strong feeling given a lot of clues?
  - jcaldas 8y ago
    > Is this something you witnessed going down in person, or did you just develop a strong feeling given a lot of clues? The latter.
    - SquishyPanda23 8y ago
      Okay thanks. I was unaware that that was happening. Sounds like a frustrating experience.
- seibelj 8y ago
crimsonalucard 8y ago
They successfully replicated a study involving replicating other studies.
LifeLiverTransp 8y ago
It might sound conspiracy. So i will try to formulate it, right away like it. Why- if i have exploits and knowledge on human weekneses of the human mind- would anyone share those? Basically, in a world, where the mind-hardware can not be fixed and improved. He who goes full black-hat first rules forever unchallenged, even unpercived? So wouldnt it be reasoNobel, to set free a quack science, to keep those with similar thirst for powerfull knowledge busy? Maybe its tabula rasa time when it comes to psychology. Only replicated studies count, and everything before simply - never existed - and with it, works and titles referencing to it.
stanfordkid 8y ago
The replication failure in psychology is inevitable. The whole field boils down to taking an incredibly complex system (human beings) ... and measuring a few variables on them across a few people. Most of the "statistical" justifications stem from assuming distributions are normal and samples are uniformly collected (both false). This is why I trust the works of clinical psychologists like Jung and Freud so much more seriously than the current approaches. It just makes a lot more sense to observe something complex descriptively, rather than start with a yes/no hypothesis and then test it with a few dozen people.
- FiberBundle 8y ago
  I'm not an expert on experimental psychology, but I'm fairly certain that the principles of how experiments are conducted in the sense that they control for other factors, choose sufficiently large samples etc., are pretty sound. An often mentioned problem in psychology research is "p-hacking", where researchers manipulate the data in a way that validates their hypotheses statistically, which seems like a more logical reason for a reproducibility crisis than the reasons you mentioned. Also just because there are some problems in the field should not cause you to dismiss science completely in favor of completely subjective speculating about how the mind works, which certainly also has value, but I don't think it's warranted to dismiss actual science in favor of it.
  - stanfordkid 8y ago
    But they're just not -- that's exactly my point. If you test a bunch of Harvard undergrads (the most common subject type in these studies) -- that's not a uniform sample. In any way whatsoever. Furthermore, even if you get a high p-value ... your experimental design can introduce so much noise. Even when you do find these high p-value correlations -- they don't really ever say much about "why" ... because they can't. So much of that is the study design. Let's say I make a study on how well people navigate a maze under the influence of alcohol vs. not. You can probably get a high p-value that they do worse while under the influence of alcohol... but it reveals very little about why. So much of that is dependent on the particular maze. In fact I bet I could design a maze specifically to prove whatever conclusion I wanted. That is the whole problem in psychology. There is very little focus given to actually characterizing cognition in any meaningful way. I like the clinical psychologists because they attempt to do exactly that, even though they have less "scientific" findings.
golemiprague 8y ago
The problem is that those type of sciences are now mainly women and women are not looking for the truth. They look for ticking boxes, getting certifications and going through the motions. We need more men in the social sciences if we want those fields to advance in any meaningful way like other fields where men are prominent, whether it is technology or music or anything else you can think about that moves forward all the time.
camelCaseOfBeer 8y ago
The problem is to become a practicing psychiatrist or even psychologist the professional must attain the hallmarks of a professional academic's career. To be a marriage counselor, you need to publish research that revealing ground breaking insights into the human experience. Go figure, we get shoddy science from people who had little interest in science to begin with. There needs to be distinct separate paths in the field for professionals and academics.
Voyage_wanderer 8y ago
Assuming we are living in a simulation. Every our attempt of understanding rules and breaking through the simulation should be considered as a security breach by another side. So it gets fixed. Hence no replication. It had been fixed, dumbs... it’s my theory and I will stick with it...
patsplat 8y ago
That and we're all fools for doubting the scientists diagnosing ADHD https://news.ycombinator.com/item?id=18562975 https://news.ycombinator.com/item?id=18562975 /eyesroll
narrator 8y ago
Economics is in worse shape in some regards. Nobel prize winner Paul Romer has called out most macroeconomics as being completely detached from reality because economic concepts and conditions that are talked about in scholarly papers cannot be identified in the real world by any method. [1] [1] https://paulromer.net/trouble-with-macroeconomics-update/index.html https://paulromer.net/trouble-with-macroeconomics-update/ind...

490 points by zwieback 8y ago

Psuedoscience at it’s finest! I love it! God bless!

> * A mention of the marshmallow test was removed from an early paragraph, since the circumstances there differ from those of other failed replications. I tried digging into the study webpage, but holy cow it's not organized in an easily digestable fashion. So does anyone know what happened with the marshmallow test replication?

ddebernardy 8y ago
It failed. https://www.theatlantic.com/family/archive/2018/06/marshmallow-test/561779/ https://www.theatlantic.com/family/archive/2018/06/marshmall...
- SeanLuke 8y ago
  That much is obvious. But your link is to a different replication study. I want to know why there is an asterisk at the bottom of this article.
- hn_throwaway_99 8y ago
  Reading your article, I wouldn't necessarily interpret it as "failed": > Instead, it suggests that the capacity to hold out for a second marshmallow is shaped in large part by a child’s social and economic background—and, in turn, that that background, not the ability to delay gratification, is what’s behind kids’ long-term success. I.e. it would still seem to say that the capability to delay gratification is tied to economic success, except that's really a confounding variable. It's also very possible that the ability to delay gratification that one gets from being more affluent is also a factor in future success.
  - taeric 8y ago
    Both of those, though, spell failure for the original experiment. It claimed that the predictor was delayed gratification. That is false as false as the predictor of future success being how successful you are in 10 years. :)
  - eezurr 8y ago
    I'm guilty of not reading the article, but I find pleasure in poking apart statistical analysis. There are so many things to consider, like average sugar intake of affluent/non-affluent families at the time of the first test and now. If both classes consume candy, perhaps the affluent have access to tastier candy (maybe important because the wealth distribution in the early 90s was more equal), so marshmallows are not tasty to them. (Note: I haven't had a marshmallow in a long time, but they are no where near as tasty as candy bought off the check out shelves). If present affluent families are educated about sugar (which was less so the case in the early 90s), they would probably feed their children less junk food, so the desire of wanting more of it is not as hard wired in their neurons, or it would taste overly sweet to them. Maybe less affluent families give their children sweets all the time because it shuts them up. Maybe the recipe of marshmallows has changed between then and now (see point 1).
ineedasername 8y ago
I think it's the Stanford marshmallow experiment in delayed gratification: children could take a single marshmallow immediately or wait a certain amount of time and get an additional reward. The results were tracked longitudinally for the participants, with positive life outcomes linked to those who were able to delay gratification as children.
- thatcat 8y ago
  I thought it was later found that the delayed gratification simply correlated with parental wealth and that is why it linked to positive outcomes later in life.
nradov 8y ago
It would be interesting to try and replicate the experiment with other treats. As a child I sort of liked marshmallows, but not a lot. So if I had been an experimental subject, I might have taken the first marshmallow not due to a lack of willpower but just because I didn't care much about marshmallows.

lifeisstillgood 8y ago

It's positive news mostly - they took 15,000 volunteers and ran 28 "big" studies across them, working with original teams to get the nuances right. So they had larger samples than originals and ran experiments multiple times across multiple samples. Where a study failed in one it mostly failed everywhere, and where it worked it mostly worked in all the experiments. So, yes, priming someone with the number 32 won't get them to bet it in the casino, but it does mean that social science can be experimental - you can build and run lab scale social science - a positive !

ALittleLight 8y ago
This is a really optimistic take. I agree with you, ultimately, this is positive because it shows that it's possible for psychology to be done and have widespread meaningful results. In the short term though this seems extremely negative as it indicates that the psychology field and researchers we have today are producing substandard results. Two parts of this article increase my negative feelings. First, they mention that these 28 studies were selected because they were well known and influential. If well known and influential studies only have a 50% reproducibility rate - what about the average study? Another point I find very troubling is that people betting online were able to predict, with high accuracy, which studies were accurate and which weren't. This tells me that the problem is an "Emperor's New Clothes" type of problem and not just a challenge with the fundamental difficulty of the subject since the subject is so yielding to the public's intuition. Instead of a problem with the fundamental difficulty, the problem seems to be with the psychology of the researchers - being unable to diagnose and prevent methodological flaws in their research. This class of failure is much less excusable to me.
- lifeisstillgood 8y ago
  For me the unrepeatable experiments suggest something even more interesting - that unless the originals were flawed or fraud, then something about the people tested has changed. We are all more aware of conmen movies, of fake news, or have read popsci books and now know what to look out for.

aj7 8y ago

They succeed in replicating “half the time.” If they did a second replication trial, I wonder what the overlap of replicated studies with the first replication trial.

ThrowMeDown01 8y ago
They did? > And they repeated those experiments many times over, with volunteers from 36 different countries, to see if the studies would replicate in some cultures and contexts but not others. So different sets of people, and different researchers (the article doesn't say it, but I doubt the same researchers traveled across the globe repeatedly).

viburnum 8y ago

Is there a list of the successfully replicated experiments? I haven't been able to find it.

clord 8y ago
I'd hope for a journal the only publishes replicated results. I think there is limited funding for this right now.

seibelj 8y ago

Even in hard sciences, there are papers that everyone in the niche knows is false, but fear to speak out about or publish a paper refuting it for fear of their career being damaged. My wife is a biochemist, and during her PhD they were working on an area of research with a handful of labs publishing about it. One lab in particular was known for a decent amount of questionable publications, but the PI was a big deal and no one would officially question anything. So they would whisper amongst each other and just ignore that paper (and all the additional papers built on top of it) because they all knew it was bullshit.

akvadrako 8y ago
Not all "hard" sciences. I'm pretty sure there are no papers in physics that are well known to be wrong but people fear speaking out, though I'm open to being proven wrong.
- qudat 8y ago
  Richard Feynman was laughed at when presenting new ideas in physics when he was an up-and-comer. The human ego tends to intervene when careers are on the line.
  - whatshisface 8y ago
    Laughing at true new ideas is actually the opposite behavior flaw of whispering about false ideas that are known to be false but are associated with too much social power.
    - nothrabannosir 8y ago
      It’s the same family: public judgement of a claim influenced by the person making it, rather than its contents. Point being: Feynman’s ideas should have been judged on their merit, rather than Richard’s career. Same for the papers published by the institute mentioned above.
      - 8y ago

ineedasername 8y ago

I don't see how people can doubt this. Positive results get published. Even without p hacking that means a significant number of experiments at the p <0.05 level are published when they're one of the 1 in 20 due to chance. Combine that with p hacking and small samples with small magnitude effects... well replication is going to be a problem. Part of the solution here might be more impetus and incentive to publish negative results.

hn_throwaway_99 8y ago
Agreed. I think all fields should require pre-registration, where studies are announced before they are run to help counter the positive-results bias. Still not a 100% fix (I could imagine a lot of studies that look like they'll show no effect then move into the "oops, nevermind, I knocked over the test tube" variety), but certainly better than what we have now
mamon 8y ago
I’ve always thought that p<0.05 is ridiculously low bar, missing at least one zero after the dot.
- vharuck 8y ago
  It should be more than enough if the experiment's replicated rnough times.
  - ineedasername 8y ago
    Yes, and with larger samples. Too many small sample experiments which, at most, should be considered a "pilot" study, and when results are positive even with p < 0.01 they should still only be considered "suggestive" until replicated, preferably with larger samples.
  - IshKebab 8y ago
    That's equivalent to having a lower p threshold in the first place though.

aligshow 8y ago

Psychology is a political pressure group directed by the American Psychological Association (APA). https://www.washingtonpost.com/news/the-fix/wp/2016/08/07/the-american-psychiatric-association-reminds-its-doctors-no-psychoanalyzing-donald-trump/?utm_term=.e81c978e0ad3 https://www.washingtonpost.com/news/the-fix/wp/2016/08/07/th... The American Psychiatric Association issues a warning: No psychoanalyzing Donald Trump https://www.psychologytoday.com/us/blog/brainstorm/201806/apa-urges-trump-change-immigration-policy-0 https://www.psychologytoday.com/us/blog/brainstorm/201806/ap... The organization is highlighting the psychological danger of family separation. https://www.apa.org/news/press/releases/2018/09/detain-child-migrants.aspx https://www.apa.org/news/press/releases/2018/09/detain-child... Statement of APA President Regarding Administration's Proposal to Detain Child Migrants Longer Than Legally Allowed > “We are dismayed that the administration continues to place the mental and physical health of migrant children and their families in jeopardy. Holding children — even with their parents — in these federal facilities for longer than the 20 days allowed by current law is unacceptable. Research has shown that immigrant detainees are particularly vulnerable to psychological stress. Furthermore, the longer the detention period, the greater the risk of depression and other mental health symptoms for immigrants who were previously exposed to interpersonal trauma. I can't remember any time when the APA criticized Obama the way it is criticizing Trump.

mcguire 8y ago

"But skeptics have argued that the misleadingly named “crisis” has more mundane explanations. ... Third, people vary, and two groups of scientists might end up with very different results if they do the same experiment on two different groups of volunteers." Isn't that the definition of a failure to replicate?

your-nanny 8y ago
I would call it a failure of interpretation. at one extreme one can assume that your sample is unique, and results inapplicable to anyone else. At the other extreme is the assumption that your sample is sufficiently representative so that your results apply to everyone. obviously in most instances it is somewhere between these, and that it "failed" is the basis of new inquiry
anvandare 8y ago
Yes, but the difference is subtle. The theory might still be true after all, but only for a particular subset. "We've tested the theory that 'swans are white', and it turns out the swans in our test were all black". (The conclusion isn't that swans aren't white, it's that some swans are, while others are black.) (add.: the article mentions the 'WEIRD' category. Obviously the applicability/set range of a theory matters. If it only holds for "people who, between 2 and 5 pm on Sunday are walking around in Reading with a bowler hat.". Well, that's nice, but not very useful.)
- ajuc 8y ago
  This is common enough that it seems psychology as a whole is more art than science. It's like alchemy - they noticed some patterns and have some success exploiting them, but the abstractions and explanations they provide are mostly bullshit, so the predictive power is basically 0.
  - anvandare 8y ago
    The problem is the Universe of which it speaks can not be reduced to a handful of beautiful and concise mathematical formulae. A group of humans (or even a single human) is more chaotic and complex in their behavior than a group of atoms ever could be. All Carbon-14 atoms are the same. No two humans, however, are.

Invictus0 8y ago

I'm not sure that there's necessarily a problem with psychology itself, but just that the number of participants in studies is consistently too small. Just like how physicists increasingly have to spend more money to get more accurate results (LIGO, LHC, redefinition of the kg, etc.), psychology needs to realize that 1000+ study participants is necessary to get valid results. Unlike physics, though, where it's easy to show that results are bad when the equipment is bad, the nature of psychology makes it easy to publish spurious results without enough support. When the psych community starts to demand higher N values, you expect fewer, lengthier, more costly, but more accurate results.

HaukeHi 8y ago

I've started a crowdfunding campaign that aims to tackle the replication crisis: Check out https://www.lets-fund.org https://www.lets-fund.org We're raising funds for Professor Chris Chambers at Cardiff University. Chambers is a leading proponent of a new and better way of doing and publishing research, called ‘Registered Reports’, where scientific papers are peer-reviewed before the results are known. This might: -make science more theory-driven, open and transparent -find methodological weaknesses prior to publication -get more papers published that fail to confirm the original hypothesis -increase the credibility of non-randomized natural experiments using observational data The funds will free up his time to focus on accelerating the widespread adoption of Registered Reports by leading scientific journals. This ‘meta-research’ project might be exceptionally high-impact because we can cause a paradigm shift in scientific culture, where Registered Reports become the gold standard for hypothesis-driven science.

jamp897 8y ago

Psychology in the west is really a form of cultural anthropology, in that it’s just measuring behaviors for a specific set of people but not the underline mind directly. So this would make sense IMO that you can’t repeat many of the studies because they’re not grounded in the fundamentals of the mind but a layer abstracted a way by cultures and habits. Akin to strudying a Toyota car and trying to generalize all of physics from it.

pessimizer 8y ago
I have a more cynical view: it's a field that almost exclusively deals in the creation and enforcement of behavioral norms; entirely political, verging on religious in its continual reification of theory-theory. I don't know what that has to do with the West, though. Behaviorism is the only real psychology.
- jamp897 8y ago
  Exactly, the west part is that we tend to think that everything in the west is the whole world or whole field, when Asia is light years ahead in this case in that they focus on the underlining logic mathematically.
- quickthrower2 8y ago
  What's your basis for this view? Any good reads you can recommend?
  - jamp897 8y ago
    The core issue is that you need to establish a norm, but without an absolute point of reference such as enlightenment it’s like trying to walk strait in a dessert which is not possible. But if you bring this point up of going in circles they act just like Plato describes in his Cave Alegory. So it’s not a new issue. I don’t know if anyone has written a book specifically about psychology being a faux science, like eugenics in a way, but someone should.
  - org3432 8y ago

MAXPOOL 8y ago

You would think that computer science can't have replication failures, but it can. And I'm talking about my own field: machine learning. There is so much hype that I suspect that people are pushing papers and trying to actively hide the irrelevance of the methods and algorithms they develop. Artificial intelligence faces reproducibility crisis http://science.sciencemag.org/content/359/6377/725 http://science.sciencemag.org/content/359/6377/725 Reproducibility in Machine Learning-Based Studies: An Example of Text Mining https://openreview.net/pdf?id=By4l2PbQ- https://openreview.net/pdf?id=By4l2PbQ- Missing data hinder replication of artificial intelligence studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-replication-artificial-intelligence-studies http://www.sciencemag.org/news/2018/02/missing-data-hinder-r... >In a survey of 400 artificial intelligence papers presented at major conferences, just 6% included code for the papers' algorithms. Some 30% included test data, whereas 54% included pseudocode, a limited summary of an algorithm.

softwaredoug 8y ago
Yes. As a practitioner it’s very frustrating I think the situation is pretty analogous to other area with replication crises like psychology and nutrition: local variables dominate. Just like our own body’s there’s a lot that just depends on the domain or the specifics of the problem being solved... This has been my experience in a related field (information retrieval). There are trends and best practices, but the market and Impossible to recplicate overhyped academic research reinforce each other.
androidgirl 8y ago
Why isn't more research and research materials open source in the AI world? I don't really understand. If one doesn't have your training dataset or your code, how could they possibly replicate your results?
- __s 8y ago
  So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved Or, more succinctly, code & data are left as an exercise for the reader

buka2009 8y ago

I WANT HACK ALL GAMES HELP PLS PLS

buka2009 8y ago

help I WANT HACK ALLGAMES

buka2009 8y ago

I VERY WANT HACK ALL GAMES PLSS HEELPP

niceworkbuddy 8y ago

We are seeing actual progress in science! This is good.

stuart78 8y ago

"Third, people vary, and two groups of scientists might end up with very different results if they do the same experiment on two different groups of volunteers." This seems to me to be such an important key to understanding this. Considering even myself, I'm not sure I would behave the same way as a subject from one study to the next (pretending there was a way to blind myself). And when you look across regional and global cultural variation, it must have an effect on participants. The results are less headline-getting if they seem to be overly qualified (e.g. we find that midwestern white women with the middle name Susan have a positive psychological response to salsa dancing) but a deeper appreciation for the complexity and diversity of individual psychology would seem superior to the current state of affairs.

deleted 8y ago

[deleted]

londonlover 8y ago

dang, what an absolute horrible time for this to occur as public faith in institutions is deteriorating.

deleted 8y ago
[deleted]
pessimizer 8y ago
Public faith in institutions is deteriorating because they do not do this. The more shit that gets thrown out, the better institutions look.
buboard 8y ago
maybe they are deteriorating because they 've been listening to bad science?

oldgradstudent 8y ago

There's no replication crisis. The replication failures are the only thing that is properly done. We have a pseudoscience crisis. And as others pointed out, it is not limited to psychology.

tmalsburg2 8y ago
There is no easy way to tell whether something is real or pseudoscience. A lot of things that we consider real today where perceived as pseudoscience in their time. Science is largly a gigantic collection of dead ends that look silly in hindsight and finding out what's good science and what's not is an integral part of the scientific process. The idea that there can be a science of just Newtons, Einsteins, and Feynmans is naive.
- viridian 8y ago
  The problem is that replication studies are almost nonexistent for most discoveries. Science is to some extent a business, and very few people with a budget have any interest in allocating resources to replicating the work someone else did, for no to little gain.
- oldgradstudent 8y ago
  If it's not reproducible it's not science. If you p-hacked your way into a statistically significant result, it is not science. You don't have to be Einstein. Even Einstein wasn't Einstein most of the time. But the results you do get must be reproducible.

forapurpose 8y ago

I haven't looked at this study, but prior replication studies have found that the effects were reproduced in almost every study, but that the effects were weaker in the replication than in the original study.[0] That raises different questions than 'psychology studies are all wrong', and I'm still waiting to see a serious discussion of them. So far, what I see is news (studies not replicated) and condemnation (like this article), from which I don't learn much. [0] That's not necessarily a condemnation of the researchers: Imagine a set of studies that, each time they are performed, have a 50-50 chance of producing the strong effects or weak effects. Which studies in that set are going to be the ones that you read about? It could be selection by randomness.

75dvtwin 8y ago

50% replication success rate. I suspect, the authors of the article are a bit too generous to call the original work 'sloppy'. And that the alternative would be: no experiments with conclusions' -- and that's, in their view, is worse. I disagree with that, I think 'being lied to', is worse than to 'not being told'. What if the original work was purposely fake just get a grant from some group (political or commercial)? If that's the case, than the field of psychology is a composed of big number of 'full-stack con-artists'.

deleted 8y ago

[deleted]

gboudrias 8y ago

As someone who is both a developper and a psychology student, your average coder knows about as much psychology as your average psychologist knows about coding. That is to say, not a whole lot. I am saddened but not surprised by psychology's replication crisis. But really it applies to all of academia, psychology is only in the spotlight because some of branches have less than stellar methodology in the first place. Social psychology gets criticized a lot, partly because it's one of the only branches rigorous enough to even be tested! Obviously I'm not in favour of this. But it's a very big machine that needs fixing, and I don't think it starts with tweaking the details; drastic changes are required if we are to gain back the lost credibility. To me, this applies both to social and hard sciences. Since this is a general problem with academia, either your field has already been called out or it has yet to be. I don't believe the current system offers an alternative (yet), though I'm cautiously optimistic about preregistration.

chiefalchemist 8y ago

It seems to me how the test subjects are selected likely matters. I presuming most studies advertise (in some form) for test subjects. That ad appeals to some but not to others, and that leaning could influence the results. In a way, the study starts there, not the lab. If the study were a landing page, with a call to action, the conversions are influenced by the ad. No doubt, 50% replication feels ugly. None the less, the fact is, the studies are slightly different. Maybe in theory that shouldn't matter, but perhaps the reality is it does. Again, per the PPC analogy.

jdoliner 8y ago

There's no such thing as a replication crisis in science, that's just a conflict-averse way of saying that some "scientists" aren't actually doing science. It's not limited to Psychology and this type of conflict-averse approach is a big part of the problem. Other scientists, the "good ones," if there are indeed any good ones left, need to realize that what's at stake is the trust of their profession as a whole. If you're not willing to expose false research in your field, then you're no longer working in science.

dustycat 8y ago
Science (academic work) is a career for many people, rather than a vocation. Rewards include high status, stable income, opportunities to travel, long vacations, etc. You can say it's not science but most "scientists" are just normal people with everyday priorities, rather than Einstein-like people who go on doing science work while working as patent clerks.

njarboe 8y ago

Richard Feynman had a speech warning about this problem back in 1974. Complete with a story about how a student was excited about doing some replication experiments on rats and was told "no, you cannot do that, because the experiment has already been done and you would be wasting time". Others have also been worried about replication problems from the 1960's on. Hopefully some of this worry sticks this time and we can get a better understanding of what is really true in these fields. Physics likes to have very well defined uncertainties on everything they observe and don't like to say something is "true" unless the fact is likely at 5 to 7 sigma. That seems like a good margin. In psychology, 2 sigma is the standard for publishing a result. [1]http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm

cultus 8y ago
P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong. This confusion, and ignorance of the prior probability of hypotheses is the reason why replication rates are so bad. In physics you can get away with not caring about the distinction, because of how accurate the measurements and precise the theories are. That doesn't fly in most other sciences, though.
- jmcgough 8y ago
  > P-values are not good measures for reliability, because they absolutely are not the same thing as the chance that a result is wrong As an undergrad, I even had tenured professors try to tell me that a p-value is the chance that a result is wrong. Most researchers in psych or bio sciences have a weak understanding of statistics, usually taking a single statistics class in undergrad, then a single one in grad.

beefman 8y ago

Paper is here: https://psyarxiv.com/9654g/ https://psyarxiv.com/9654g/

entwife 8y ago
Thanks. It's usually more informative to read the original source.

jcaldas 8y ago

It's somewhat heart-warming to read the comments here about machine learning. I did my PhD in machine learning from 2007 to 2012, and the main reason I left research was because of the widespread fraud. Most papers reported an improved performance over some other methods in very specific data sets, but source code was almost always not provided. Once, I dug so deeply into a very highly cited paper that I understood not only that the results were faked, but precisely the tricks that were used to fake them. I believe scientific fraud arises primarily from two causes: - Publish or perish. Everyone's desperate to publish. Some Principal Investigators have a new paper roughly every other week! - Careerism. For some highly ambitious people, publishing papers comes before everything else, even if that means committing fraud. This happens even with highly successful researchers, who have the occasional brilliant, highly cited paper, but who also publish a lot of incremental, dubious work. P.S. Mildly off-topic, but I love the Ethereum research community at https://ethresear.ch/ https://ethresear.ch/ , precisely because it is so open and transparent! I wish an equivalent community existed for machine learning.

SquishyPanda23 8y ago
> the main reason I left research was because of the widespread fraud. I'm super curious about this. I had a similar experience in another field. Is this something you witnessed going down in person, or did you just develop a strong feeling given a lot of clues?
- jcaldas 8y ago
  > Is this something you witnessed going down in person, or did you just develop a strong feeling given a lot of clues? The latter.
  - SquishyPanda23 8y ago
    Okay thanks. I was unaware that that was happening. Sounds like a frustrating experience.
seibelj 8y ago

crimsonalucard 8y ago

They successfully replicated a study involving replicating other studies.

LifeLiverTransp 8y ago

It might sound conspiracy. So i will try to formulate it, right away like it. Why- if i have exploits and knowledge on human weekneses of the human mind- would anyone share those? Basically, in a world, where the mind-hardware can not be fixed and improved. He who goes full black-hat first rules forever unchallenged, even unpercived? So wouldnt it be reasoNobel, to set free a quack science, to keep those with similar thirst for powerfull knowledge busy? Maybe its tabula rasa time when it comes to psychology. Only replicated studies count, and everything before simply - never existed - and with it, works and titles referencing to it.

stanfordkid 8y ago

The replication failure in psychology is inevitable. The whole field boils down to taking an incredibly complex system (human beings) ... and measuring a few variables on them across a few people. Most of the "statistical" justifications stem from assuming distributions are normal and samples are uniformly collected (both false). This is why I trust the works of clinical psychologists like Jung and Freud so much more seriously than the current approaches. It just makes a lot more sense to observe something complex descriptively, rather than start with a yes/no hypothesis and then test it with a few dozen people.

FiberBundle 8y ago
I'm not an expert on experimental psychology, but I'm fairly certain that the principles of how experiments are conducted in the sense that they control for other factors, choose sufficiently large samples etc., are pretty sound. An often mentioned problem in psychology research is "p-hacking", where researchers manipulate the data in a way that validates their hypotheses statistically, which seems like a more logical reason for a reproducibility crisis than the reasons you mentioned. Also just because there are some problems in the field should not cause you to dismiss science completely in favor of completely subjective speculating about how the mind works, which certainly also has value, but I don't think it's warranted to dismiss actual science in favor of it.
- stanfordkid 8y ago
  But they're just not -- that's exactly my point. If you test a bunch of Harvard undergrads (the most common subject type in these studies) -- that's not a uniform sample. In any way whatsoever. Furthermore, even if you get a high p-value ... your experimental design can introduce so much noise. Even when you do find these high p-value correlations -- they don't really ever say much about "why" ... because they can't. So much of that is the study design. Let's say I make a study on how well people navigate a maze under the influence of alcohol vs. not. You can probably get a high p-value that they do worse while under the influence of alcohol... but it reveals very little about why. So much of that is dependent on the particular maze. In fact I bet I could design a maze specifically to prove whatever conclusion I wanted. That is the whole problem in psychology. There is very little focus given to actually characterizing cognition in any meaningful way. I like the clinical psychologists because they attempt to do exactly that, even though they have less "scientific" findings.

golemiprague 8y ago

The problem is that those type of sciences are now mainly women and women are not looking for the truth. They look for ticking boxes, getting certifications and going through the motions. We need more men in the social sciences if we want those fields to advance in any meaningful way like other fields where men are prominent, whether it is technology or music or anything else you can think about that moves forward all the time.

camelCaseOfBeer 8y ago

The problem is to become a practicing psychiatrist or even psychologist the professional must attain the hallmarks of a professional academic's career. To be a marriage counselor, you need to publish research that revealing ground breaking insights into the human experience. Go figure, we get shoddy science from people who had little interest in science to begin with. There needs to be distinct separate paths in the field for professionals and academics.

Voyage_wanderer 8y ago

Assuming we are living in a simulation. Every our attempt of understanding rules and breaking through the simulation should be considered as a security breach by another side. So it gets fixed. Hence no replication. It had been fixed, dumbs... it’s my theory and I will stick with it...

patsplat 8y ago

That and we're all fools for doubting the scientists diagnosing ADHD https://news.ycombinator.com/item?id=18562975 https://news.ycombinator.com/item?id=18562975 /eyesroll

narrator 8y ago

Economics is in worse shape in some regards. Nobel prize winner Paul Romer has called out most macroeconomics as being completely detached from reality because economic concepts and conditions that are talked about in scholarly papers cannot be identified in the real world by any method. [1] [1] https://paulromer.net/trouble-with-macroeconomics-update/index.html https://paulromer.net/trouble-with-macroeconomics-update/ind...