17 ms·

If you think psychological science is bad, imagine how bad it was in 1999

275 points by ruaraidh 5y ago

Tarsul 5y ago
I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet scenario (and probably with participants who are not diverse).
- camjohnson26 5y ago
  Appeal to authority is a big reason. The academic leaders in the field were incentivized to find breakthroughs, or to treat any experiment they ran as a breakthrough, because that’s where their influence and reputation came from. People are so used to trusting experts that they didn’t realize that finding the truth wasn’t the primary motivator for some people.
- enkid 5y ago
  Tangent, but the Stanford Prison Experiment is an awful example of science. It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point. I hate that it's often the only psychological experiment many people are familiar with.
  - seph-reed 5y ago
    Calling it an awful example of science seems a bit extreme. There is definitely something to learn from such an experiment, albeit not what was intended.
  - mateo411 5y ago
    What about the Milgram experiment? This is also a very well known psychological experiment. Was the science behind the Milgram experiment rigorous?
    - enkid 5y ago
      From what I have read (I'm not a psychologist, so take everything I say with a grain of salt), the Milgram experiment had some major ethical issues and doesn't meet modern standards for statistical evidence. There's also accusations that the data may have been manipulated and the results don't directly support the claim that Milgram was making. For example, a large number of participants may have been aware that the actor wasn't really suffering when they administered the punishment. This would seriously skew the results.
    - jbullock35 5y ago
      Milgram's obedience studies didn't involve randomized treatments, and he had small numbers of subjects (typically around 40) in each of his many conditions. On the other hand, Milgram's investigations were serious, systematic, and in good faith -- which makes them worlds better than the Stanford Prison Experiment. Another reply in this thread suggests that "a large number of participants may have been aware that the actor wasn't really suffering when they administered the punishment." I've studied the topic and found no evidence of this point. In addition, the claim is hard to square with many subjects' reactions -- for example, their nervous laughter and their frequent protests, even as they continued to deliver what they thought were harmful electric shocks. See https://news.ycombinator.com/item?id=25928569 https://news.ycombinator.com/item?id=25928569 for more on efforts to replicate Milgram's results. Tl;dr: there is no equivalence between the Stanford Prison Experiment and Milgram's work on obedience. Milgram's work was superior.
  - 99_00 5y ago
    >the Stanford Prison Experiment is an awful example of science. This may be true, but I don't think the evidence you give supports your assertion. >It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point "prove a point" is the hypothesis "created the conditions" is the experiment "collect the data to prove that point" is the observation
    - enkid 5y ago
      No, an experiment should not be set up to prove a hypothesis, it should be set up to test a hypothesis. In one the hypothesis is falsifiable, and in the other it's not.
      - 99_00 5y ago
        If people didn't act the way the experimenter expected wouldn't that disprove their hypothesis in this case?
        enkid 5y ago
        No, because people did behave differently than he expected, and he coached them to behave in line with his expectations. He wasn't a neutral observer, he was a guiding force and the superintendent of the pretend prison. This is what I mean when I say it wasn't an experiment.
- naasking 5y ago
  The problem is not just with "belief", but in the process itself. Non-replicable studies are actually cited more than replicable studies: https://advances.sciencemag.org/content/7/21/eabd1705 https://advances.sciencemag.org/content/7/21/eabd1705 Science journalists probably are vulnerable to the same thing influences that lead scientists to do this, except they have even less review on their claims and so they become pop culture sound bites. As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising, and per the above link, reviewers apply lower standards on these papers in the hopes of finding something truly interesting and/or exciting. Not just in the results mind you, sometimes papers also apply a novel methodology that might be worth wider discussion. I'm not sure that's worth the reduction in credibility though.
  - theptip 5y ago
    > As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising This seems the likely explanation; I saw a paper recently that showed that lay people can predict what will replicate with above-chance accuracy[1]. I imagine scientists are even better than lay people at this. So non-replicable results are almost by definition surprising (i.e. they are hypotheses that don't match our current model of how the world works), and surprising results are definitely better news than unsurprising results. [1](https://journals.sagepub.com/doi/full/10.1177/2515245920919667 https://journals.sagepub.com/doi/full/10.1177/25152459209196...)
  - edanm 5y ago
    > As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising I mean, a kind of well known thing is that in general, something false can be more interesting than something true, since it has more degrees of freedom. You can make up anything you want - the truth has to conform to what's actually real.
- TaupeRanger 5y ago
  Same reason we believed eggs will give you heart disease. Science reporting is terrible and the general education system doesn't teach rational skepticism, it teaches unconditional trust of intellectual authority.
  - reedjosh 5y ago
    Indeed! And in reality eggs are basically a superfood.
  - denimnerd42 5y ago
    every mainstream thought about diet is untrustworthy to me due to the above type of example :/
    - loudtieblahblah 5y ago
      it is. mainstream diets will tell you: * saturated fats are bad (lies told us by a crappy Ancel Keys study, promoted for decades by processed food companies (like Kellogs) ran by Seventh Day Adventists who were convinced "meat led man to dangerous impulses and temptation") * polyunsaturated fats are good. The American Heart Assocation had an article up for years that went as far as to claim Omega6s are heart healthy. They only recently took it down this year. But we know they're inflammatory and we know we're consuming 25-100x more Omega6s than we ever would before the industrial invention of seed-oils being shoved into every product imaginable (bread, cereal, granola, anything that comes in a box, feed given to animals meant for meat production) here's a webmd article on it: https://www.webmd.com/heart/news/20090126/expert-panel-omega-6-wont-hurt-heart https://www.webmd.com/heart/news/20090126/expert-panel-omega... \ * sunlight creates high cancer risk (ignoring that cancer is unlikely, treatment if caught early has a high survival rate, and the risk of not having vitamin d throughout your life risks far more likely autoimmune issues, depression, anxiety and even certain cancers and inflammatory disease). * sugar is good for you. Sure, they'll specify processed sugars are bad for you or "added sugar", but common wisdom will accept a NET (subtract fiber) 200-300g carb diet as acceptable. Grain is still often listed as the most important and largest part of the food pyramid. The reality is - all mainstream health advice, including that which you'll get from your doctor who got a whole single nutrition class in school, ensures that processed foods don't loose business on the front end and the medical/pharma industries don't lose money on the back end. even in the push for a more vegetarian/blue-zone diet world - they're doing so by promoting meat alternatives like "Beyond Meat" which is chock full of so much seed oil and other processed substances, it's mainstreaming vegetarianism-as-fast-food. McDonald's burger.. is still a McDonald's burger and you shouldn't be eating it. If a factory isn't making it at scale, shoving it in a box, branding it and ensuring you don't have to spend any time making/cooking/preparing whole, fresh foods (those pesky things that tend to have short shelf lives and are costly to Ag businesses), then your PCP, the government, most food businesses, your medical insurance company, absolutely no one of any kind of "authority" isn't going to promote it highly. They'll do ANYTHING except remove seed oils. They'll make your potato chips out of broccoli and carrots and still drench them in sunflower or canola oil. They'll reduce the salt. they'll make shit out of beets. And still manage to make it horrible for you. The MSM regurgitates "health" info regarding diets in a way that acts as advertising for these orgs.
- batguano 5y ago
  I assume you are referring to "The Stanford Prison Anecdote"?
- hpoe 5y ago
  They are believed because they are sensational, they are titillating, and it gives people the excuse for their bad behavior by believing that everyone would be a monster in the right circumstances.
- Blikkentrekker 5y ago
  Who is “we”? This has been criticized since 1999 and far longer back. How long ago is the Rosenhan Experiment again? Dare I say that a majority have always held a dismissive, critical view of such matters. But of course, those that hold dismissive views of it are not the ones who work in such fields, and certainly not at the top ready to implement changes, so it can continue to persist and go on despite of being highly criticized. At least when I studied physics at a university around what must have been 2005, most of the students and professors there were highly critical of softer science and it often came up that some of these papers popped up and were viciously criticized for clear and obvious systematic errors in the methodology.
- loudtieblahblah 5y ago
  same reason we believe observational studies in nutrition, or worse - studies on animals. if you have the weight of peer review or at least a well documented study, then the media runs wild with it's claims, it gets shoved into textbooks, then governments shape policy on those claims, corporations and medical practices sell gimmicks, books, supplements, therapy and plans of action to heal you... it all becomes lies, half-truths, bad data all just repeating itself ad naseum until "truth" is established in the public consciousness. Quacks on the web, the American Heart Association, your local doctor's office will all pedal garbage based on the bad data. And once it's well established as true, backing away from it is hard because it's become so woven, institutionally. This is why people think saturated fat and sunlight are bad or at least a net-negative. Even modern medicine, psychology and nutrition sciences all have horrible replication crisis's and we're no better in rejecting the nonsense now than we were then.
- aaron695 5y ago
  SPE was needed at the time because everyone knew the Germans were like us, but needed a framework to say it. It also opened up the idea the Japanese were like us too. It allowed us to say what we believed and build on that. It's not science. But science isn't the only way forward.
  - acchow 5y ago
    Can't we just look at history and conclude that we are "just like the Germans"? Why do we need to shroud it in pseudo-science?
brnt 5y ago
Psychological science is now thoroughly privatized. Don't read papers if this subject is of interest to your, look al big tech, ad tech, troll factories, influencer science, media consolidation, for-pay research-charities (great rabbithole to dive into btw). Academics are thoroughly out of the loop on this one, as are we all incidentally.
- CactusOnFire 5y ago
  I became a Data Scientist because in 3rd year, I found the OKCupid data blog and realized their research had methodological soundness these "20 undergraduate lab students forced to be there" research never would. You trade 'shot-in-the-dark lab experiments' for 'a clear and obvious agenda'. The trick is to just make sure the agenda isn't morally reprehensible.
  - mistrial9 5y ago
    more than one third of the entire world population believes that OKCupid is "morally reprehensible" (!)
    - stickfigure 5y ago
      This is an interesting statement. Do you have links to something more I can read? Who thinks dating sites are morally reprehensible?
    - burnte 5y ago
      Religious biases don't count as valid moral objections. Religion has a strong negative effect on critical thinking skills (1, 2). If you dogs/pigs/shellfish/mixed-fabrics are inherently unclean or that women are property, your objections to a dating website are probably not going to have seriousl scientific ramifications. 1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ 2. https://www.google.com/search?q=Religion+has+a+strong+negative+effect+on+critical+thinking+skills https://www.google.com/search?q=Religion+has+a+strong+negati...
23B1 5y ago
"Science"
jcims 5y ago
I have no basis to argue against the article, but I would think there is more incentive to game it now so I wouldn't take it as a given that things have improved.
- ouid 5y ago
  Bonferroni effects have, at least in the popular consciousnes, become much more front and center. I wonder how many people on the stre can define p-hacking. Most educated people know about the reproducibility crisis, and I would wager that around half of them understand the cause. That seems like it might create a better environment.
  - SV_BubbleTime 5y ago
    > around half of them understand the cause I’m in the half that doesn’t know, apparently. I considered that with hard sciences, the cost, time, equipment, conditions, etc were limiting factors. I think this paper on subatomic particles changing into antimatter is interesting, it was observed in a unique facility, once, under some condition that can never repeat, etc. Just kinda have to take your word for it. As to soft sciences… I really don’t know there. Give me a hint?
    - ouid 5y ago
      Bonferroni effects. In the parent article referred to as researcher degrees of freedom, in pop culture referred to as p-hacking. In machine learning, it's referred to as overfitting. They are all the same phenomenon, of having more than one hypothesis, and then using statistical measures that assume that you only have one hypothesis to claim confidence. Journals only publishing positive results is an example of this phenomenon. Changing your hypothesis to match your data is another such example.
tgv 5y ago
And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.
- thechao 5y ago
  Which why P-scores need to be proportional to some nontrivial inverse factor of the number of experiments done in the field. Are there 10000 researchers doing 20000 experiments per year? Take 20000, multiply by the number of years we expect an academic to do hands on work (30?); invert: you need a P-score better than 1:600000.
  - pytonslange 5y ago
    This! Bonferroni correction across all conducted studies/analyses. I will suggest this next time I am reviewing a paper with crazy claims and shitty statistics.
    - yxwvut 5y ago
      I know this is mostly a joke, but what you really want is Benjamini–Hochberg correction, unless you want to prevent even a single false discovery in all of science. FDR vs FWER
- prof-dr-ir 5y ago
  I have a similar horror story. A medical student worked hard to analyze, say, 40 x-rays out of hundreds available. He found no significant evidence for some hypothesis. When he told his supervisor, the reply was: "Well then you should just analyze some more x-rays. I'm sure you'll have a statistically significant result at some point."
- chromaton 5y ago
  Somehow one of the 3 experimental groups has 16 people, whereas the other groups have 14 people. Wonder why...
m12k 5y ago
When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?
- throwawaysea 5y ago
  There was a past discussion about peer review on HN: https://news.ycombinator.com/item?id=20607259 https://news.ycombinator.com/item?id=20607259 Another related discussion was about the grievance studies scandal, which also touches on peer review and academic rigor in journals: https://news.ycombinator.com/item?id=18127811 https://news.ycombinator.com/item?id=18127811
- javert 5y ago
  The incentive system that academic scientists live under explains why they don't push for studying metascience. If that's going to happen, it has to come from outside the government-science complex. Good luck with that...
- ryanmarsh 5y ago
  Similarly, I heard someone the other day assert that no one had done an double blind placebo controlled trial of the effects of FDA regulation.
  - hannob 5y ago
    That is probably true, but do you have any proposal how such a trial should be done? RCTs are good when they can be done and I'm all for doing more of them and too often there's no good excuse for not doing them. But at some level things just get impractical.
- logicchains 5y ago
  >When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.
mykowebhn 5y ago
I worked at a well-known clinical psychology lab at an Ivy League school many years ago. There was so much "gaming" of the results. For example, our PI didn't want to include a subject in our study because his scores weren't elevated enough, and our PI was worried that his score wouldn't drop enough which would adversely impact our results. Another example: our active treatment therapists knew exactly what they were treating for, and our study was measuring improvements in the condition that was being treated. However, the control therapists had no idea what they were treating for, and we purposefully kept this information from them!
- kizer 5y ago
  Hence the “replication crisis”. Replication of experiment and differences in outcome highlight the “core” result. When the results are just totally different, obviously there was “gaming” or just a bad experiment.
- mattkrause 5y ago
  The first example doesn't seem that bad. No intervention cannot reduce something if it isn't there to begin with. The screening should be disclosed in the methods (and, ideally, pre-specified), but you do need to account for floor/ceiling effects somehow.
antiterra 5y ago
The part that confused me the most, as someone who has never done research science, was how the questions were casually categorized as gender, ethnicity or no-identity salient. If someone is from Brazil and is of second or third generation Japanese descent, how much of the questions are 'salient' to Brazilian identity vs Japanese? There's an unspoken implication that part of the 'good at math' stereotype relies to some degree on speaking a non-English language at home, which I don't think is a safe assumption at all. > In the Asian identity-salient condition, participants (n = 16) were asked (a) whether their parents or grandparents spoke any languages other than English, (b) what languages they knew, (c) what languages they spoke at home, (d) what opportunities they had to speak other languages on campus, (c) what percentage of these opportunities were found in their residence halls, and (f) how many generations of their family had lived in America.
- dash2 5y ago
  But this would bias results towards zero, no?
publicola1990 5y ago
Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.
- programmarchy 5y ago
  I'd bet that even more dubious science is packaged by large corporate interests. Take Big Tobacco, who had doctors saying smoking is healthy, or the sugar industry, corrupting guidelines and policy on coronary heart disease [1]. And medicine is not immune either, with financial conflicts of interest and pharmaceutical sponsorships correlating with outcomes. [2] [1] https://www.npr.org/sections/thetwo-way/2016/09/13/493739074/50-years-ago-sugar-industry-quietly-paid-scientists-to-point-blame-at-fat https://www.npr.org/sections/thetwo-way/2016/09/13/493739074... [2] https://en.wikipedia.org/wiki/Metascience#Medicine https://en.wikipedia.org/wiki/Metascience#Medicine
  - reedjosh 5y ago
    Careful now... You might make someone skeptical of vaccines. /s
  - derbOac 5y ago
    I don't disagree with you, but might provocatively suggest that federal funding in the US vis-a-vis a large university could also be considered a large corporate interest at this point.
- Sr_developer 5y ago
  There is a cottage industry of what you may call,if you feel uncharitable, bullshit peddlers: Simon Sinek Cal Newport Charles Duhigg Mark Manson Ryan Hollyday Malcolm Gladwell The list is endless to be honest, they are each different on their own way but they have the following common points: - They are on this for the money, so expect them to be always pushing their books, products, next book, next tour, next program.Hustling, hustling,hustling. - Their grandiose pronouncements with little or not serious backing. - Their unwarranted sense of speaking from a position of authority - The over-simplication and stupid generalization of what it is messy, complex and very much unique.
kizer 5y ago
Interesting how as I began reading the excerpts from the paper I had to stop myself and realize that the author himself had ironically conferred a bias against the paper, LOL. I was primed to find all the flaws. I think psychology and sociology are legitimate and worthy studies, but they run into issues with the scientific method itself due to the ambiguous and “high-level” nature of their concepts and theories. It’s hard to create meaningful, repeatable experiments. So perhaps it should be emphasized how important it is to put effort into constructing experiment... and in particular keeping the subjects unaware of what it being tested. There are probably many great examples of experiments done.
failwhaleshark 5y ago
If it is not a hard science, then it is likely a variant of philosophy.
- ouid 5y ago
  I think you could effectively argue that mathematics is a branch of philosophy.
  - failwhaleshark 5y ago
    You can construct proofs for mathematics, but not for the existence or refutation of a divine creator.
    - Afton 5y ago
      I think you have a narrow view of philosophy. Very little modern philosophy has anything to do with that kind of question.
      - failwhaleshark 5y ago
        Nope, you're projecting and using a strawman. That's not the point. :)
    - bfors 5y ago
      True, but with math you also eventually get down to axioms that can't be proven.
      - rajin444 5y ago
        I don't think we know if we can know everything or not. That seems like _the_ question to answer.
        kirubakaran 5y ago
        https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
andi999 5y ago
Actually I do not think that paper is that bad compared to majority (well I do not think Psychology currently is a science)
deleted 5y ago
[deleted]
29athrowaway 5y ago
At least they have not independently rediscovered integration yet. https://care.diabetesjournals.org/content/17/2/152.abstract https://care.diabetesjournals.org/content/17/2/152.abstract https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf
- anthk 5y ago
  I don't get the downvotes.
stevetodd 5y ago
Reading the other comments about the lack of integrity in scientific research that they’ve witnessed, is it any wonder why people might be skeptical about important issues like climate change, vaccines, masks, etc? There’s great research being done and a larger part of the population is becoming more and more skeptical because lies are being published and touted. Science needs to clean its house.
- hypersoar 5y ago
  The skepticism about climate change is due much less to issues with the scientific community that than it is to decades of propoganda funded by people who fear losing money to efforts to fight it.
  - SV_BubbleTime 5y ago
    Are you suggesting there is only money, power, and propaganda by deniers of climate change? Don’t you think it’s possible there are people exaggerating it for money, and power? I don’t think ”they use propaganda and everyone who disagrees is just wrong” is a good argument when the topic is “we know in other areas there are issues with the scientific community so why not this one”. We can still have the right answer and have gotten there the wrong way.
- fullshark 5y ago
  Independent replication needs to be incentivized, either as part of tenure process or have grad students do it as part of getting a PhD or something. Then the results are published in a journal of replication studies.
theknocker 5y ago
In 1999, they at least thought they needed a model (as opposed to policy conclusions), i.e. they were at least actually attempting science. I don't know what this article is supposed to be or why I should care about some random guy's random polemic about some random paper.
Blikkentrekker 5y ago
I went through it in a clinical sense around that time and this was at the time that I had no legal right to demand to read reports about me for correction, which came later, and when I did it, the tunnel vision in it explained many things about the conversation. What I remember most is that the clinical psychologist kept fishing as to why I covered my face with my hair and I kept saying that there is no reason other than gravity and that I cannot control that my hair obscures parts of my face and the report contained that I did it on purpose to hide my face, which I'm fairly certain I did not, but it seemed that this was really what the clinical psychologist settled on early and continued to search evidence in support of.
vlovich123 5y ago
This is a flawed logical argument. Imagine this article from 1830: "If you think phrenology is bad, imagine how bad it was in 1810". Rinse & repeat for the appropriate time frames for alchemy, astrology, or any other field that tried to misapply science. Just because a field is studied for a long time or tries to apply the scientific method doesn't lend credence to the approach. All it means is, at best, we've managed to toss some things that are now obviously wrong/flaws. In 20 years we'll be doing the same to things we "know" today OR the flaws will remain because we don't have the math/science to demonstrate the flaws more obviously & there's social pressure to keep "building" (even if the foundation is flawed). However, as we should all be aware, false knowledge grows exponentially more quickly than our true understanding of the universe because our imagination is limitless.
- vlovich123 5y ago
  To anyone downvoting me, consider this HN article from not too long ago: https://news.ycombinator.com/item?id=27489927 https://news.ycombinator.com/item?id=27489927 The general premise with these studies is that if an effect size is real, then a preliminary study would show something interesting. To my knowledge, statistically that is a nonsense argument. Small sample sizes suffer from various small sample effects to the point that you can't predict either way (otherwise there wouldn't be a point in doing a larger study). To add insult to injury, all of these kinds of studies are only on local college students, which further invalidates any potential information gleamed from a preliminary study. TLDR: The way science is done in the social sciences is fundamentally flawed & the fact that limited funding ensures that's the case doesn't excuse that a significant enough part of the body of knowledge isn't reliable.
loloquwowndueo 5y ago
Gee you talk about 1999 like it was more than 20 years ago. Oh wait … (damn I’m old)
dfxm12 5y ago
Do we have to imagine? I'm sure many scientists in the field today were working in 1999. There's certainly enough data from the time to look at this in some level of detail, but Cherry picking an article and pointing out its flaws doesn't really prove much about the state of the field back then. Maybe if the author mentioned that this article was highly regarded back then, there would be a point, but for all we know, the article was thought poorly of at the time and contemporary scientists just thought it slipped through the cracks. It also doesn't talk about new controls in place today that would prevent a similarly poor article from being published, or even a system of "retracting" poor articles. I don't really trust that everything being published today is without flaw. After all, the other examples of bad science given are fairly recent.
gregoreous 5y ago
Those dummies. They had perfectly good time machines in 1999 and they didn't even use them to visit our enlightened era. I bet they did all their calculations on some old Windows 95 system instead of investing in solid multiple core machines. Still, they don't hold a candle to Gregor Mendel failing to incorporate DNA in his genetic work, if you can believe it.
splithalf 5y ago
“Don’t hate the player, hate the game.” Indeed. It is a scale problem. We have too many producers of research, too few destroyers of research, like Gelman. Show me the incentive and I can tell you the outcome. Encourage the whole world to become “experts” and then be amazed as the reverence and trust in expertise is devalued. That’s us.
torginus 5y ago
What I don't understand is psychology research (at least in academia) does not seem to have moved behind the "we locked three dozen college kids in a classroom and had them perform some bizarre acts, through which we hope to pierce the veil of human nature" - style research. I feel like if something good could have come out of the social media age - is that we have documented the natural behaviour of a vast number of people over long stretches of time. I think this is the sort of invaluable data, that has the potential to advance the quantitative understanding of human nature.
- B-Con 5y ago
  I've always been perplexed by this, too. Aren't the two most important aspects of the research the data set and the study methodology? Why on earth would you skimp so heavily one of them? I don't work in the sciences, but this kind of nonsense doesn't exist in the "actual" sciences. Physicists spend loads of money producing just the right experiment conditions and documenting the manner the experiment was created in. The dataset is incredibly important and very rigorously examined. But in psych, the dataset is basically an after thought. "Oh by the way, we chose a small handful of kids who happened to be free at that time, with no reason to believe there's any geo, social, educational, political, or ethnic background diversity, it probably cost us like $200 plus some pizza. Now let's print the results in $5 million worth of textbooks for a few decades!" I don't buy the funding argument. A professor probably costs the university 100-150k/yr and will be working on a small handful (2-6, ish?) of projects. Buying an hour of a subject's time for a study must cost, what, $30/hr? Shouldn't they be allocating a minimum of $50k in funding for the actual research, and dropping at least $10k for a good dataset? I don't buy the argument that most experiments don't yield good results so the university is wary of funding them. At a minimum they should follow up a cheap test with promising results with a real experiment that has actual funding before everyone gets all excited about it.
deleted 5y ago
[deleted]
jrochkind1 5y ago
The realities of science as actually practiced rather complicates the religion of "trust the science!" (which usually actually means "trust the scientists!")
- pjmorris 5y ago
  “In God we trust. All others must bring data.” - W. Edwards Deming.
  - fullshark 5y ago
    I don't even trust data anymore
    - pjmorris 5y ago
      Agreed. It's not like the good old days when data was about one physical process that didn't have too many downstream impacts.
- colechristensen 5y ago
  The science-as-religion people think that science delivers truth in the same way any other theology does. I took a philosophy class filled with people in science degree programs and a few of my classmates were often vocally upset about how nothing was certain in philosophy and everything had multiple sides to it. That was very eye opening, many of these people were soon to be graduated and through their entire educational career they had only been exposed to Truth to the extent that being shown debate and disagreement on a topic made them upset. You're not supposed to "trust the science" you're supposed to trust the process to approach the truth. If you can't read multiple arguments on the same topic and analyze them, you really don't get it at all (and waaay too many people with degrees can't do this).
  - SV_BubbleTime 5y ago
    > You're not supposed to "trust the science" you're supposed to trust the process to approach the truth. If you can't read multiple arguments on the same topic and analyze them, you really don't get it at all (and waaay too many people with degrees can't do this). Agree entirely. I’ll add that I go slightly further. If you can’t take your pet topic, and can’t make even a slightly good faith argument against yourself, you have no business with strong feelings on it. I have a wedge issue topic I am an expert on. I could argue against myself, both effectively, and in an actual compromise that no one wants. Yet… people who argue against “my side” are constantly using complete bullshit science from the 1980/1990s when governments literally weaponized depts and ivy leagues to push for “evidence” to support their desired policy changes. These people now tell me to “trust the science”, “I’m sure this researcher at Harvard is wrong and you’re right”, and ”this article from CNN / FOX / VOX / WAPO proves you are wrong and I refuse to consider they have an agenda”. Worst part that there is no shame of willful ignorance, they “trust” the people they claim can’t be wrong, simple, done. Why should they bother to acknowledge another side - if they do it means everything else needs reevaluation too.
PatentlyDC123 5y ago
Is there a method of flagging papers in scientific journals that have been criticized or refuted, e.g., by later studies or proven inability to replicate the data? In legal research services, like Lexis Nexis or Westlaw, many cases are "flagged" when a later case or statute reverses, narrows, or otherwise affects the earlier case. This system warns lawyers that they may not be able to cite the flagged case in their current work. Of course legal research services also come with their own issues and costs; some of which are likely associated with this system.
- walrus 5y ago
  A journal can publish a retraction. The website Retraction Watch[1] aggregates these retractions and provides a database that you can query. Reference management software like Zotero[2] can use this to monitor your collection of papers and notify you when one is retracted. [1] https://retractionwatch.com/ https://retractionwatch.com/ [2] https://www.zotero.org/blog/retracted-item-notifications/ https://www.zotero.org/blog/retracted-item-notifications/
  - earthscienceman 5y ago
    Yeah. They could. But few (zero?) studies are retracted for the sake of being proven incorrect later. And, to be far, it would be ridiculous. Imagine having your career nullified because when you're 60 some major break through shows that your studies aren't relevant anymore. Your work was good when you did it, but now there's something new. It's kind of the definition of scientific progress. However, as a counter example, in my very narrow specialty there is a well known lab that has produced highly cited bogus studies. I've personally published opposing results and said, "these studies are wrong for these reasons" using almost exactly those words. Should they be retracted? Absolutely. Will they ever be? No. Because, of course, the publisher and the authors just point the finger back at me and say "no, you're wrong!" and that's more than enough to keep the vague debate going.
- hypersoar 5y ago
  An issue that comes to mind is, ironically, authority. Who decides when a paper has been discredited? I can see all sorts of incentive problems with such a system. On the other hand, Westlaw is only cataloging what has already been decided. If the Supreme Court overturns a prior case, then it's overturned whether you agree with the reasoning or not.
jedberg 5y ago
I studied CogSci in the late 90s, which involved taking psychology courses. Every psych course that ended in zero required you to participate in psychology experiments. The profs like to say that 18-22, mostly white/asian, mostly rich kids are the most studied group of people in the USA. Or course we now know that a ton of psychology research doesn't actually apply to people outside that small narrow window of people.
- thaumasiotes 5y ago
  It doesn't apply within that window either.
- philwelch 5y ago
  The social sciences have a term—“WEIRD” (Western, Educated, Industrialized, Rich and Democratic) to describe the population that most psychological and social science research has studied the most deeply. As the acronym implies, these populations are not normal across human history or even the modern world. One of the fascinating concepts in abnormal psychology is the notion of a “culture-bound syndrome”—a mental illness that only occurs in a specific cultural milieu. I wonder how many mental illnesses are actually culture-bound syndromes of WEIRD culture?
PicassoCTs 5y ago
My 5 cents. There is not one but two psychological sciences at the moment. One is public, publicly funded and in very bad shape, with most of its results being not reproduceable (reproduction crisis), and a partial destruction happening via neuro-sciences. Who destroy, but do not offer large-scale replacement theories, that could encompass the whole species and are not in contradiction to other neuro-science results. And then there is the second faction (Disclaimer: I can not proof what is deduced after this disclaimer.) There are several cooperations and at least one government, which had the chance to large scale collect data on the population. This data is a psychological gold-mine, if explored properly. One could query such a behavioral database and more important - enact virtual experiments. Out of all male humans, who curse in front of the tv in the evening, filter out those who get into a car accident, plot the increase in cursing in front of the tv. If taken to the extreme, this new, data-mining behavior sciences, could create a agent based model of the species in all variations and collect data only to check the expected outcome of a societal change against the real outcome, with spot samples. I have my own little pet theories, how humanity would look to this privatized psychology, but i digress. I think, academic psychology should have full access to all cooperation databases that contain behavioral data.
- SV_BubbleTime 5y ago
  > I think, academic psychology should have full access to all cooperation databases that contain behavioral data. No thanks. We already have quack science from the 1950s still driving policy discussions on wedge issues today. I don’t want more convincing shit, I want less shit.
fungiblecog 5y ago
This is what happens when everyone is told they're awesome and nobody is allowed to fail
anthk 5y ago
1999 to me was almost yesterday.
PostThisTooFast 5y ago
What a BS headline
wolverine876 5y ago
> If you think psychological science is bad I know it's often repeated, but is there evidence that "psychological science" is worse (by some measure and to a significant degree) than other social sciences? As long as we're talking about science, let's look at evidence! ;) Anyone studying human behavior, on any scale, has the added challenge that it's so complex that you can't really isolate simple mechanics, as you can in physics or chemistry. People are far more complex than a molecule, or even a billion molecules. BTW, I'm aware of the well-known cite, the NY Times article on the 'reproducability crisis' from a few years ago. Few seem to have read the article: The results were reproduced, but many were at lower strength than the original research. That's important, but it's not like the researchers just complete missed and results were arbitrary.

275 points by ruaraidh 5y ago

I still don't understand why we "believed" studies that claimed knowledge about typical behavior and subconscious decisions while those studies were only based on a single experiment with limited participants, e.g. Stanford Prison Experiment. These type of studies should ALWAYS be looked at critically and not just because they fail to reproduce but because they are based on a very small sample size in a very discreet scenario (and probably with participants who are not diverse).

camjohnson26 5y ago
Appeal to authority is a big reason. The academic leaders in the field were incentivized to find breakthroughs, or to treat any experiment they ran as a breakthrough, because that’s where their influence and reputation came from. People are so used to trusting experts that they didn’t realize that finding the truth wasn’t the primary motivator for some people.
enkid 5y ago
Tangent, but the Stanford Prison Experiment is an awful example of science. It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point. I hate that it's often the only psychological experiment many people are familiar with.
- seph-reed 5y ago
  Calling it an awful example of science seems a bit extreme. There is definitely something to learn from such an experiment, albeit not what was intended.
- mateo411 5y ago
  What about the Milgram experiment? This is also a very well known psychological experiment. Was the science behind the Milgram experiment rigorous?
  - enkid 5y ago
    From what I have read (I'm not a psychologist, so take everything I say with a grain of salt), the Milgram experiment had some major ethical issues and doesn't meet modern standards for statistical evidence. There's also accusations that the data may have been manipulated and the results don't directly support the claim that Milgram was making. For example, a large number of participants may have been aware that the actor wasn't really suffering when they administered the punishment. This would seriously skew the results.
  - jbullock35 5y ago
    Milgram's obedience studies didn't involve randomized treatments, and he had small numbers of subjects (typically around 40) in each of his many conditions. On the other hand, Milgram's investigations were serious, systematic, and in good faith -- which makes them worlds better than the Stanford Prison Experiment. Another reply in this thread suggests that "a large number of participants may have been aware that the actor wasn't really suffering when they administered the punishment." I've studied the topic and found no evidence of this point. In addition, the claim is hard to square with many subjects' reactions -- for example, their nervous laughter and their frequent protests, even as they continued to deliver what they thought were harmful electric shocks. See https://news.ycombinator.com/item?id=25928569 https://news.ycombinator.com/item?id=25928569 for more on efforts to replicate Milgram's results. Tl;dr: there is no equivalence between the Stanford Prison Experiment and Milgram's work on obedience. Milgram's work was superior.
- 99_00 5y ago
  >the Stanford Prison Experiment is an awful example of science. This may be true, but I don't think the evidence you give supports your assertion. >It was a researcher who wanted to prove a point and created the conditions to collect the data to prove that point "prove a point" is the hypothesis "created the conditions" is the experiment "collect the data to prove that point" is the observation
  - enkid 5y ago
    No, an experiment should not be set up to prove a hypothesis, it should be set up to test a hypothesis. In one the hypothesis is falsifiable, and in the other it's not.
    - 99_00 5y ago
      If people didn't act the way the experimenter expected wouldn't that disprove their hypothesis in this case?
      - enkid 5y ago
        No, because people did behave differently than he expected, and he coached them to behave in line with his expectations. He wasn't a neutral observer, he was a guiding force and the superintendent of the pretend prison. This is what I mean when I say it wasn't an experiment.
naasking 5y ago
The problem is not just with "belief", but in the process itself. Non-replicable studies are actually cited more than replicable studies: https://advances.sciencemag.org/content/7/21/eabd1705 https://advances.sciencemag.org/content/7/21/eabd1705 Science journalists probably are vulnerable to the same thing influences that lead scientists to do this, except they have even less review on their claims and so they become pop culture sound bites. As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising, and per the above link, reviewers apply lower standards on these papers in the hopes of finding something truly interesting and/or exciting. Not just in the results mind you, sometimes papers also apply a novel methodology that might be worth wider discussion. I'm not sure that's worth the reduction in credibility though.
- theptip 5y ago
  > As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising This seems the likely explanation; I saw a paper recently that showed that lay people can predict what will replicate with above-chance accuracy[1]. I imagine scientists are even better than lay people at this. So non-replicable results are almost by definition surprising (i.e. they are hypotheses that don't match our current model of how the world works), and surprising results are definitely better news than unsurprising results. [1](https://journals.sagepub.com/doi/full/10.1177/2515245920919667 https://journals.sagepub.com/doi/full/10.1177/25152459209196...)
- edanm 5y ago
  > As for why non-replicable results are cited more, I'd speculate that non-replicable results are often more unintuitive and surprising I mean, a kind of well known thing is that in general, something false can be more interesting than something true, since it has more degrees of freedom. You can make up anything you want - the truth has to conform to what's actually real.
TaupeRanger 5y ago
Same reason we believed eggs will give you heart disease. Science reporting is terrible and the general education system doesn't teach rational skepticism, it teaches unconditional trust of intellectual authority.
- reedjosh 5y ago
  Indeed! And in reality eggs are basically a superfood.
- denimnerd42 5y ago
  every mainstream thought about diet is untrustworthy to me due to the above type of example :/
  - loudtieblahblah 5y ago
    it is. mainstream diets will tell you: * saturated fats are bad (lies told us by a crappy Ancel Keys study, promoted for decades by processed food companies (like Kellogs) ran by Seventh Day Adventists who were convinced "meat led man to dangerous impulses and temptation") * polyunsaturated fats are good. The American Heart Assocation had an article up for years that went as far as to claim Omega6s are heart healthy. They only recently took it down this year. But we know they're inflammatory and we know we're consuming 25-100x more Omega6s than we ever would before the industrial invention of seed-oils being shoved into every product imaginable (bread, cereal, granola, anything that comes in a box, feed given to animals meant for meat production) here's a webmd article on it: https://www.webmd.com/heart/news/20090126/expert-panel-omega-6-wont-hurt-heart https://www.webmd.com/heart/news/20090126/expert-panel-omega... \ * sunlight creates high cancer risk (ignoring that cancer is unlikely, treatment if caught early has a high survival rate, and the risk of not having vitamin d throughout your life risks far more likely autoimmune issues, depression, anxiety and even certain cancers and inflammatory disease). * sugar is good for you. Sure, they'll specify processed sugars are bad for you or "added sugar", but common wisdom will accept a NET (subtract fiber) 200-300g carb diet as acceptable. Grain is still often listed as the most important and largest part of the food pyramid. The reality is - all mainstream health advice, including that which you'll get from your doctor who got a whole single nutrition class in school, ensures that processed foods don't loose business on the front end and the medical/pharma industries don't lose money on the back end. even in the push for a more vegetarian/blue-zone diet world - they're doing so by promoting meat alternatives like "Beyond Meat" which is chock full of so much seed oil and other processed substances, it's mainstreaming vegetarianism-as-fast-food. McDonald's burger.. is still a McDonald's burger and you shouldn't be eating it. If a factory isn't making it at scale, shoving it in a box, branding it and ensuring you don't have to spend any time making/cooking/preparing whole, fresh foods (those pesky things that tend to have short shelf lives and are costly to Ag businesses), then your PCP, the government, most food businesses, your medical insurance company, absolutely no one of any kind of "authority" isn't going to promote it highly. They'll do ANYTHING except remove seed oils. They'll make your potato chips out of broccoli and carrots and still drench them in sunflower or canola oil. They'll reduce the salt. they'll make shit out of beets. And still manage to make it horrible for you. The MSM regurgitates "health" info regarding diets in a way that acts as advertising for these orgs.
batguano 5y ago
I assume you are referring to "The Stanford Prison Anecdote"?
hpoe 5y ago
They are believed because they are sensational, they are titillating, and it gives people the excuse for their bad behavior by believing that everyone would be a monster in the right circumstances.
Blikkentrekker 5y ago
Who is “we”? This has been criticized since 1999 and far longer back. How long ago is the Rosenhan Experiment again? Dare I say that a majority have always held a dismissive, critical view of such matters. But of course, those that hold dismissive views of it are not the ones who work in such fields, and certainly not at the top ready to implement changes, so it can continue to persist and go on despite of being highly criticized. At least when I studied physics at a university around what must have been 2005, most of the students and professors there were highly critical of softer science and it often came up that some of these papers popped up and were viciously criticized for clear and obvious systematic errors in the methodology.
loudtieblahblah 5y ago
same reason we believe observational studies in nutrition, or worse - studies on animals. if you have the weight of peer review or at least a well documented study, then the media runs wild with it's claims, it gets shoved into textbooks, then governments shape policy on those claims, corporations and medical practices sell gimmicks, books, supplements, therapy and plans of action to heal you... it all becomes lies, half-truths, bad data all just repeating itself ad naseum until "truth" is established in the public consciousness. Quacks on the web, the American Heart Association, your local doctor's office will all pedal garbage based on the bad data. And once it's well established as true, backing away from it is hard because it's become so woven, institutionally. This is why people think saturated fat and sunlight are bad or at least a net-negative. Even modern medicine, psychology and nutrition sciences all have horrible replication crisis's and we're no better in rejecting the nonsense now than we were then.
aaron695 5y ago
SPE was needed at the time because everyone knew the Germans were like us, but needed a framework to say it. It also opened up the idea the Japanese were like us too. It allowed us to say what we believed and build on that. It's not science. But science isn't the only way forward.
- acchow 5y ago
  Can't we just look at history and conclude that we are "just like the Germans"? Why do we need to shroud it in pseudo-science?

brnt 5y ago

Psychological science is now thoroughly privatized. Don't read papers if this subject is of interest to your, look al big tech, ad tech, troll factories, influencer science, media consolidation, for-pay research-charities (great rabbithole to dive into btw). Academics are thoroughly out of the loop on this one, as are we all incidentally.

CactusOnFire 5y ago
I became a Data Scientist because in 3rd year, I found the OKCupid data blog and realized their research had methodological soundness these "20 undergraduate lab students forced to be there" research never would. You trade 'shot-in-the-dark lab experiments' for 'a clear and obvious agenda'. The trick is to just make sure the agenda isn't morally reprehensible.
- mistrial9 5y ago
  more than one third of the entire world population believes that OKCupid is "morally reprehensible" (!)
  - stickfigure 5y ago
    This is an interesting statement. Do you have links to something more I can read? Who thinks dating sites are morally reprehensible?
  - burnte 5y ago
    Religious biases don't count as valid moral objections. Religion has a strong negative effect on critical thinking skills (1, 2). If you dogs/pigs/shellfish/mixed-fabrics are inherently unclean or that women are property, your objections to a dating website are probably not going to have seriousl scientific ramifications. 1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5742220/ 2. https://www.google.com/search?q=Religion+has+a+strong+negative+effect+on+critical+thinking+skills https://www.google.com/search?q=Religion+has+a+strong+negati...

23B1 5y ago

"Science"

jcims 5y ago

I have no basis to argue against the article, but I would think there is more incentive to game it now so I wouldn't take it as a given that things have improved.

ouid 5y ago
Bonferroni effects have, at least in the popular consciousnes, become much more front and center. I wonder how many people on the stre can define p-hacking. Most educated people know about the reproducibility crisis, and I would wager that around half of them understand the cause. That seems like it might create a better environment.
- SV_BubbleTime 5y ago
  > around half of them understand the cause I’m in the half that doesn’t know, apparently. I considered that with hard sciences, the cost, time, equipment, conditions, etc were limiting factors. I think this paper on subatomic particles changing into antimatter is interesting, it was observed in a unique facility, once, under some condition that can never repeat, etc. Just kinda have to take your word for it. As to soft sciences… I really don’t know there. Give me a hint?
  - ouid 5y ago
    Bonferroni effects. In the parent article referred to as researcher degrees of freedom, in pop culture referred to as p-hacking. In machine learning, it's referred to as overfitting. They are all the same phenomenon, of having more than one hypothesis, and then using statistical measures that assume that you only have one hypothesis to claim confidence. Journals only publishing positive results is an example of this phenomenon. Changing your hypothesis to match your data is another such example.

tgv 5y ago

And that's without knowing how the study was really executed. Where I worked, there was a relatively successful postdoc who presented the results of his p<.05 significant pilot study. When asked, he said it wasn't the first pilot. It was the 20th.

thechao 5y ago
Which why P-scores need to be proportional to some nontrivial inverse factor of the number of experiments done in the field. Are there 10000 researchers doing 20000 experiments per year? Take 20000, multiply by the number of years we expect an academic to do hands on work (30?); invert: you need a P-score better than 1:600000.
- pytonslange 5y ago
  This! Bonferroni correction across all conducted studies/analyses. I will suggest this next time I am reviewing a paper with crazy claims and shitty statistics.
  - yxwvut 5y ago
    I know this is mostly a joke, but what you really want is Benjamini–Hochberg correction, unless you want to prevent even a single false discovery in all of science. FDR vs FWER
prof-dr-ir 5y ago
I have a similar horror story. A medical student worked hard to analyze, say, 40 x-rays out of hundreds available. He found no significant evidence for some hypothesis. When he told his supervisor, the reply was: "Well then you should just analyze some more x-rays. I'm sure you'll have a statistically significant result at some point."
chromaton 5y ago
Somehow one of the 3 experimental groups has 16 people, whereas the other groups have 14 people. Wonder why...

m12k 5y ago

When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science?

throwawaysea 5y ago
There was a past discussion about peer review on HN: https://news.ycombinator.com/item?id=20607259 https://news.ycombinator.com/item?id=20607259 Another related discussion was about the grievance studies scandal, which also touches on peer review and academic rigor in journals: https://news.ycombinator.com/item?id=18127811 https://news.ycombinator.com/item?id=18127811
javert 5y ago
The incentive system that academic scientists live under explains why they don't push for studying metascience. If that's going to happen, it has to come from outside the government-science complex. Good luck with that...
ryanmarsh 5y ago
Similarly, I heard someone the other day assert that no one had done an double blind placebo controlled trial of the effects of FDA regulation.
- hannob 5y ago
  That is probably true, but do you have any proposal how such a trial should be done? RCTs are good when they can be done and I'm all for doing more of them and too often there's no good excuse for not doing them. But at some level things just get impractical.
logicchains 5y ago
>When you think about it, it's insane that metascience isn't studied more. We're going to question every little detail about the universe, but we're just going to take on faith that peer-review and publication in journals is an effective method for weeding out "bad" science? In a sense we do have this: engineering and finance. Engineering turns good hard science into new tools, machines and weapons, and Finance turns good (predictive) soft science into new ways to make money.

mykowebhn 5y ago

I worked at a well-known clinical psychology lab at an Ivy League school many years ago. There was so much "gaming" of the results. For example, our PI didn't want to include a subject in our study because his scores weren't elevated enough, and our PI was worried that his score wouldn't drop enough which would adversely impact our results. Another example: our active treatment therapists knew exactly what they were treating for, and our study was measuring improvements in the condition that was being treated. However, the control therapists had no idea what they were treating for, and we purposefully kept this information from them!

kizer 5y ago
Hence the “replication crisis”. Replication of experiment and differences in outcome highlight the “core” result. When the results are just totally different, obviously there was “gaming” or just a bad experiment.
mattkrause 5y ago
The first example doesn't seem that bad. No intervention cannot reduce something if it isn't there to begin with. The screening should be disclosed in the methods (and, ideally, pre-specified), but you do need to account for floor/ceiling effects somehow.

antiterra 5y ago

The part that confused me the most, as someone who has never done research science, was how the questions were casually categorized as gender, ethnicity or no-identity salient. If someone is from Brazil and is of second or third generation Japanese descent, how much of the questions are 'salient' to Brazilian identity vs Japanese? There's an unspoken implication that part of the 'good at math' stereotype relies to some degree on speaking a non-English language at home, which I don't think is a safe assumption at all. > In the Asian identity-salient condition, participants (n = 16) were asked (a) whether their parents or grandparents spoke any languages other than English, (b) what languages they knew, (c) what languages they spoke at home, (d) what opportunities they had to speak other languages on campus, (c) what percentage of these opportunities were found in their residence halls, and (f) how many generations of their family had lived in America.

dash2 5y ago
But this would bias results towards zero, no?

publicola1990 5y ago

Much of such dubious "science" is packaged by self help authours as miracle principles which are going to set ones life right. Popular books like "Power Of Habit" and Cal Newport's books also seem to fall into this category, and seem to enjoy huge patronage, even in HN.

programmarchy 5y ago
I'd bet that even more dubious science is packaged by large corporate interests. Take Big Tobacco, who had doctors saying smoking is healthy, or the sugar industry, corrupting guidelines and policy on coronary heart disease [1]. And medicine is not immune either, with financial conflicts of interest and pharmaceutical sponsorships correlating with outcomes. [2] [1] https://www.npr.org/sections/thetwo-way/2016/09/13/493739074/50-years-ago-sugar-industry-quietly-paid-scientists-to-point-blame-at-fat https://www.npr.org/sections/thetwo-way/2016/09/13/493739074... [2] https://en.wikipedia.org/wiki/Metascience#Medicine https://en.wikipedia.org/wiki/Metascience#Medicine
- reedjosh 5y ago
  Careful now... You might make someone skeptical of vaccines. /s
- derbOac 5y ago
  I don't disagree with you, but might provocatively suggest that federal funding in the US vis-a-vis a large university could also be considered a large corporate interest at this point.
Sr_developer 5y ago
There is a cottage industry of what you may call,if you feel uncharitable, bullshit peddlers: Simon Sinek Cal Newport Charles Duhigg Mark Manson Ryan Hollyday Malcolm Gladwell The list is endless to be honest, they are each different on their own way but they have the following common points: - They are on this for the money, so expect them to be always pushing their books, products, next book, next tour, next program.Hustling, hustling,hustling. - Their grandiose pronouncements with little or not serious backing. - Their unwarranted sense of speaking from a position of authority - The over-simplication and stupid generalization of what it is messy, complex and very much unique.

kizer 5y ago

Interesting how as I began reading the excerpts from the paper I had to stop myself and realize that the author himself had ironically conferred a bias against the paper, LOL. I was primed to find all the flaws. I think psychology and sociology are legitimate and worthy studies, but they run into issues with the scientific method itself due to the ambiguous and “high-level” nature of their concepts and theories. It’s hard to create meaningful, repeatable experiments. So perhaps it should be emphasized how important it is to put effort into constructing experiment... and in particular keeping the subjects unaware of what it being tested. There are probably many great examples of experiments done.

failwhaleshark 5y ago

If it is not a hard science, then it is likely a variant of philosophy.

ouid 5y ago
I think you could effectively argue that mathematics is a branch of philosophy.
- failwhaleshark 5y ago
  You can construct proofs for mathematics, but not for the existence or refutation of a divine creator.
  - Afton 5y ago
    I think you have a narrow view of philosophy. Very little modern philosophy has anything to do with that kind of question.
    - failwhaleshark 5y ago
      Nope, you're projecting and using a strawman. That's not the point. :)
  - bfors 5y ago
    True, but with math you also eventually get down to axioms that can't be proven.
    - rajin444 5y ago
      I don't think we know if we can know everything or not. That seems like _the_ question to answer.
      - kirubakaran 5y ago
        https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...

andi999 5y ago

Actually I do not think that paper is that bad compared to majority (well I do not think Psychology currently is a science)

deleted 5y ago

[deleted]

29athrowaway 5y ago

At least they have not independently rediscovered integration yet. https://care.diabetesjournals.org/content/17/2/152.abstract https://care.diabetesjournals.org/content/17/2/152.abstract https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf

anthk 5y ago
I don't get the downvotes.

stevetodd 5y ago

Reading the other comments about the lack of integrity in scientific research that they’ve witnessed, is it any wonder why people might be skeptical about important issues like climate change, vaccines, masks, etc? There’s great research being done and a larger part of the population is becoming more and more skeptical because lies are being published and touted. Science needs to clean its house.

hypersoar 5y ago
The skepticism about climate change is due much less to issues with the scientific community that than it is to decades of propoganda funded by people who fear losing money to efforts to fight it.
- SV_BubbleTime 5y ago
  Are you suggesting there is only money, power, and propaganda by deniers of climate change? Don’t you think it’s possible there are people exaggerating it for money, and power? I don’t think ”they use propaganda and everyone who disagrees is just wrong” is a good argument when the topic is “we know in other areas there are issues with the scientific community so why not this one”. We can still have the right answer and have gotten there the wrong way.
fullshark 5y ago
Independent replication needs to be incentivized, either as part of tenure process or have grad students do it as part of getting a PhD or something. Then the results are published in a journal of replication studies.

theknocker 5y ago

In 1999, they at least thought they needed a model (as opposed to policy conclusions), i.e. they were at least actually attempting science. I don't know what this article is supposed to be or why I should care about some random guy's random polemic about some random paper.

Blikkentrekker 5y ago

I went through it in a clinical sense around that time and this was at the time that I had no legal right to demand to read reports about me for correction, which came later, and when I did it, the tunnel vision in it explained many things about the conversation. What I remember most is that the clinical psychologist kept fishing as to why I covered my face with my hair and I kept saying that there is no reason other than gravity and that I cannot control that my hair obscures parts of my face and the report contained that I did it on purpose to hide my face, which I'm fairly certain I did not, but it seemed that this was really what the clinical psychologist settled on early and continued to search evidence in support of.

vlovich123 5y ago

This is a flawed logical argument. Imagine this article from 1830: "If you think phrenology is bad, imagine how bad it was in 1810". Rinse & repeat for the appropriate time frames for alchemy, astrology, or any other field that tried to misapply science. Just because a field is studied for a long time or tries to apply the scientific method doesn't lend credence to the approach. All it means is, at best, we've managed to toss some things that are now obviously wrong/flaws. In 20 years we'll be doing the same to things we "know" today OR the flaws will remain because we don't have the math/science to demonstrate the flaws more obviously & there's social pressure to keep "building" (even if the foundation is flawed). However, as we should all be aware, false knowledge grows exponentially more quickly than our true understanding of the universe because our imagination is limitless.

vlovich123 5y ago
To anyone downvoting me, consider this HN article from not too long ago: https://news.ycombinator.com/item?id=27489927 https://news.ycombinator.com/item?id=27489927 The general premise with these studies is that if an effect size is real, then a preliminary study would show something interesting. To my knowledge, statistically that is a nonsense argument. Small sample sizes suffer from various small sample effects to the point that you can't predict either way (otherwise there wouldn't be a point in doing a larger study). To add insult to injury, all of these kinds of studies are only on local college students, which further invalidates any potential information gleamed from a preliminary study. TLDR: The way science is done in the social sciences is fundamentally flawed & the fact that limited funding ensures that's the case doesn't excuse that a significant enough part of the body of knowledge isn't reliable.

loloquwowndueo 5y ago

Gee you talk about 1999 like it was more than 20 years ago. Oh wait … (damn I’m old)

dfxm12 5y ago

Do we have to imagine? I'm sure many scientists in the field today were working in 1999. There's certainly enough data from the time to look at this in some level of detail, but Cherry picking an article and pointing out its flaws doesn't really prove much about the state of the field back then. Maybe if the author mentioned that this article was highly regarded back then, there would be a point, but for all we know, the article was thought poorly of at the time and contemporary scientists just thought it slipped through the cracks. It also doesn't talk about new controls in place today that would prevent a similarly poor article from being published, or even a system of "retracting" poor articles. I don't really trust that everything being published today is without flaw. After all, the other examples of bad science given are fairly recent.

gregoreous 5y ago

Those dummies. They had perfectly good time machines in 1999 and they didn't even use them to visit our enlightened era. I bet they did all their calculations on some old Windows 95 system instead of investing in solid multiple core machines. Still, they don't hold a candle to Gregor Mendel failing to incorporate DNA in his genetic work, if you can believe it.

splithalf 5y ago

“Don’t hate the player, hate the game.” Indeed. It is a scale problem. We have too many producers of research, too few destroyers of research, like Gelman. Show me the incentive and I can tell you the outcome. Encourage the whole world to become “experts” and then be amazed as the reverence and trust in expertise is devalued. That’s us.

torginus 5y ago

What I don't understand is psychology research (at least in academia) does not seem to have moved behind the "we locked three dozen college kids in a classroom and had them perform some bizarre acts, through which we hope to pierce the veil of human nature" - style research. I feel like if something good could have come out of the social media age - is that we have documented the natural behaviour of a vast number of people over long stretches of time. I think this is the sort of invaluable data, that has the potential to advance the quantitative understanding of human nature.

B-Con 5y ago
I've always been perplexed by this, too. Aren't the two most important aspects of the research the data set and the study methodology? Why on earth would you skimp so heavily one of them? I don't work in the sciences, but this kind of nonsense doesn't exist in the "actual" sciences. Physicists spend loads of money producing just the right experiment conditions and documenting the manner the experiment was created in. The dataset is incredibly important and very rigorously examined. But in psych, the dataset is basically an after thought. "Oh by the way, we chose a small handful of kids who happened to be free at that time, with no reason to believe there's any geo, social, educational, political, or ethnic background diversity, it probably cost us like $200 plus some pizza. Now let's print the results in $5 million worth of textbooks for a few decades!" I don't buy the funding argument. A professor probably costs the university 100-150k/yr and will be working on a small handful (2-6, ish?) of projects. Buying an hour of a subject's time for a study must cost, what, $30/hr? Shouldn't they be allocating a minimum of $50k in funding for the actual research, and dropping at least $10k for a good dataset? I don't buy the argument that most experiments don't yield good results so the university is wary of funding them. At a minimum they should follow up a cheap test with promising results with a real experiment that has actual funding before everyone gets all excited about it.

deleted 5y ago

[deleted]

jrochkind1 5y ago

The realities of science as actually practiced rather complicates the religion of "trust the science!" (which usually actually means "trust the scientists!")

pjmorris 5y ago
“In God we trust. All others must bring data.” - W. Edwards Deming.
- fullshark 5y ago
  I don't even trust data anymore
  - pjmorris 5y ago
    Agreed. It's not like the good old days when data was about one physical process that didn't have too many downstream impacts.
colechristensen 5y ago
The science-as-religion people think that science delivers truth in the same way any other theology does. I took a philosophy class filled with people in science degree programs and a few of my classmates were often vocally upset about how nothing was certain in philosophy and everything had multiple sides to it. That was very eye opening, many of these people were soon to be graduated and through their entire educational career they had only been exposed to Truth to the extent that being shown debate and disagreement on a topic made them upset. You're not supposed to "trust the science" you're supposed to trust the process to approach the truth. If you can't read multiple arguments on the same topic and analyze them, you really don't get it at all (and waaay too many people with degrees can't do this).
- SV_BubbleTime 5y ago
  > You're not supposed to "trust the science" you're supposed to trust the process to approach the truth. If you can't read multiple arguments on the same topic and analyze them, you really don't get it at all (and waaay too many people with degrees can't do this). Agree entirely. I’ll add that I go slightly further. If you can’t take your pet topic, and can’t make even a slightly good faith argument against yourself, you have no business with strong feelings on it. I have a wedge issue topic I am an expert on. I could argue against myself, both effectively, and in an actual compromise that no one wants. Yet… people who argue against “my side” are constantly using complete bullshit science from the 1980/1990s when governments literally weaponized depts and ivy leagues to push for “evidence” to support their desired policy changes. These people now tell me to “trust the science”, “I’m sure this researcher at Harvard is wrong and you’re right”, and ”this article from CNN / FOX / VOX / WAPO proves you are wrong and I refuse to consider they have an agenda”. Worst part that there is no shame of willful ignorance, they “trust” the people they claim can’t be wrong, simple, done. Why should they bother to acknowledge another side - if they do it means everything else needs reevaluation too.

PatentlyDC123 5y ago

Is there a method of flagging papers in scientific journals that have been criticized or refuted, e.g., by later studies or proven inability to replicate the data? In legal research services, like Lexis Nexis or Westlaw, many cases are "flagged" when a later case or statute reverses, narrows, or otherwise affects the earlier case. This system warns lawyers that they may not be able to cite the flagged case in their current work. Of course legal research services also come with their own issues and costs; some of which are likely associated with this system.

walrus 5y ago
A journal can publish a retraction. The website Retraction Watch[1] aggregates these retractions and provides a database that you can query. Reference management software like Zotero[2] can use this to monitor your collection of papers and notify you when one is retracted. [1] https://retractionwatch.com/ https://retractionwatch.com/ [2] https://www.zotero.org/blog/retracted-item-notifications/ https://www.zotero.org/blog/retracted-item-notifications/
- earthscienceman 5y ago
  Yeah. They could. But few (zero?) studies are retracted for the sake of being proven incorrect later. And, to be far, it would be ridiculous. Imagine having your career nullified because when you're 60 some major break through shows that your studies aren't relevant anymore. Your work was good when you did it, but now there's something new. It's kind of the definition of scientific progress. However, as a counter example, in my very narrow specialty there is a well known lab that has produced highly cited bogus studies. I've personally published opposing results and said, "these studies are wrong for these reasons" using almost exactly those words. Should they be retracted? Absolutely. Will they ever be? No. Because, of course, the publisher and the authors just point the finger back at me and say "no, you're wrong!" and that's more than enough to keep the vague debate going.
hypersoar 5y ago
An issue that comes to mind is, ironically, authority. Who decides when a paper has been discredited? I can see all sorts of incentive problems with such a system. On the other hand, Westlaw is only cataloging what has already been decided. If the Supreme Court overturns a prior case, then it's overturned whether you agree with the reasoning or not.

jedberg 5y ago

I studied CogSci in the late 90s, which involved taking psychology courses. Every psych course that ended in zero required you to participate in psychology experiments. The profs like to say that 18-22, mostly white/asian, mostly rich kids are the most studied group of people in the USA. Or course we now know that a ton of psychology research doesn't actually apply to people outside that small narrow window of people.

thaumasiotes 5y ago
It doesn't apply within that window either.
philwelch 5y ago
The social sciences have a term—“WEIRD” (Western, Educated, Industrialized, Rich and Democratic) to describe the population that most psychological and social science research has studied the most deeply. As the acronym implies, these populations are not normal across human history or even the modern world. One of the fascinating concepts in abnormal psychology is the notion of a “culture-bound syndrome”—a mental illness that only occurs in a specific cultural milieu. I wonder how many mental illnesses are actually culture-bound syndromes of WEIRD culture?

PicassoCTs 5y ago

My 5 cents. There is not one but two psychological sciences at the moment. One is public, publicly funded and in very bad shape, with most of its results being not reproduceable (reproduction crisis), and a partial destruction happening via neuro-sciences. Who destroy, but do not offer large-scale replacement theories, that could encompass the whole species and are not in contradiction to other neuro-science results. And then there is the second faction (Disclaimer: I can not proof what is deduced after this disclaimer.) There are several cooperations and at least one government, which had the chance to large scale collect data on the population. This data is a psychological gold-mine, if explored properly. One could query such a behavioral database and more important - enact virtual experiments. Out of all male humans, who curse in front of the tv in the evening, filter out those who get into a car accident, plot the increase in cursing in front of the tv. If taken to the extreme, this new, data-mining behavior sciences, could create a agent based model of the species in all variations and collect data only to check the expected outcome of a societal change against the real outcome, with spot samples. I have my own little pet theories, how humanity would look to this privatized psychology, but i digress. I think, academic psychology should have full access to all cooperation databases that contain behavioral data.

SV_BubbleTime 5y ago
> I think, academic psychology should have full access to all cooperation databases that contain behavioral data. No thanks. We already have quack science from the 1950s still driving policy discussions on wedge issues today. I don’t want more convincing shit, I want less shit.

fungiblecog 5y ago

This is what happens when everyone is told they're awesome and nobody is allowed to fail

anthk 5y ago

1999 to me was almost yesterday.

PostThisTooFast 5y ago

What a BS headline

wolverine876 5y ago

> If you think psychological science is bad I know it's often repeated, but is there evidence that "psychological science" is worse (by some measure and to a significant degree) than other social sciences? As long as we're talking about science, let's look at evidence! ;) Anyone studying human behavior, on any scale, has the added challenge that it's so complex that you can't really isolate simple mechanics, as you can in physics or chemistry. People are far more complex than a molecule, or even a billion molecules. BTW, I'm aware of the well-known cite, the NY Times article on the 'reproducability crisis' from a few years ago. Few seem to have read the article: The results were reproduced, but many were at lower strength than the original research. That's important, but it's not like the researchers just complete missed and results were arbitrary.