19 ms·

What happens when the public lose confidence in academic findings?

124 points by ContrarianBrit 5y ago

ContrarianBrit 5y ago
'Following several crises, quantitative social scientists now worry about the validity of methods. Such developments have political implications. There may be greatest risk to progressive ideologies, these being predicated on the notion that politicians can improve society. If scholars worry about findings, how can we be confident that policy interventions are effective?'
- UnpossibleJim 5y ago
  I mean, the validity of the social sciences methods were, in quite a few cases, questionable. The reproducibility of such studies led to something of a crisis of their own, did they not? https://www.wired.com/story/social-science-reproducibility/ https://www.wired.com/story/social-science-reproducibility/ https://www.psychologytoday.com/us/basics/replication-crisis https://www.psychologytoday.com/us/basics/replication-crisis
h2odragon 5y ago
Charismatic leaders with a Narrative will become the organizing poles of society; the notion of a greater shared Truth will fade further. People with the ability to think for themselves and the desire to expend their efforts towards real goals will be elevated beyond contact with reality, or burned as heathens trying to undermine the rest of the tribe.
bombcar 5y ago
The dirty secret is people don't base their decisions on academic findings. They make their decision and then search out findings that support it to use as arguments for it.
- scottious 5y ago
  For those who don't know, this is called motivated reasoning or confirmation bias. Once you can give a name to it, it's easier to notice it all around you. I'd argue that it's one of the biases that plagues humanity more than any other, our brains seem to be hard-wired to engage in motivated reasoning
- ContrarianBrit 5y ago
  Moral foundation theory is intriguing and crucially important.
- marcus_holmes 5y ago
  I passed my Masters on this basis. I worked out what I thought about the topic, then searched the journals for the required number of papers whose abstracts agreed with me. I didn't learn a damn thing from the process.
  - jbjohns 5y ago
    Actually it sounds like you did.
    - droopyEyelids 5y ago
      Object Lesson, n. "a striking practical example of some principle or ideal."
- coffeecat 5y ago
  I'm sure that applies in some cases, but it doesn't seem universally true. Personally, I hate exercising but I do it because the epidemiologists say I should. And I love drinking, but I don't because the epidemiologists say I shouldn't. Smoking has declined pretty significantly over the past several decades, no?
corndoge 5y ago
The Third Sack
- dredmorbius 5y ago
  ???
throwawaygh 5y ago
TL;DR: The hackjob happening in the social sciences will be used to launch ad hom attacks on much more rigorous scientific disciplines.
- hirundo 5y ago
  Optimistically, the hackjob happening in the social and other sciences will be used to launch a deep examination of epistemic practices. This does seem to have occurred in social sciences to some extent. There are no sciences so rigorous that they couldn't benefit from it.
  - ContrarianBrit 5y ago
    Agreed. Kuhn is really worth reading - one of the greatest books of the 20th century.
- ContrarianBrit 5y ago
  But the hard sciences have also had problems with replication. It's worth reading the Structure of Scientific Revolutions by Thomas Kuhn; to some extent, science is socially embedded.
- UncleMeat 5y ago
  The social sciences are the only ones actually engaging in replication to detect the problems. The replication crisis in, say, software engineering research is not published but instead is implicitly understood by members of the community. In grad school I was told to generally assume that the tool with the second highest line in the key graph was the best tool. The top line was invariably the author's tool and its results rarely translated to other experiments. But the tool with the second line was usable by somebody other than its original author on a different problem space.
  - throwawaygh 5y ago
    > software engineering research Empirical SE research is part of the social sciences. The subset of SE research that doesn't push hard for collaboration with social science departments is doing just fine.
ldjkfkdsjnv 5y ago
I think that the existence of leetcode interviews, where tech companies ignore your background as much as possible, and want to see the skills upfront, is a condemnation of education. It's essentially saying, we don't believe your degree says anything about your ability to perform the job. The only reason academia hasnt been completely uprooted, is: 1. Some sects of society use academic prestige as a product. i.e. this harvard educated mckinsey consultant will optimize your business! Or, this harvard educated financier will invest your money carefully! 2. There's no way to have some leetcode like test for most jobs. 3. There used to be more signal in academic credentials, and older sects of society still believe those signals exist. Its the same with doctors, older generations treat doctors like infallible royalty, younger are much more skeptical theres a huge caveat that some top people in the world do end up at top academic institutions. its just they keep mixing in these political appointees, agendas, etc
- 62951413 5y ago
  I think that the existence of leetcode interviews, where tech companies ignore your background as much as possible, is a condemnation of my 20+ years of building real-life systems. And a means of filtering out people over 40.
- Swizec 5y ago
  > It's essentially saying, we don't believe your degree says anything about your ability to perform the job. Unfortunately I have worked with people with a master’s in computer science from an american university who couldn’t write a coherent sentence in English. Let alone write code or design a system. I’ve also interviewed people with illustrious careers who just weren’t a fit for the role they applied to. Credentialing is in poor shape in our industry.
  - ldjkfkdsjnv 5y ago
    Yeah coding is funny, because the code has to work. Theres no getting around it, and most resources for understanding how to get things to work are available on the internet.
rhema 5y ago
Why did people lose faith in the institutions? Because the institutions deserve it. Academic findings only have so much scope. It's nuanced and the news reporters, kindergarten teachers, and scientists themselves often overclaim the impact of the findings. All of the short term incentives reward overclaiming over nuance and no one should be surprised that the zeitgeist is where it is.
- christophilus 5y ago
  Yep. It’s an incentive issue (as are most issues, I’m beginning to think). It’s been really hard not to become too hardened in skepticism as I get older.
  - s1artibartfast 5y ago
    Skepticism is good, but not to be confused with dismissiveness! Intentional curation and omitting topics you don't care about from your information diet is healthy.
- scottious 5y ago
  but this is not entirely academia's fault. I'm reminded of the ongoing conversation about alcohol's impact on our health. Over the decades, if you've consumed mainstream media, you've been told everything from "2 glasses of wine per day is good for you" to "no amount of alcohol is good for you" and everything in between. What's difficult is that it's hard to communicate the nuance of what a study actually studied. How many people were involved? What were the methods used? What was the effect size? What were the confounding factors that couldn't be controlled for? None of this can be effectively communicated in a world that reads headlines and probably not much beyond that. And of course, the media does sensationalize because they're incentivized to. You'll click a headline that says "alcohol is good for you, says science" but probably won't click a headline that tries to communicate the nuance.
  - s1artibartfast 5y ago
    I think the end result is the same. Outsiders should be highly skeptical of reporting on topics which take decades of expertise to start to understand. People seem to feel compelled to form an opinion on every topic, and then think that whatever their take is must be informed. It is okay to not have an opinion on a topic! If someone is particularly interested in obtaining an armature understanding of a subject, by all means, they should jump in and form on opinion, but maintaining some humility is essential.
HPsquared 5y ago
Too much low-quality content, caused by "publish or perish" and funding that doesn't care about quality of results.
- UncleMeat 5y ago
  I disagree with this. Academics don't have too much trouble wading through research and synthesizing results. Laypeople so rarely interface with actual research that the quality of an individual paper is not going to contribute to an overall opinion of academics. Virtually no layperson is forming a coherent opinion of academic research based on an in depth understanding of the replication crisis. Further, the criticism of academia as being full of egg-headed elites who are completely disconnected from the public need has been around for decades - far longer than the current systemic funding and tenure problems.
  - marcus_holmes 5y ago
    I think it's more that the "publish or perish" model has emphasised important/newsworthy results, which has then fed a model of science reporting (and I include the journals in this) that have exaggerated results to the point where trust is strained and science reporting is incredible (in the other sense of the word). The public have been bombarded with ridiculous, counter-intuitive "science" for decades, only for a lot of it to be quietly withdrawn or superceded later. It erodes trust. And that's without taking into consideration the shady corporate-science bullshit like the fat vs sugar debate in nutrition science. If a company can buy a paper that says whatever it wants, and get it peer-reviewed and published in a journal, why would we trust science? Should we even?
    - JJMcJ 5y ago
      > newsworthy In medicine, the Journal of the AMA and the New England Journal of Medicine seem to be the main sources of articles that reach the public's eye. The editor of the New England Journal once complained that it was troubling how papers that had only tentative results, more suggestions for further research than conclusions, would get blown out of proportion in the mainstream news media. Some of the burden is on the journalists, not the researchers.
twofornone 5y ago
The academic system requires a minimum degree of good faith conduct. That honor system effectively does not exist in modern western soft sciences, because ideologues abuse the trust to push their woke agendas with (deliberately?) sloppy science, and the peer review process, thoroughly infiltrated, is unwilling to cast a sufficiently critical eye either for fear of reprisal or out of complicitness. The result of this hijacking is a rigid dogma which manifests as a pseudoscientific justification for the reverse racism that is tearing through western society. That's all before the other deep, systemic issues like perverse incentives and the related replication crisis touched upon in the article. This article is a nice overview of the modern moral panic gripping academia and spilling over into everything else. At this point laypeople are right to question academia as it's once noble purpose has been totally coopted by ascientific progressive politiking.
PaulHoule 5y ago
I don't think a lot of people are paying attention to the replication crisis and similar problems in academia. Instead their own ideas about legitimacy are emotional. The US has a focus on "input legitimacy" which is partially "we do it the way they show in Schoolhouse Rock", and partially about the way politician's communication styles make people feel. The Chinese model based on "output legitimacy" (do institutions demonstrate competence in action) has its own problems (what if things go wrong that we can't control) but should be at least partially emulated in the West. (The flip side of "what if things go wrong" is a reflexive habit of making excuses, sometimes even before they're needed.) If Ukraine survives, for instance, winning the war will make the Ukraine be perceived as one of the most legitimate in the world as opposed to an illegitimate cesspool of corruption. It is astonishing how 2014 was a wake up call for Ukraine to build up its military competence and how Ukraine responded to that.
- mike_hearn 5y ago
  I think a lot of ideas about academic legitimacy are neither based on a deep understanding of the replication crisis, nor are they emotional. The fact is, a lot of bad claims coming out of academia don't require any particular expertise to spot. Did you need to be an expert psychologist to realize that Power Posing probably wasn't right? http://www.bbc.co.uk/news/av/world-41521671/100-women-amy-cuddy-on-how-posture-can-boost-your-confidence http://www.bbc.co.uk/news/av/world-41521671/100-women-amy-cu... Very doubtful. You just need common sense. Like, if this actually worked then you'd be seeing it spread very fast everywhere, but in reality it apparently needs a constant flow of TED Talks to get people to take it seriously. Ditto for priming studies, etc. More recently there's been COVID. The problems here aren't exactly the same as the replication crisis, partly because nobody seems to try and replicate anything in the first place. Really IMHO the problems are much worse. Nothing could possibly destroy confidence in academic and public sector expertise faster than being told masks don't work, followed by they work so well they're now mandatory, followed in turn by seeing case graphs where switching mandates on/off very clearly has no effect whatsoever. The constant flow of bad models that were never acknowledged is particularly destructive. Most people can't articulate why the models always seemed to be wrong except through vague generalities like "bad assumptions", but it doesn't matter. Epidemiological modelling is toast in the public mind, along with the lockdowns they led to. The understanding that they're unreliable is fact based, not emotional, even if the details aren't there. At any rate, having done deep dives into too many trash COVID papers in the past two years, I can't say knowing the exact details of why they're wrong makes much difference. The important thing is actually the social problems surrounding them, in particular the lack of any institutional acknowledgement or response that there's an issue. Observe how discussion of the replication crisis started with and still mostly comes from individual researchers, not institutional leaders. We can see there is no psychological acceptance of how badly they've been getting it wrong because, for example, Imperial College London (home of Prof Lockdown himself) just banned parents from attending their children's graduation ceremony due to COVID: https://dailysceptic.org/2022/03/31/imperial-college-london-bans-parents-from-graduation-ceremonies-just-in-case/ https://dailysceptic.org/2022/03/31/imperial-college-london-... This is at a time of record high cases and after the Queen got it at 95. Nobody cared, it was in the news for barely a day. ICL comes across as a delusional institution with this sort of act, and combined with all the stories of activist craziness on campus, it's getting widely noticed by the general public.
tokai 5y ago
The original paper from Ahlskog & Oskarsson is about bias when applying three domains of determinants to measure political preferences. It feels very much like the author Thomas Prosser has an axe to grind when he includes this study of methodology (in a subsub field of sociology) with the larger issue of replication.
unlikelymordant 5y ago
My field is signal processing (and i've published quite a few papers), but my take on what papers are is a little different. Their purpose was originally, and i think still, a way for researchers to communicate problems they have solved to other researchers. The fact that some subset of papers turns out to be wrong, or that the public doesnt understand them is immaterial. The repositories of knowledge about which papers are good, and which are bad are the research labs. Ask a professor at a research lab which papers have results that can't be reproduced, and which are high quality, and they'll know immediately. When there are statements in the article "Such results erode confidence in academic work", the confidence is only eroded in the papers that are bad, which is how it has always worked. Confidence isn't lost in every paper, or the scientific process, but laypeople and administrators see some stat about "30%" of papers have some error, then all of must science is wrong.
- mrcsd 5y ago
  This is a fair point, but for the layperson who cannot distinguish between the 30% of papers that are bad and the remaining 70% it may as well be a denunciation of all science, since they themselves perhaps cannot tell the difference. While papers remain the main standard by which science is communicated maybe there is a place for those in the scientific community that do know to be more vocal about which papers are worth reading.
  - shmel 5y ago
    This is something that people have to accept: the target audience of scientific papers is other researchers, not laypeople. Ideally we need more effort spent on synthesis of prior research. In my field (ML) very few prominent researchers write overview papers summarizing state of the art even over last few years. There is little academic incentive in doing so. Laypeople should read books written for them (like Hawking's books), not bleeding edge research papers.
    - SpicyLemonZest 5y ago
      The problem is deeper than this, though. All of the "scientific method" theory which is meant to ensure reliable results applies only to experiments and observational studies. Syntheses are just prose - there's no particular reason you should have confidence in a synthesis if you don't think the author shares your values or it contradicts something else you strongly believe is true.
eole666 5y ago
The author should have state from the start that the article he bases this upon is about social science and not the whole academic and science world, it's kind of misleading Social science has always been full of incertainty and biased results, it seems rather normal the public don't trust it blindfully.
- derbOac 5y ago
  There's a sort of sentiment that shows up here on HN -- it's not everyone but it's consistent -- that the problems with replicability are limited to the social sciences, and don't apply elsewhere, which is not true. It's not everyone of course, there's pushback, but I'm surprised and concerned that it persists. Just last week or so there was a paper in Nature about how fMRI studies -- a modern backbone of neuroscience work -- have also been mostly unreplicable. This follows a litany of other papers showing the same thing. That Nature paper made transparent allusions to genetics which invested tons of money in studies which were also largely unreplicable, with fairly obvious power issues that were ignored. When the replicability crisis started getting attention, there were interviews with pharmacology corporation scientists saying that the pharm industry tends to internally assume that 2/3 of published pharmacology studies will not replicate when applied to the real world. There were studies showing that oncology had similar rates of problems, and so forth and so on. The social sciences is fuzzy as you say but they also tend to turn the microscope on science itself first. Meta-analysis in its modern form (as opposed to early statistical methodology papers) comes from psychology; this is just more of the same. There probably are differences across fields but I think few fields are immune. Also, I would push back on the idea that the general public's mistrust is limited to the social sciences. I would argue that issues with climate change skepticism are driven in part by a general distrust of academic findings and scientists, and many of the problems during the pandemic with skepticism about vaccination, masking policies, and so forth were driven by skepticism about biomedical research. Possner said "such challenges do not affect academic confidence in the reality of (say) climate change and Coronavirus"; I was puzzled by this precisely because I couldn't tell if he was being ironic, or pointing these out as areas where there is a contrast between academic confidence and public confidence in those findings, as if to say "well look at how much the public distrusts that; now imagine when academic confidence is eroded further?" My biggest problem with the linked piece, in fact, is that it is framed around the question "what happens when the public loses confidence in academic findings" when that problem is already here to a big extent. We saw what happened during the pandemic. That's what happens. Academics to some extent is falling apart, and it's not limited to the social sciences. The problems with misttrust are running deep and I think academic and political communities have their head in the sand, rather than taking their structural problems seriously. It's always the messenger: the irrational public, the muddle-headed social scientists who are exposing the problems, it's never the corruption in academics itself.
323 5y ago
Studies in nutrition are a meme in themselves. Eating eggs gives you cancer. Next week it protects you from cancer. Coffee cause hearth problems. Next week it prevents heart problems. And so on and on for basically everything we eat.
- shantnutiwari 5y ago
  yeah, I noticed the same thing-- Fad driven science. For some time, Kale was faddy, every study was recommending eating it. REcently, I read somewhere it gives you cancer. groan
- fsflover 5y ago
  I don't know which websites you are using. Respected organization have been providing reliable and non-changing advices for a long time already: https://www.hsph.harvard.edu/nutritionsource/what-should-you-eat/ https://www.hsph.harvard.edu/nutritionsource/what-should-you....
  - TameAntelope 5y ago
    This advice is extremely vague (on purpose) and doesn't answer any of the more challenging questions, e.g. "What is the full, comprehensive list of specific consumables should I avoid, and the reasons for doing so?"
    - fsflover 5y ago
      "What to avoid?" is a wrong question. The right question is "What to prefer?". And it's answered: https://www.hsph.harvard.edu/nutritionsource/healthy-eating-plate/ https://www.hsph.harvard.edu/nutritionsource/healthy-eating-....
      - TameAntelope 5y ago
        Answered vaguely and in a way that's incompatible with a lifestyle that doesn't desire daily grocery store trips or eating in places where you don't have complete control over the menu... I'd rather know what to avoid, even if you think that question is "wrong" (I don't agree).
PennRobotics 5y ago
When I last looked, Google Scholar doesn't take any action on paper retractions. It seems there is no reporting system in place. Researchers Brock and Thornley of New Zealand published and then almost immediately retracted a paper linking COVID-19 vaccines and a much-higher-than-previously-found miscarriage rate. Scholar still lists the original paper posted---with no cover page indicating the retraction. (They do have a separate link to the retraction press release if you look for it.) The main result on Scholar links to a half dozen "not-so-academic" websites (ellinikahoaxes.gr, wokeguru.org, resistance-mondiale.com) hosting the same paper. I'm willing to bet even if the original paper could be removed, independent websites would continue to try relisting it. While it's important to keep a record of bad research rather than scrubbing it away from history entirely, it seems feasible to have a reporting system in place for paper retractions, where the title link includes [retracted dd-mm-yyyy] or something. This was my small taste of disappointment (aka loss of confidence) in a relatively popular academic tool.
graycat 5y ago
> What happens when the public lose confidence in academic findings? When some X accumulates a track record of not working, people give up on X. Maybe then X will get improved, and slowly people will give it another chance. The OP seemed to be mostly about the quality of the science in the social sciences. Hmm .... But, in medicine, there are efforts to be careful. E.g., can test a new drug with double blind randomized placebo controlled trials. Maybe that methodology is good enough to be trusted? It appears that I trusted it when I got two shots and a booster from Moderna against Covid. In math, hmm ..., if there is a claim of solid proof, published in a respected peer reviewed journal, and so far no one has found any flaw in the proof, then some trust can be justified. The gravitational detection work of LIGO is astounding, and I'm trusting it. And I have yet to check in detail the math of general relativity -- it's on my long term TODO list! And I trust the claims for the Webb Space Telescope, and the reasoning for its location at a Lagrange 2 point. So, in medicine, math, and physics, I can have some trust! For the social sciences, my view is influenced by my wife and brother, both of whom tried those, got Ph.D. degrees, etc.
- slowmovintarget 5y ago
  Not everyone operates on Bayesian Reasoning. They should, but it requires an awful lot of thought and practice. Updating credences can be painful.
waoush 5y ago
This seems to be heavily focused on studying social phenomena and less so about things like engineering papers or Physics etc. The author himself specializes in social-oriented research. I think it would be better to look at how successful non-social studies are vs humanities. It is debated on whether or not Sociology is even a science at all. For what it is worth, I happen to know that big tech companies follow research at big name CS universities with a bit of interest. At times, even borrowing the research itself in their products. At one point, a professor I worked under was quoted in Android documentation.
zackmorris 5y ago
Mission accomplished. The US has been an a downward slide into privatization and disenfranchisement since at least the early 80s, probably earlier. I was born in the late 70s and have watched the gradual dismantlement of public institutions my entire life. Where this matters is: academics used to be able to conduct pure research. Now all they do is fundraise and hustle for grants. Anyone used to be able to go to college. Now the gatekeepers decide who has enough money or connections to attend. We even used to have news and (gasp) investigative journalism and public broadcasting and periodic reports to the public on the state of publicly funded research or some great public works project. Now we have infotainment and the suppression of any real tech that might materially improve lives by reducing the workload of the average person. They only fixate on phantom tech that distracts and enslaves. Now wonder the public doesn't care. If we really want to get serious about fixing this stuff, it's going to involve at least in some part, a total rejection of vast swaths of the status quo. I'm all for that, I think most Gen Xers are for that, and certainly most people younger than that are for that. But there is just so much concentrated wealth now diluting social movements. And so much momentum behind the previous generation in congress that will do everything it can to hold onto power in the face of so many calamities we face, because at the end of the day, it's not their problem so they're just never gonna get the memo. If a spiritual revolution away from wealth concentration isn't already underway, it's going to start with some version of "your money's no good here." And individuals withdrawing their contribution from institutions which concentrate wealth and power.
- dimitrios1 5y ago
  While privatization and the fundraising rat race for grants certainly started us down this trend, academia has been it's own worse enemy recently, and it's primary output these days seems to be ideology and activism, rather than cold, hard, factual, evidenced based research and science, so much so, that the activism and ideology has worked its way into the science itself. It's a poisoned well at this point, and it's sad, having worked in a research lab at university myself in the computer vision department, there is still tons of cool things going on, but they all get massively overshadowed by the hot button issues and whatever the Social Justice cause du jour is.
bsedlm 5y ago
> What happens when the public lose confidence in academic findings? flat-earthers half-joking aside, I have come to understand that the academic institutions are part of the state (perhaps I should say civilization?). the point being that academia (as the part of state that I'm saying it is) has a primary function of ensuring stability (same as the most well known face of the state i.e. government); this is mind-bending when faced with the idea that academic research is at the forefront of advancing civilization into uncharted waters (...a PhD must provide an original novel contribution...). Seems that at some point artists became showbiz employees and at that same time, only scholars kept 'advancing the state of the art'. Which is a role historically fulfilled by real artists of the sort that ain't doing it for money nor fame.
kkfx 5y ago
There is a fallacy in comprehend the meaning of Scientific method: confidence in nice and needed to cooperate, but in Science no one should (despite almost all regularly do) trust blindly anything. Of course we can't verify the entire involved knowledge any time, and that's why the concept of collaborative peer review was born, not much different than the concept of "chains of trust": interest researchers in something verify some others publications and add up their own confirmations or doubt, "the network" do the rest. The problem there is that such review and network does not really exists. Oh yes there is the ArXiv, HAL, ... papers circulate etc, but the "mainstream" is a network of hi priced journals and hi paid corporate-backed research + equally hi paid PR in the press. That's why people loose confidence: they do not have ontological means to know but they smell actual Science is mostly commerce and just very little Science. IMVHO the sole way to correct that is build the above mentioned networks. Public, publicly well-founded universities that do research and reviews for the society, very-well separated from the private sector, and share this knowledge with the entire world. At that point the SciHub will be a public, legal and well founded platform and PR can do something but not enough to contaminate knowledge for private interest. At that point people would still not be able to verify, witch is not good, but they have something worth their confidence.
chaps 5y ago
At the start of the pandemic, an academic at a well-known university asked me to review a paper of his, kinda sorta to build a relationship and to eventually publish something of my own based on his paper. The claim of the paper was basically that white (non-hispanic) communities were significantly more affected by COVID when compared against other races in similar income brackets. Lots of statistics, analysis against census data, etc to support the argument and such. The source of the data from his paper was from an "open data" portal of COVID deaths, published by the city's coroner's office. Thing is, the data published by the coroner's office didn't include a lot of fields that I'd already received from FOIA. Fields missing like the name of the person who died. Since the paper's conclusions didn't pass a gut check, I dug in and found that just glancing through the names of those marked down as "white", a majority clearly had names with Hispanic origins. I shared with the researcher that his findings were probably really, really wrong and I was responded with something like, "even if 50% are marked wrong, the findings are still accurate". Some name comparison [1] and spot checking later.. and it ended up being about 80% of hispanic-origin names were marked as "white". The researcher eventually conceded that the paper was wrong, but it was definitely of an eye opening experience into how the academic world works. I lost a lot of faith in the academic world after that. [1] https://www.census.gov/data/developers/data-sets/surnames.html https://www.census.gov/data/developers/data-sets/surnames.ht...
- drugstorecowboy 5y ago
  I am definitely not a scientist, but speculating on race because of names you find to be too hispanic to be white feels like spectacularly bad science to me. This is not my field though so perhaps that is normal an accepted.
  - chaps 5y ago
    The point of name-based analysis wasn't for publication or even "science", but to instead show that the analysis done by the researcher was very likely wrong in order to prevent its publication. There is no way in hell I would ever publish anything that relied on name-based analysis as an accurate measure, that would be deeply unethical as far as I'm concerned. I did ask around with these exact concerns in-mind, which led to this paper by the census in their evaluation of this exact problem (the Passel-Word Spanish Surname List was what I compared against): https://www.census.gov/library/working-papers/1993/demo/POP-twps0004.html https://www.census.gov/library/working-papers/1993/demo/POP-... So, I get it, and I had the same concerns as you, but I like to believe I followed proper due diligence in making sure my approach was reasonable. Especially against the risk of the paper being published (consider how the paper could be used in racist narratives).
ALittleLight 5y ago
One of my favorite stories in the Bible is when the priests of Baal are challenged to a religion showdown by Elijah. The idea is each side will pray for an altar to be lit on fire, whichever altar is lit on fire is the altar of the true religion. I won't spoil the ending - but I think the setup makes a lot of sense. If there is a guy who can ignite stuff by simply asking that it be lit then I would really like to know more about that guy's religion. I have the same opinion of academics - show me what you can light on fire. For example, I don't believe in physics because it makes intuitive sense to me, on the contrary, I find it confusing and counter-intuitive (sometimes). I believe in physics because the people who make magic boxes of light and sound that move fast and connect across vast distances tell me they did so via the knowledge they derived from physics. It's not how I would've guessed the universe would work, but who am I to argue with a blazing altar? Judging by this standard I find many academic disciplines are severely wanting - there is a lot of chanting and waving of hands, but very little smoke. If I hear on the news that there's going to be an eclipse, I'll go out and look. If I hear on the news that eggs are bad for you, I'm going to roll my eyes.
- andrewmcwatters 5y ago
  You don’t “believe” in physics, you understand there is some truth to it based on direct evidence. Belief is accepting something to be true on the foundation of faith—not having direct evidence. “Now faith is the assurance of things hoped for, the conviction of things not seen.” - Hebrews 11:1 (ESV)
  - MattPalmer1086 5y ago
    I disagree with this interpretation of the word belief. A belief may be founded in something, or it may not be. Unquestioning belief without evidence requires faith. Belief is not a synonym for faith.
    - andrewmcwatters 5y ago
      It literally is.
rayiner 5y ago
We're about to find out! The scientific community has done itself a huge disservice in recent years in two ways: 1) Refusing to police non-rigorous fields invoking scientific authority. As the article notes, even rigorous fields are full of results that cannot be reproduced. The situation is even worse in non-rigorous fields. In his famous essay on cargo-cult science, Richard Feynman specifically called out "educational and psychological studies" as "examples of what I would like to call Cargo Cult Science." https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm. Yet today, there is a movement to accord any PhD the respect of the title "doctor," as if an EdD was an authority similar to a PhD in physics. People do this to be polite but it has grave consequences. The respect people accord "science" is, at the end of the day, built on results. We trust doctors because they delivered concrete results that medicine men could not. In the long run, people will notice that EdD is often wrong, because it's not a rigorous field, and that will tarnish the credibility of everyone with a PhD, because the lay public quite reasonably doesn't understand these distinctions in rigor. 2) Allowing scientific authority to be invoked for political purposes. There is a difference between scientific results, and the value systems of individual scientists. Science can, for example, create a vaccine. It cannot address the political question of whether anyone should get the vaccine--that question relies on cost/benefit analysis that, in turn, depends on individual value systems. Or in another example, science can tell you the probability of spreading a disease in a grocery store versus a church, but cannot tell you whether grocery stores or churches are more important. Moreover, most people intuitively understand this distinction. There are, of course, some people who will ignore scientific facts that are as plain as day. But there are a lot more people who will chafe at feeling like they are being told "I'm smarter than you, so you have to accept my values." That, in the long run, will undermine their trust in scientific results. And if we, as highly educated people who understand "science" are honest with ourselves, that skepticism may not be unwarranted. We know that biases exist in the system. For example, it is well known that publication bias exists: studies reporting positive results get published, while studies reporting negative results don't get published: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3341407/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3341407/. As science becomes more politicized, it's inevitable that other kinds of publication bias will affect not only the reported results, but even the kinds of questions that are studied.
godelski 5y ago
I think a lot of comments here are vastly oversimplifying the problem (including this one). People have lost confidence in academic findings due to a wide variety of reasons. There's many people to blame here. There's academics, journals, news/reporting, politicians, and more. Let's look at a few (far from exhaustive and still pretty shallow analysis). Academics: Incentives are frequently to publish fast and often. This encourages work with less detail, higher uncertainty/error, and less impactful work. You simply cannot produce as high quality work in 3 months as you can in 1 year (or 3) or research. Often we want PhD students to produce 5 first author papers in top tier journals before graduating. That's 5-6 years to produce just as many top tier papers (realistically 3-4 given your first 2 years are more class based). It is possible to put out a high quality and high impact paper every year, yes. But this is also a very difficult task. Journals: There's no need to explain this one to anyone who has gone through the review process. Everyone I know that has attempted to publish papers has had several papers where they question what the reviewer actually read because it definitely wasn't their paper. Or where reviewers are vastly under-qualified to review the topic at hand. These are common occurrences. (Note: this more often causes false negatives rather than false positives due to incentive alignments) This creates legitimate complaints and can often be misunderstood from people outside academia. Reporting: We all have seen this. Things like "eating chocolate is healthy" and other things that are obviously wrong. We've seen universities oversell their research. We see news channels oversell or vastly misunderstand research (despite having easy access to experts and that authors are frequently happy to discuss their research). The public also doesn't understand things like confidence levels. They don't understand that we have a different confidence level in anthropocentric climate change vs how much healthier a glass of wine a day is. This isn't their fault though. We should hold the reporters (academic or news sources) accountable since their are structured to be the science communicators. Instead they are politicizing and adding opinions rather than explaining. Politics: We all know this one. Everything is politicized. Not just that, but to the point where things are "left" or "right". If the left supports one thing then the right opposes it and vise versa. This is far worse than both sides accepting the issue and coming at it from different angles (which I've seen this happen and both angles also not be in line with the actual research). Covid is a great example: people on the left over-estimate the dangers and people on the right vastly underestimate them (these are not equally biased positions and I'm not claiming they are, just noting that both have errors, not the magnitude of the errors). Climate change is another example, where right denies and left only focuses on pop/hip topics like tree planting or banning straws and not larger issues. (I'll note that in my anecdotal observations it seems that the magnitude of error is frequently smaller on the left). So we have a system where we aren't encouraged to do high quality research (not meaning it doesn't happen, there are a lot of researchers), research (of any kind) is difficult to get properly peer reviewed, it is poorly explained to laymen, and results become tribal. Why does the public lose confidence in academic findings? You tell me.
TameAntelope 5y ago
I'd argue that absolutely nothing "happens" when the public loses confidence in academic findings, and honestly the public should generally butt out of the entire process. It's not written for them, it's not discussed by them in any rigorous way, their input is not valued or noted, and generally they're completely wrong in very basic ways. Should academic research be done in secret or in a way that actively prevents the public from accessing their results? No. Should we disabuse ourselves of the notion that the public matters to this process? Yes. None of it matters to the lives of nearly anyone until it's gone through many, many layers of filtering. We, the public, ought to let that filtering play out, rather than meddle in the process
- sockpuppet_12 5y ago
  Public tax funds academia through the govt grants that funds the research.
Animats 5y ago
The worst problems come from economics. As an academic discipline, economics has too little predictive power and too much influence on policy. We have no idea when the economy is close to a crash. The effect of interest rates and tax rates on the economy is poorly understood, and interventions often don't produce intended effects. This is a big problem.
stuckinhell 5y ago
We are finding out in America. After years and years of academic failings from food pyramid nonsense to repeated social science studies failing to replicate. The public is on the tipping point of outright rejecting the science (or more so the holier than thou priestess-esque science class). At my gym, all the personal trainers boldly reject food recommendations from the government or scientific papers. They have applied the scientific method themselves to find a system and foods that work, and even have their own online communities. They also can replicate their results with their clients. At my women's book club, I've noticed more and more mom's are homeschooling their kids in pods, it went from 0% before covid to nearly 50% now, and I live in a pretty affluent liberal area. My husband's fantasy football group talked about inflation since the start of covid, and now are extremely distrustful of the "media" and "experts". I'm watching the breakdown of establishment trust in my own life in a very liberal circle of friends and family. I can't imagine how much worse it is in red america.
- WalterBright 5y ago
  > I can't imagine how much worse it is in red america. I've lost enormous faith in the media since 2015, when it became obvious that most of the news (other than the weather report) is simply propaganda or clickbait. For example, take the recent airliner crash in China. NBC news ran a video clip of it flying straight down into the ground. That clip turned out to be a fake. The news anchors are completely shameless in the promulgation of their agendas.
- tempnow987 5y ago
  Somewhat similar. I'm technically highly liberal (I maxed out as a dem donor in a number of races). We are told inflation is passing or a temporary blip by the top econ people in govt. Meanwhile, wage pressure, housing price pressures and more are through the roof. We are told that a portfolio of stocks that invests even in part in fossil fuels is doomed (economically) even though it is a scarce commodity from many volatile regions and that evidence is that ESG wildly outperforms. We are told climate change is the biggest threat, but efforts to solve it are largely forbidden (ie, no nuclear) unless they involve massive sacrifice instead of innovation. We are told it is critical to ban a plastic straw in a restaurant that goes in the garbage but that the tons of trash literally blowing into the ocean from folks in RV's along the water who can't be bothered to throw trash into the cans right there is OK and there should be no consequence. We are told our new LED bulbs won't ficker or buzz when dimmed and will last our lifetimes. Masks won't protect us supposedly from an AIRBORNE virus. A higher risk of arrest by younger men for things like murder is a sign of discrimination against young men, vs any chance of a difference in underlying incidence of murder by young men. We are told Hunter Biden has acted with integrity, that the details of corrupt chinese payments on a laptop are "Russian Disinformation". Um - no. Our kids don't get invited to be on boards of companies they have no experience with, even though they didn't have issues with drugs and are good students. The issue for me is that you don't need to have any knowledge of science to see just HUGE internal inconsistencies in a lot of the "preaching" we receive. This results in a massive reduction of confidence in authority which is a big negative to the liberal movement in my view which is supportive of a bigger role of govt in our lives.
YeGoblynQueenne 5y ago
>> For example, the Blair governments championed evidence-based policymaking. Well, the Blair government is remembered for going to war against Iraq on the basis of evidence that was not sufficiently validated and that was later found to be fabricated: More specifically, the report blamed Secret Intelligence Service (better known as MI6) head Sir Richard Dearlove who presented so-called "hot" intelligence about alleged weapons of mass destruction provided by an Iraqi with "phenomenal access" to high levels in the Iraqi government directly to Blair, without first confirming its accuracy.[43] The investigators found that references to this intelligence in government reports were over-certain and did not adequately stress uncertainties and nuance. The informant was later found to have been lying. The Chilcot report states that "personal intervention [by Dearlove] and its urgency gave added weight to a report that had not been properly evaluated and would have coloured the perception of ministers and senior officials".[44] The day after the report was published, Blair conceded that he should have challenged such intelligence reports before relying on them to justify military action in Iraq.[43][45] https://en.wikipedia.org/wiki/Iraq_Inquiry#Case_for_war_was_deficient https://en.wikipedia.org/wiki/Iraq_Inquiry#Case_for_war_was_... So perhaps it's not only the fault of those pesky liberal intellectuals and their rabid postmodernism that the public has lost its confidence to all sorts of institutions. I also blame the Monty Pythons, for example.
theknocker 5y ago

'Following several crises, quantitative social scientists now worry about the validity of methods. Such developments have political implications. There may be greatest risk to progressive ideologies, these being predicated on the notion that politicians can improve society. If scholars worry about findings, how can we be confident that policy interventions are effective?'

UnpossibleJim 5y ago
I mean, the validity of the social sciences methods were, in quite a few cases, questionable. The reproducibility of such studies led to something of a crisis of their own, did they not? https://www.wired.com/story/social-science-reproducibility/ https://www.wired.com/story/social-science-reproducibility/ https://www.psychologytoday.com/us/basics/replication-crisis https://www.psychologytoday.com/us/basics/replication-crisis

h2odragon 5y ago

Charismatic leaders with a Narrative will become the organizing poles of society; the notion of a greater shared Truth will fade further. People with the ability to think for themselves and the desire to expend their efforts towards real goals will be elevated beyond contact with reality, or burned as heathens trying to undermine the rest of the tribe.

bombcar 5y ago

The dirty secret is people don't base their decisions on academic findings. They make their decision and then search out findings that support it to use as arguments for it.

scottious 5y ago
For those who don't know, this is called motivated reasoning or confirmation bias. Once you can give a name to it, it's easier to notice it all around you. I'd argue that it's one of the biases that plagues humanity more than any other, our brains seem to be hard-wired to engage in motivated reasoning
ContrarianBrit 5y ago
Moral foundation theory is intriguing and crucially important.
marcus_holmes 5y ago
I passed my Masters on this basis. I worked out what I thought about the topic, then searched the journals for the required number of papers whose abstracts agreed with me. I didn't learn a damn thing from the process.
- jbjohns 5y ago
  Actually it sounds like you did.
  - droopyEyelids 5y ago
    Object Lesson, n. "a striking practical example of some principle or ideal."
coffeecat 5y ago
I'm sure that applies in some cases, but it doesn't seem universally true. Personally, I hate exercising but I do it because the epidemiologists say I should. And I love drinking, but I don't because the epidemiologists say I shouldn't. Smoking has declined pretty significantly over the past several decades, no?

corndoge 5y ago

The Third Sack

dredmorbius 5y ago
???

throwawaygh 5y ago

TL;DR: The hackjob happening in the social sciences will be used to launch ad hom attacks on much more rigorous scientific disciplines.

hirundo 5y ago
Optimistically, the hackjob happening in the social and other sciences will be used to launch a deep examination of epistemic practices. This does seem to have occurred in social sciences to some extent. There are no sciences so rigorous that they couldn't benefit from it.
- ContrarianBrit 5y ago
  Agreed. Kuhn is really worth reading - one of the greatest books of the 20th century.
ContrarianBrit 5y ago
But the hard sciences have also had problems with replication. It's worth reading the Structure of Scientific Revolutions by Thomas Kuhn; to some extent, science is socially embedded.
UncleMeat 5y ago
The social sciences are the only ones actually engaging in replication to detect the problems. The replication crisis in, say, software engineering research is not published but instead is implicitly understood by members of the community. In grad school I was told to generally assume that the tool with the second highest line in the key graph was the best tool. The top line was invariably the author's tool and its results rarely translated to other experiments. But the tool with the second line was usable by somebody other than its original author on a different problem space.
- throwawaygh 5y ago
  > software engineering research Empirical SE research is part of the social sciences. The subset of SE research that doesn't push hard for collaboration with social science departments is doing just fine.

ldjkfkdsjnv 5y ago

I think that the existence of leetcode interviews, where tech companies ignore your background as much as possible, and want to see the skills upfront, is a condemnation of education. It's essentially saying, we don't believe your degree says anything about your ability to perform the job. The only reason academia hasnt been completely uprooted, is: 1. Some sects of society use academic prestige as a product. i.e. this harvard educated mckinsey consultant will optimize your business! Or, this harvard educated financier will invest your money carefully! 2. There's no way to have some leetcode like test for most jobs. 3. There used to be more signal in academic credentials, and older sects of society still believe those signals exist. Its the same with doctors, older generations treat doctors like infallible royalty, younger are much more skeptical theres a huge caveat that some top people in the world do end up at top academic institutions. its just they keep mixing in these political appointees, agendas, etc

62951413 5y ago
I think that the existence of leetcode interviews, where tech companies ignore your background as much as possible, is a condemnation of my 20+ years of building real-life systems. And a means of filtering out people over 40.
Swizec 5y ago
> It's essentially saying, we don't believe your degree says anything about your ability to perform the job. Unfortunately I have worked with people with a master’s in computer science from an american university who couldn’t write a coherent sentence in English. Let alone write code or design a system. I’ve also interviewed people with illustrious careers who just weren’t a fit for the role they applied to. Credentialing is in poor shape in our industry.
- ldjkfkdsjnv 5y ago
  Yeah coding is funny, because the code has to work. Theres no getting around it, and most resources for understanding how to get things to work are available on the internet.

rhema 5y ago

Why did people lose faith in the institutions? Because the institutions deserve it. Academic findings only have so much scope. It's nuanced and the news reporters, kindergarten teachers, and scientists themselves often overclaim the impact of the findings. All of the short term incentives reward overclaiming over nuance and no one should be surprised that the zeitgeist is where it is.

christophilus 5y ago
Yep. It’s an incentive issue (as are most issues, I’m beginning to think). It’s been really hard not to become too hardened in skepticism as I get older.
- s1artibartfast 5y ago
  Skepticism is good, but not to be confused with dismissiveness! Intentional curation and omitting topics you don't care about from your information diet is healthy.
scottious 5y ago
but this is not entirely academia's fault. I'm reminded of the ongoing conversation about alcohol's impact on our health. Over the decades, if you've consumed mainstream media, you've been told everything from "2 glasses of wine per day is good for you" to "no amount of alcohol is good for you" and everything in between. What's difficult is that it's hard to communicate the nuance of what a study actually studied. How many people were involved? What were the methods used? What was the effect size? What were the confounding factors that couldn't be controlled for? None of this can be effectively communicated in a world that reads headlines and probably not much beyond that. And of course, the media does sensationalize because they're incentivized to. You'll click a headline that says "alcohol is good for you, says science" but probably won't click a headline that tries to communicate the nuance.
- s1artibartfast 5y ago
  I think the end result is the same. Outsiders should be highly skeptical of reporting on topics which take decades of expertise to start to understand. People seem to feel compelled to form an opinion on every topic, and then think that whatever their take is must be informed. It is okay to not have an opinion on a topic! If someone is particularly interested in obtaining an armature understanding of a subject, by all means, they should jump in and form on opinion, but maintaining some humility is essential.

HPsquared 5y ago

Too much low-quality content, caused by "publish or perish" and funding that doesn't care about quality of results.

UncleMeat 5y ago
I disagree with this. Academics don't have too much trouble wading through research and synthesizing results. Laypeople so rarely interface with actual research that the quality of an individual paper is not going to contribute to an overall opinion of academics. Virtually no layperson is forming a coherent opinion of academic research based on an in depth understanding of the replication crisis. Further, the criticism of academia as being full of egg-headed elites who are completely disconnected from the public need has been around for decades - far longer than the current systemic funding and tenure problems.
- marcus_holmes 5y ago
  I think it's more that the "publish or perish" model has emphasised important/newsworthy results, which has then fed a model of science reporting (and I include the journals in this) that have exaggerated results to the point where trust is strained and science reporting is incredible (in the other sense of the word). The public have been bombarded with ridiculous, counter-intuitive "science" for decades, only for a lot of it to be quietly withdrawn or superceded later. It erodes trust. And that's without taking into consideration the shady corporate-science bullshit like the fat vs sugar debate in nutrition science. If a company can buy a paper that says whatever it wants, and get it peer-reviewed and published in a journal, why would we trust science? Should we even?
  - JJMcJ 5y ago
    > newsworthy In medicine, the Journal of the AMA and the New England Journal of Medicine seem to be the main sources of articles that reach the public's eye. The editor of the New England Journal once complained that it was troubling how papers that had only tentative results, more suggestions for further research than conclusions, would get blown out of proportion in the mainstream news media. Some of the burden is on the journalists, not the researchers.

twofornone 5y ago

The academic system requires a minimum degree of good faith conduct. That honor system effectively does not exist in modern western soft sciences, because ideologues abuse the trust to push their woke agendas with (deliberately?) sloppy science, and the peer review process, thoroughly infiltrated, is unwilling to cast a sufficiently critical eye either for fear of reprisal or out of complicitness. The result of this hijacking is a rigid dogma which manifests as a pseudoscientific justification for the reverse racism that is tearing through western society. That's all before the other deep, systemic issues like perverse incentives and the related replication crisis touched upon in the article. This article is a nice overview of the modern moral panic gripping academia and spilling over into everything else. At this point laypeople are right to question academia as it's once noble purpose has been totally coopted by ascientific progressive politiking.

PaulHoule 5y ago

I don't think a lot of people are paying attention to the replication crisis and similar problems in academia. Instead their own ideas about legitimacy are emotional. The US has a focus on "input legitimacy" which is partially "we do it the way they show in Schoolhouse Rock", and partially about the way politician's communication styles make people feel. The Chinese model based on "output legitimacy" (do institutions demonstrate competence in action) has its own problems (what if things go wrong that we can't control) but should be at least partially emulated in the West. (The flip side of "what if things go wrong" is a reflexive habit of making excuses, sometimes even before they're needed.) If Ukraine survives, for instance, winning the war will make the Ukraine be perceived as one of the most legitimate in the world as opposed to an illegitimate cesspool of corruption. It is astonishing how 2014 was a wake up call for Ukraine to build up its military competence and how Ukraine responded to that.

mike_hearn 5y ago
I think a lot of ideas about academic legitimacy are neither based on a deep understanding of the replication crisis, nor are they emotional. The fact is, a lot of bad claims coming out of academia don't require any particular expertise to spot. Did you need to be an expert psychologist to realize that Power Posing probably wasn't right? http://www.bbc.co.uk/news/av/world-41521671/100-women-amy-cuddy-on-how-posture-can-boost-your-confidence http://www.bbc.co.uk/news/av/world-41521671/100-women-amy-cu... Very doubtful. You just need common sense. Like, if this actually worked then you'd be seeing it spread very fast everywhere, but in reality it apparently needs a constant flow of TED Talks to get people to take it seriously. Ditto for priming studies, etc. More recently there's been COVID. The problems here aren't exactly the same as the replication crisis, partly because nobody seems to try and replicate anything in the first place. Really IMHO the problems are much worse. Nothing could possibly destroy confidence in academic and public sector expertise faster than being told masks don't work, followed by they work so well they're now mandatory, followed in turn by seeing case graphs where switching mandates on/off very clearly has no effect whatsoever. The constant flow of bad models that were never acknowledged is particularly destructive. Most people can't articulate why the models always seemed to be wrong except through vague generalities like "bad assumptions", but it doesn't matter. Epidemiological modelling is toast in the public mind, along with the lockdowns they led to. The understanding that they're unreliable is fact based, not emotional, even if the details aren't there. At any rate, having done deep dives into too many trash COVID papers in the past two years, I can't say knowing the exact details of why they're wrong makes much difference. The important thing is actually the social problems surrounding them, in particular the lack of any institutional acknowledgement or response that there's an issue. Observe how discussion of the replication crisis started with and still mostly comes from individual researchers, not institutional leaders. We can see there is no psychological acceptance of how badly they've been getting it wrong because, for example, Imperial College London (home of Prof Lockdown himself) just banned parents from attending their children's graduation ceremony due to COVID: https://dailysceptic.org/2022/03/31/imperial-college-london-bans-parents-from-graduation-ceremonies-just-in-case/ https://dailysceptic.org/2022/03/31/imperial-college-london-... This is at a time of record high cases and after the Queen got it at 95. Nobody cared, it was in the news for barely a day. ICL comes across as a delusional institution with this sort of act, and combined with all the stories of activist craziness on campus, it's getting widely noticed by the general public.

tokai 5y ago

The original paper from Ahlskog & Oskarsson is about bias when applying three domains of determinants to measure political preferences. It feels very much like the author Thomas Prosser has an axe to grind when he includes this study of methodology (in a subsub field of sociology) with the larger issue of replication.

unlikelymordant 5y ago

My field is signal processing (and i've published quite a few papers), but my take on what papers are is a little different. Their purpose was originally, and i think still, a way for researchers to communicate problems they have solved to other researchers. The fact that some subset of papers turns out to be wrong, or that the public doesnt understand them is immaterial. The repositories of knowledge about which papers are good, and which are bad are the research labs. Ask a professor at a research lab which papers have results that can't be reproduced, and which are high quality, and they'll know immediately. When there are statements in the article "Such results erode confidence in academic work", the confidence is only eroded in the papers that are bad, which is how it has always worked. Confidence isn't lost in every paper, or the scientific process, but laypeople and administrators see some stat about "30%" of papers have some error, then all of must science is wrong.

mrcsd 5y ago
This is a fair point, but for the layperson who cannot distinguish between the 30% of papers that are bad and the remaining 70% it may as well be a denunciation of all science, since they themselves perhaps cannot tell the difference. While papers remain the main standard by which science is communicated maybe there is a place for those in the scientific community that do know to be more vocal about which papers are worth reading.
- shmel 5y ago
  This is something that people have to accept: the target audience of scientific papers is other researchers, not laypeople. Ideally we need more effort spent on synthesis of prior research. In my field (ML) very few prominent researchers write overview papers summarizing state of the art even over last few years. There is little academic incentive in doing so. Laypeople should read books written for them (like Hawking's books), not bleeding edge research papers.
  - SpicyLemonZest 5y ago
    The problem is deeper than this, though. All of the "scientific method" theory which is meant to ensure reliable results applies only to experiments and observational studies. Syntheses are just prose - there's no particular reason you should have confidence in a synthesis if you don't think the author shares your values or it contradicts something else you strongly believe is true.

eole666 5y ago

The author should have state from the start that the article he bases this upon is about social science and not the whole academic and science world, it's kind of misleading Social science has always been full of incertainty and biased results, it seems rather normal the public don't trust it blindfully.

derbOac 5y ago
There's a sort of sentiment that shows up here on HN -- it's not everyone but it's consistent -- that the problems with replicability are limited to the social sciences, and don't apply elsewhere, which is not true. It's not everyone of course, there's pushback, but I'm surprised and concerned that it persists. Just last week or so there was a paper in Nature about how fMRI studies -- a modern backbone of neuroscience work -- have also been mostly unreplicable. This follows a litany of other papers showing the same thing. That Nature paper made transparent allusions to genetics which invested tons of money in studies which were also largely unreplicable, with fairly obvious power issues that were ignored. When the replicability crisis started getting attention, there were interviews with pharmacology corporation scientists saying that the pharm industry tends to internally assume that 2/3 of published pharmacology studies will not replicate when applied to the real world. There were studies showing that oncology had similar rates of problems, and so forth and so on. The social sciences is fuzzy as you say but they also tend to turn the microscope on science itself first. Meta-analysis in its modern form (as opposed to early statistical methodology papers) comes from psychology; this is just more of the same. There probably are differences across fields but I think few fields are immune. Also, I would push back on the idea that the general public's mistrust is limited to the social sciences. I would argue that issues with climate change skepticism are driven in part by a general distrust of academic findings and scientists, and many of the problems during the pandemic with skepticism about vaccination, masking policies, and so forth were driven by skepticism about biomedical research. Possner said "such challenges do not affect academic confidence in the reality of (say) climate change and Coronavirus"; I was puzzled by this precisely because I couldn't tell if he was being ironic, or pointing these out as areas where there is a contrast between academic confidence and public confidence in those findings, as if to say "well look at how much the public distrusts that; now imagine when academic confidence is eroded further?" My biggest problem with the linked piece, in fact, is that it is framed around the question "what happens when the public loses confidence in academic findings" when that problem is already here to a big extent. We saw what happened during the pandemic. That's what happens. Academics to some extent is falling apart, and it's not limited to the social sciences. The problems with misttrust are running deep and I think academic and political communities have their head in the sand, rather than taking their structural problems seriously. It's always the messenger: the irrational public, the muddle-headed social scientists who are exposing the problems, it's never the corruption in academics itself.

323 5y ago

Studies in nutrition are a meme in themselves. Eating eggs gives you cancer. Next week it protects you from cancer. Coffee cause hearth problems. Next week it prevents heart problems. And so on and on for basically everything we eat.

shantnutiwari 5y ago
yeah, I noticed the same thing-- Fad driven science. For some time, Kale was faddy, every study was recommending eating it. REcently, I read somewhere it gives you cancer. groan
fsflover 5y ago
I don't know which websites you are using. Respected organization have been providing reliable and non-changing advices for a long time already: https://www.hsph.harvard.edu/nutritionsource/what-should-you-eat/ https://www.hsph.harvard.edu/nutritionsource/what-should-you....
- TameAntelope 5y ago
  This advice is extremely vague (on purpose) and doesn't answer any of the more challenging questions, e.g. "What is the full, comprehensive list of specific consumables should I avoid, and the reasons for doing so?"
  - fsflover 5y ago
    "What to avoid?" is a wrong question. The right question is "What to prefer?". And it's answered: https://www.hsph.harvard.edu/nutritionsource/healthy-eating-plate/ https://www.hsph.harvard.edu/nutritionsource/healthy-eating-....
    - TameAntelope 5y ago
      Answered vaguely and in a way that's incompatible with a lifestyle that doesn't desire daily grocery store trips or eating in places where you don't have complete control over the menu... I'd rather know what to avoid, even if you think that question is "wrong" (I don't agree).

PennRobotics 5y ago

When I last looked, Google Scholar doesn't take any action on paper retractions. It seems there is no reporting system in place. Researchers Brock and Thornley of New Zealand published and then almost immediately retracted a paper linking COVID-19 vaccines and a much-higher-than-previously-found miscarriage rate. Scholar still lists the original paper posted---with no cover page indicating the retraction. (They do have a separate link to the retraction press release if you look for it.) The main result on Scholar links to a half dozen "not-so-academic" websites (ellinikahoaxes.gr, wokeguru.org, resistance-mondiale.com) hosting the same paper. I'm willing to bet even if the original paper could be removed, independent websites would continue to try relisting it. While it's important to keep a record of bad research rather than scrubbing it away from history entirely, it seems feasible to have a reporting system in place for paper retractions, where the title link includes [retracted dd-mm-yyyy] or something. This was my small taste of disappointment (aka loss of confidence) in a relatively popular academic tool.

graycat 5y ago

> What happens when the public lose confidence in academic findings? When some X accumulates a track record of not working, people give up on X. Maybe then X will get improved, and slowly people will give it another chance. The OP seemed to be mostly about the quality of the science in the social sciences. Hmm .... But, in medicine, there are efforts to be careful. E.g., can test a new drug with double blind randomized placebo controlled trials. Maybe that methodology is good enough to be trusted? It appears that I trusted it when I got two shots and a booster from Moderna against Covid. In math, hmm ..., if there is a claim of solid proof, published in a respected peer reviewed journal, and so far no one has found any flaw in the proof, then some trust can be justified. The gravitational detection work of LIGO is astounding, and I'm trusting it. And I have yet to check in detail the math of general relativity -- it's on my long term TODO list! And I trust the claims for the Webb Space Telescope, and the reasoning for its location at a Lagrange 2 point. So, in medicine, math, and physics, I can have some trust! For the social sciences, my view is influenced by my wife and brother, both of whom tried those, got Ph.D. degrees, etc.

slowmovintarget 5y ago
Not everyone operates on Bayesian Reasoning. They should, but it requires an awful lot of thought and practice. Updating credences can be painful.

waoush 5y ago

This seems to be heavily focused on studying social phenomena and less so about things like engineering papers or Physics etc. The author himself specializes in social-oriented research. I think it would be better to look at how successful non-social studies are vs humanities. It is debated on whether or not Sociology is even a science at all. For what it is worth, I happen to know that big tech companies follow research at big name CS universities with a bit of interest. At times, even borrowing the research itself in their products. At one point, a professor I worked under was quoted in Android documentation.

zackmorris 5y ago

Mission accomplished. The US has been an a downward slide into privatization and disenfranchisement since at least the early 80s, probably earlier. I was born in the late 70s and have watched the gradual dismantlement of public institutions my entire life. Where this matters is: academics used to be able to conduct pure research. Now all they do is fundraise and hustle for grants. Anyone used to be able to go to college. Now the gatekeepers decide who has enough money or connections to attend. We even used to have news and (gasp) investigative journalism and public broadcasting and periodic reports to the public on the state of publicly funded research or some great public works project. Now we have infotainment and the suppression of any real tech that might materially improve lives by reducing the workload of the average person. They only fixate on phantom tech that distracts and enslaves. Now wonder the public doesn't care. If we really want to get serious about fixing this stuff, it's going to involve at least in some part, a total rejection of vast swaths of the status quo. I'm all for that, I think most Gen Xers are for that, and certainly most people younger than that are for that. But there is just so much concentrated wealth now diluting social movements. And so much momentum behind the previous generation in congress that will do everything it can to hold onto power in the face of so many calamities we face, because at the end of the day, it's not their problem so they're just never gonna get the memo. If a spiritual revolution away from wealth concentration isn't already underway, it's going to start with some version of "your money's no good here." And individuals withdrawing their contribution from institutions which concentrate wealth and power.

dimitrios1 5y ago
While privatization and the fundraising rat race for grants certainly started us down this trend, academia has been it's own worse enemy recently, and it's primary output these days seems to be ideology and activism, rather than cold, hard, factual, evidenced based research and science, so much so, that the activism and ideology has worked its way into the science itself. It's a poisoned well at this point, and it's sad, having worked in a research lab at university myself in the computer vision department, there is still tons of cool things going on, but they all get massively overshadowed by the hot button issues and whatever the Social Justice cause du jour is.

bsedlm 5y ago

> What happens when the public lose confidence in academic findings? flat-earthers half-joking aside, I have come to understand that the academic institutions are part of the state (perhaps I should say civilization?). the point being that academia (as the part of state that I'm saying it is) has a primary function of ensuring stability (same as the most well known face of the state i.e. government); this is mind-bending when faced with the idea that academic research is at the forefront of advancing civilization into uncharted waters (...a PhD must provide an original novel contribution...). Seems that at some point artists became showbiz employees and at that same time, only scholars kept 'advancing the state of the art'. Which is a role historically fulfilled by real artists of the sort that ain't doing it for money nor fame.

kkfx 5y ago

There is a fallacy in comprehend the meaning of Scientific method: confidence in nice and needed to cooperate, but in Science no one should (despite almost all regularly do) trust blindly anything. Of course we can't verify the entire involved knowledge any time, and that's why the concept of collaborative peer review was born, not much different than the concept of "chains of trust": interest researchers in something verify some others publications and add up their own confirmations or doubt, "the network" do the rest. The problem there is that such review and network does not really exists. Oh yes there is the ArXiv, HAL, ... papers circulate etc, but the "mainstream" is a network of hi priced journals and hi paid corporate-backed research + equally hi paid PR in the press. That's why people loose confidence: they do not have ontological means to know but they smell actual Science is mostly commerce and just very little Science. IMVHO the sole way to correct that is build the above mentioned networks. Public, publicly well-founded universities that do research and reviews for the society, very-well separated from the private sector, and share this knowledge with the entire world. At that point the SciHub will be a public, legal and well founded platform and PR can do something but not enough to contaminate knowledge for private interest. At that point people would still not be able to verify, witch is not good, but they have something worth their confidence.

chaps 5y ago

At the start of the pandemic, an academic at a well-known university asked me to review a paper of his, kinda sorta to build a relationship and to eventually publish something of my own based on his paper. The claim of the paper was basically that white (non-hispanic) communities were significantly more affected by COVID when compared against other races in similar income brackets. Lots of statistics, analysis against census data, etc to support the argument and such. The source of the data from his paper was from an "open data" portal of COVID deaths, published by the city's coroner's office. Thing is, the data published by the coroner's office didn't include a lot of fields that I'd already received from FOIA. Fields missing like the name of the person who died. Since the paper's conclusions didn't pass a gut check, I dug in and found that just glancing through the names of those marked down as "white", a majority clearly had names with Hispanic origins. I shared with the researcher that his findings were probably really, really wrong and I was responded with something like, "even if 50% are marked wrong, the findings are still accurate". Some name comparison [1] and spot checking later.. and it ended up being about 80% of hispanic-origin names were marked as "white". The researcher eventually conceded that the paper was wrong, but it was definitely of an eye opening experience into how the academic world works. I lost a lot of faith in the academic world after that. [1] https://www.census.gov/data/developers/data-sets/surnames.html https://www.census.gov/data/developers/data-sets/surnames.ht...

drugstorecowboy 5y ago
I am definitely not a scientist, but speculating on race because of names you find to be too hispanic to be white feels like spectacularly bad science to me. This is not my field though so perhaps that is normal an accepted.
- chaps 5y ago
  The point of name-based analysis wasn't for publication or even "science", but to instead show that the analysis done by the researcher was very likely wrong in order to prevent its publication. There is no way in hell I would ever publish anything that relied on name-based analysis as an accurate measure, that would be deeply unethical as far as I'm concerned. I did ask around with these exact concerns in-mind, which led to this paper by the census in their evaluation of this exact problem (the Passel-Word Spanish Surname List was what I compared against): https://www.census.gov/library/working-papers/1993/demo/POP-twps0004.html https://www.census.gov/library/working-papers/1993/demo/POP-... So, I get it, and I had the same concerns as you, but I like to believe I followed proper due diligence in making sure my approach was reasonable. Especially against the risk of the paper being published (consider how the paper could be used in racist narratives).

ALittleLight 5y ago

One of my favorite stories in the Bible is when the priests of Baal are challenged to a religion showdown by Elijah. The idea is each side will pray for an altar to be lit on fire, whichever altar is lit on fire is the altar of the true religion. I won't spoil the ending - but I think the setup makes a lot of sense. If there is a guy who can ignite stuff by simply asking that it be lit then I would really like to know more about that guy's religion. I have the same opinion of academics - show me what you can light on fire. For example, I don't believe in physics because it makes intuitive sense to me, on the contrary, I find it confusing and counter-intuitive (sometimes). I believe in physics because the people who make magic boxes of light and sound that move fast and connect across vast distances tell me they did so via the knowledge they derived from physics. It's not how I would've guessed the universe would work, but who am I to argue with a blazing altar? Judging by this standard I find many academic disciplines are severely wanting - there is a lot of chanting and waving of hands, but very little smoke. If I hear on the news that there's going to be an eclipse, I'll go out and look. If I hear on the news that eggs are bad for you, I'm going to roll my eyes.

andrewmcwatters 5y ago
You don’t “believe” in physics, you understand there is some truth to it based on direct evidence. Belief is accepting something to be true on the foundation of faith—not having direct evidence. “Now faith is the assurance of things hoped for, the conviction of things not seen.” - Hebrews 11:1 (ESV)
- MattPalmer1086 5y ago
  I disagree with this interpretation of the word belief. A belief may be founded in something, or it may not be. Unquestioning belief without evidence requires faith. Belief is not a synonym for faith.
  - andrewmcwatters 5y ago
    It literally is.

rayiner 5y ago

We're about to find out! The scientific community has done itself a huge disservice in recent years in two ways: 1) Refusing to police non-rigorous fields invoking scientific authority. As the article notes, even rigorous fields are full of results that cannot be reproduced. The situation is even worse in non-rigorous fields. In his famous essay on cargo-cult science, Richard Feynman specifically called out "educational and psychological studies" as "examples of what I would like to call Cargo Cult Science." https://calteches.library.caltech.edu/51/2/CargoCult.htm https://calteches.library.caltech.edu/51/2/CargoCult.htm. Yet today, there is a movement to accord any PhD the respect of the title "doctor," as if an EdD was an authority similar to a PhD in physics. People do this to be polite but it has grave consequences. The respect people accord "science" is, at the end of the day, built on results. We trust doctors because they delivered concrete results that medicine men could not. In the long run, people will notice that EdD is often wrong, because it's not a rigorous field, and that will tarnish the credibility of everyone with a PhD, because the lay public quite reasonably doesn't understand these distinctions in rigor. 2) Allowing scientific authority to be invoked for political purposes. There is a difference between scientific results, and the value systems of individual scientists. Science can, for example, create a vaccine. It cannot address the political question of whether anyone should get the vaccine--that question relies on cost/benefit analysis that, in turn, depends on individual value systems. Or in another example, science can tell you the probability of spreading a disease in a grocery store versus a church, but cannot tell you whether grocery stores or churches are more important. Moreover, most people intuitively understand this distinction. There are, of course, some people who will ignore scientific facts that are as plain as day. But there are a lot more people who will chafe at feeling like they are being told "I'm smarter than you, so you have to accept my values." That, in the long run, will undermine their trust in scientific results. And if we, as highly educated people who understand "science" are honest with ourselves, that skepticism may not be unwarranted. We know that biases exist in the system. For example, it is well known that publication bias exists: studies reporting positive results get published, while studies reporting negative results don't get published: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3341407/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3341407/. As science becomes more politicized, it's inevitable that other kinds of publication bias will affect not only the reported results, but even the kinds of questions that are studied.

godelski 5y ago

I think a lot of comments here are vastly oversimplifying the problem (including this one). People have lost confidence in academic findings due to a wide variety of reasons. There's many people to blame here. There's academics, journals, news/reporting, politicians, and more. Let's look at a few (far from exhaustive and still pretty shallow analysis). Academics: Incentives are frequently to publish fast and often. This encourages work with less detail, higher uncertainty/error, and less impactful work. You simply cannot produce as high quality work in 3 months as you can in 1 year (or 3) or research. Often we want PhD students to produce 5 first author papers in top tier journals before graduating. That's 5-6 years to produce just as many top tier papers (realistically 3-4 given your first 2 years are more class based). It is possible to put out a high quality and high impact paper every year, yes. But this is also a very difficult task. Journals: There's no need to explain this one to anyone who has gone through the review process. Everyone I know that has attempted to publish papers has had several papers where they question what the reviewer actually read because it definitely wasn't their paper. Or where reviewers are vastly under-qualified to review the topic at hand. These are common occurrences. (Note: this more often causes false negatives rather than false positives due to incentive alignments) This creates legitimate complaints and can often be misunderstood from people outside academia. Reporting: We all have seen this. Things like "eating chocolate is healthy" and other things that are obviously wrong. We've seen universities oversell their research. We see news channels oversell or vastly misunderstand research (despite having easy access to experts and that authors are frequently happy to discuss their research). The public also doesn't understand things like confidence levels. They don't understand that we have a different confidence level in anthropocentric climate change vs how much healthier a glass of wine a day is. This isn't their fault though. We should hold the reporters (academic or news sources) accountable since their are structured to be the science communicators. Instead they are politicizing and adding opinions rather than explaining. Politics: We all know this one. Everything is politicized. Not just that, but to the point where things are "left" or "right". If the left supports one thing then the right opposes it and vise versa. This is far worse than both sides accepting the issue and coming at it from different angles (which I've seen this happen and both angles also not be in line with the actual research). Covid is a great example: people on the left over-estimate the dangers and people on the right vastly underestimate them (these are not equally biased positions and I'm not claiming they are, just noting that both have errors, not the magnitude of the errors). Climate change is another example, where right denies and left only focuses on pop/hip topics like tree planting or banning straws and not larger issues. (I'll note that in my anecdotal observations it seems that the magnitude of error is frequently smaller on the left). So we have a system where we aren't encouraged to do high quality research (not meaning it doesn't happen, there are a lot of researchers), research (of any kind) is difficult to get properly peer reviewed, it is poorly explained to laymen, and results become tribal. Why does the public lose confidence in academic findings? You tell me.

TameAntelope 5y ago

I'd argue that absolutely nothing "happens" when the public loses confidence in academic findings, and honestly the public should generally butt out of the entire process. It's not written for them, it's not discussed by them in any rigorous way, their input is not valued or noted, and generally they're completely wrong in very basic ways. Should academic research be done in secret or in a way that actively prevents the public from accessing their results? No. Should we disabuse ourselves of the notion that the public matters to this process? Yes. None of it matters to the lives of nearly anyone until it's gone through many, many layers of filtering. We, the public, ought to let that filtering play out, rather than meddle in the process

sockpuppet_12 5y ago
Public tax funds academia through the govt grants that funds the research.

Animats 5y ago

The worst problems come from economics. As an academic discipline, economics has too little predictive power and too much influence on policy. We have no idea when the economy is close to a crash. The effect of interest rates and tax rates on the economy is poorly understood, and interventions often don't produce intended effects. This is a big problem.

stuckinhell 5y ago

We are finding out in America. After years and years of academic failings from food pyramid nonsense to repeated social science studies failing to replicate. The public is on the tipping point of outright rejecting the science (or more so the holier than thou priestess-esque science class). At my gym, all the personal trainers boldly reject food recommendations from the government or scientific papers. They have applied the scientific method themselves to find a system and foods that work, and even have their own online communities. They also can replicate their results with their clients. At my women's book club, I've noticed more and more mom's are homeschooling their kids in pods, it went from 0% before covid to nearly 50% now, and I live in a pretty affluent liberal area. My husband's fantasy football group talked about inflation since the start of covid, and now are extremely distrustful of the "media" and "experts". I'm watching the breakdown of establishment trust in my own life in a very liberal circle of friends and family. I can't imagine how much worse it is in red america.

WalterBright 5y ago
> I can't imagine how much worse it is in red america. I've lost enormous faith in the media since 2015, when it became obvious that most of the news (other than the weather report) is simply propaganda or clickbait. For example, take the recent airliner crash in China. NBC news ran a video clip of it flying straight down into the ground. That clip turned out to be a fake. The news anchors are completely shameless in the promulgation of their agendas.
tempnow987 5y ago
Somewhat similar. I'm technically highly liberal (I maxed out as a dem donor in a number of races). We are told inflation is passing or a temporary blip by the top econ people in govt. Meanwhile, wage pressure, housing price pressures and more are through the roof. We are told that a portfolio of stocks that invests even in part in fossil fuels is doomed (economically) even though it is a scarce commodity from many volatile regions and that evidence is that ESG wildly outperforms. We are told climate change is the biggest threat, but efforts to solve it are largely forbidden (ie, no nuclear) unless they involve massive sacrifice instead of innovation. We are told it is critical to ban a plastic straw in a restaurant that goes in the garbage but that the tons of trash literally blowing into the ocean from folks in RV's along the water who can't be bothered to throw trash into the cans right there is OK and there should be no consequence. We are told our new LED bulbs won't ficker or buzz when dimmed and will last our lifetimes. Masks won't protect us supposedly from an AIRBORNE virus. A higher risk of arrest by younger men for things like murder is a sign of discrimination against young men, vs any chance of a difference in underlying incidence of murder by young men. We are told Hunter Biden has acted with integrity, that the details of corrupt chinese payments on a laptop are "Russian Disinformation". Um - no. Our kids don't get invited to be on boards of companies they have no experience with, even though they didn't have issues with drugs and are good students. The issue for me is that you don't need to have any knowledge of science to see just HUGE internal inconsistencies in a lot of the "preaching" we receive. This results in a massive reduction of confidence in authority which is a big negative to the liberal movement in my view which is supportive of a bigger role of govt in our lives.

YeGoblynQueenne 5y ago

>> For example, the Blair governments championed evidence-based policymaking. Well, the Blair government is remembered for going to war against Iraq on the basis of evidence that was not sufficiently validated and that was later found to be fabricated: More specifically, the report blamed Secret Intelligence Service (better known as MI6) head Sir Richard Dearlove who presented so-called "hot" intelligence about alleged weapons of mass destruction provided by an Iraqi with "phenomenal access" to high levels in the Iraqi government directly to Blair, without first confirming its accuracy.[43] The investigators found that references to this intelligence in government reports were over-certain and did not adequately stress uncertainties and nuance. The informant was later found to have been lying. The Chilcot report states that "personal intervention [by Dearlove] and its urgency gave added weight to a report that had not been properly evaluated and would have coloured the perception of ministers and senior officials".[44] The day after the report was published, Blair conceded that he should have challenged such intelligence reports before relying on them to justify military action in Iraq.[43][45] https://en.wikipedia.org/wiki/Iraq_Inquiry#Case_for_war_was_deficient https://en.wikipedia.org/wiki/Iraq_Inquiry#Case_for_war_was_... So perhaps it's not only the fault of those pesky liberal intellectuals and their rabid postmodernism that the public has lost its confidence to all sorts of institutions. I also blame the Monty Pythons, for example.

theknocker 5y ago