9 ms·

A flawed paper in management science has been cited more than 6k times

• shermozle 9mo ago
  Management WHAT?
• dgxyz 9mo ago
  Not even surprised. My daughter tried to reproduce a well-cited paper a couple of years back as part of her research project. It was not possible. They pushed for a retraction but university don't want to do it because it would cause political issues as one of the peer-reviewers is tenured at another closely associated university. She almost immediately fucked off and went to work in the private sector.
  • jruohonen 9mo ago
    > They pushed for a retraction ... That's not right; retractions should only be for research misconduct cases. It is a problem with the article's recommendations too. Even if a correction is published that the results may not hold, the article should stay where it is. But I agree with the point about replications, which are much needed. That was also the best part in the article, i.e. "stop citing single studies as definitive".
    • dgxyz 9mo ago
      I will add it's a little more complicated than I wanted to let on here as I don't identify it in the process. But it definitely was misconduct on this one. I read the paper as well. My background is mathematics and statistics and the data was quite frankly synthesised.
      • jruohonen 9mo ago
        Okay, but to return to replications, publishers could incentivize replications by linking replication studies directly on a paper's website location. In fact, you could even have a collection of DOIs for these purposes, including for datasets. With this point in mind, what I find depressing is that the journal declined a follow-up comment. But the article is generally weird or even harmful too. Going to social media with these things and all; we have enough of that "pretty" stuff already.
        • dgxyz 9mo ago
          Agree completely on all points. However there are two problems with it. Firstly it's a step towards gamification and having tried that model in a fintech on reputation scoring, it was a bit of a disaster. Secondarily, very few studies are replicated in the first place unless there is a demand for linked research to replicate it before building on it. There are also entire fields which are mostly populated by bullshit generators. And they actively avoid replication studies. Certain branches of psychology are rather interesting in that space.
          • jruohonen 9mo ago
            > Certain branches of psychology are rather interesting in that space. Maybe, I cannot say, but what I can say is that CS is in the midst of a huge replication crisis because LLM research cannot be replicated by definition. So I'd perhaps tone down the claims about other fields.
            • dgxyz 9mo ago
              Another good example that for sure. You won't find me having any positive comments about LLMs.
  • kelipso 9mo ago
    It’s much much more likely that she did something wrong trying to replicate it than the paper was wrong. Did she try to contact the authors, discuss with her advisor? Pushing for retraction just like that and going off to private sector is…idk it’s a decision.
    • dgxyz 9mo ago
      It went on for a few months. The source data for the paper was synthesised and it was like trying to get blood out of a stone trying to get hold of it, clearly because they knew they were in trouble. Lots of research money was wasted trying to reproduce it. She was just done with it then and a pharma company said "hey you fed up with this shit and like money?" and she was and does. edit: as per the other comment, my background is mathematics and statistics after engineering. I went into software but still have connections back to academia which I left many years ago because it was a political mess more than anything. Oh and I also like money.
  • dekhn 9mo ago
    A single failure to reproduce a well-cited paper does not constitute grounds for a retraction unless the failure somehow demonstrates the paper is provably incorrect.
• nairboon 9mo ago
  Nowadays high citation numbers don't mean anymore what they used to. I've seen too many highly cited papers with issues that keep getting referenced, probably because people don't really read the sources anymore and just copy-paste the citations. On my side-project todo list, I have an idea for a scientific service that overlays a "trust" network over the citation graph. Papers that uncritically cite other work that contains well-known issues should get tagged as "potentially tainted". Authors and institutions that accumulate too many of such sketchy works should be labeled equally. Over time this would provide an additional useful signal vs. just raw citation numbers. You could also look for citation rings and tag them. I think that could be quite useful but requires a bit of work.
  • boelboel 9mo ago
    Going to conferences seeing researchers who've built a career doing subpar (sometimes blatantly 'fake') work has made me grow increasingly wary of experts. Worst is lots of people just seem to go along with it. Still I'm skeptical about any sort of system trying to figure out 'trust'. There's too much on the line for researchers/students/... to the point where anything will eventually be gamed. Just too many people trying to get into the system (and getting in is the most important part).
    • mezyt 9mo ago
      The worse system is already getting gamed. There's already too much on the line for researchers/students, so they don't admit any wrong doing or retract anything. What's the worse that could happen by adding a layer of trust in the h-index ?
      • boelboel 9mo ago
        I think it could end up helping a bit in the short term. But in the end an even more complicated system (even if in principle better) will reward those spending time gaming it even more. The system ends up promoting an even more conservative culture. What might start great will end up with groups and institutions being even more protective of 'their truths' to avoid getting tainted. Don't think there's any system which can avoid these sort of things, people were talking about this before WW1, globalisation just put it in overdrive.
• renewiltord 9mo ago
  Family member tried to do work relying on previous results from a biotech lab. Couldn’t do it. Tried to reproduce. Doesn’t work. Checked work carefully. Faked. Switched labs and research subject. Risky career move, but. Now has a career. Old lab is in mental black box. Never to be touched again. Talked about it years ago https://news.ycombinator.com/item?id=26125867 https://news.ycombinator.com/item?id=26125867 Others said they’d never seen it. So maybe it’s rare. But no one will tell you even if they encounter. Guaranteed career blackball.
  • rcxdude 9mo ago
    I haven't identified an outright fake one but in my experience (mainly in sensor development) most papers are at the very least optimistic or are glossing over some major limitations in the approach. They should be treated as a source of ideas to try instead of counted on. I've also seen the resistance that results from trying to investigate or even correct an issue in a key result of a paper. Even before it's published the barrier can be quite high (and I must admit that since it's not my primary focus and my name was not on it, I did not push as hard as I could have on it)
    • dekhn 9mo ago
      When I was a postdoc, I wrote up the results from a paper based on theories from my advisor. The paper wasn't very good- all the results were bad. Overnight, my advisor rewrote all the results of the paper, partly juicing the results, and partly obscuring the problems, all while glossing over the limitations. She then submitted it to a (very low prestige) journal. I read the submitted version and told her it wasn't OK. She withdrew the paper and I left her lab shortly after. I simply could not stand the tendency to juice up papers, and I didn't want to have my reputation tainted by a paper that was false (I'm OK with my reputation being tainted by a paper that was just not very good). What really bothers me is when authors intentionally leave out details of their method. There was a hot paper (this was ~20 years ago) about a computational biology technique ("evolutionary trace") and when we did the journal club, we tried to reproduce their results- which started with writing an implementation from their description. About half way through, we realized that the paper left out several key steps, and we were able to infer roughly what they did, but as far as we could tell, it was an intentional omission made to keep the competition from catching up quickly.
• thewanderer1983 9mo ago
  did “not impact the main text, analyses, or findings.” Made me think of the black spoon error being off by a factor of 10 and the author also said it didn't impact the main findings. https://statmodeling.stat.columbia.edu/2024/12/13/how-a-simple-math-error-sparked-a-panic-about-black-plastic-kitchen-utensils/ https://statmodeling.stat.columbia.edu/2024/12/13/how-a-simp...
• flowerthoughts 9mo ago
  > This doesn’t mean that the authors of that paper are bad people! > We should distinguish the person from the deed. We all know good people who do bad things > They were just in situations where it was easier to do the bad thing than the good thing I can't believe I just read that. What's the bar for a bad person if you haven't passed it at "it was simply easier to do the bad thing?" In this case, it seems not owning up to the issues is the bad part. That's a choice they made. Actually, multiple choices at different times, it seems. If you keep choosing the easy path instead of the path that is right for those that depend on you, it's easier for me to just label you a bad person.
  • psychoslave 9mo ago
    Seems fair in the frame of what is responded. But there is a concern which goes out of the "they" here. Actually, "they" could just as well not exist, and all narrative in the article be some LLM hallucination, we are still training ourself how we respond to this or that behavior we can observe and influence how we will act in the future. If we go with the easy path labeling people as root cause, that's the habit we are forging for ourself. We are missing the opportunity to hone our sense of nuance and critical thought about the wider context which might be a better starting point to tackle the underlying issue. Of course, name and shame is still there in the rhetorical toolbox, and everyone and their dog is able to use it even when rage and despair is all that stay in control of one mouth. Using it with relevant parcimony however is not going to happen from mere reactive habits.
  • j3th9n 9mo ago
    [flagged]
    • Hasnep 9mo ago
      Focusing on criticising people's actions and being lenient and not judging the person's character is literally centuries old, I don't think you can say it's because of woke.
      • deleted 9mo ago
• necovek 9mo ago
  Being practical, and understanding the gamification of citation counts and research metrics today, instead of going for a replication study and trying to prove a negative, I'd instead go for contrarian research which shows a different result (or possibly excludes the original result; or possibly doesn't even if it does not confirm it). These probably have bigger chance of being published as you are providing a "novel" result, instead of fighting the get-along culture (which is, honestly, present in the workplace as well). But ultimately, they are (research-wise! but not politically) harder to do because they possibly mean you have figured out an actual thing. Not saying this is the "right" approach, but it might be a cheaper, more practical way to get a paper turned around. Whether we can work this out in research in a proper way is linked to whether we can work this out everywhere else? How many times have you seen people tap each other on the back despite lousy performance and no results? It's just easier to switch private positions vs research positions, so you'll have more of them not afraid to highlight bad job, and well, there's this profit that needs to pay your salary too.
  • em500 9mo ago
    Most of these studies get published based on elaborate constructions of essentially t-tests for differences in means between groups. Showing the opposite means showing no statistical difference, which is almost impossible to get published, for very human reasons.
    • necovek 9mo ago
      My point was exactly not to do that (which is really an unsuccesfull replication), but instead to find the actual, live correlation between the same input rigourously documented and justified, and new "positive" conclusion. As I said, harder from a research perspective, but if you can show, for instance, that sustainable companies are less profitable with a better study, you have basically contradicted the original one.
• deleted 9mo ago
  [deleted]
• psychoslave 9mo ago
  Social fame is fundamentally unscalable, as it operates in limited room on the scene and even less in the few spot lights. Benefits we can get from collective works, including scientific endeavors, are indefinitely large, as in far more important than what can be held in the head of any individual. Incitives are just irrelevant as far as global social good is concerned.
• fnord123 9mo ago
  > Stop citing single studies as definitive. They are not. Check if the ones you are reading or citing have been replicated. And from the comments: > From my experience in social science, including some experience in managment studies specifically, researchers regularly belief things – and will even give policy advice based on those beliefs – that have not even been seriously tested, or have straight up been refuted. Sometimes people use fewer than one non replicatable studies. They invent studies and use that! An example is the "Harvard Goal Study" that is often trotted out at self-review time at companies. The supposed study suggests that people who write down their goals are more likely to achieve them than people who do not. However, Harvard itself cannot find such a study existing: https://ask.library.harvard.edu/faq/82314 https://ask.library.harvard.edu/faq/82314
  • KingMob 9mo ago
    Definitely ignore single studies, no matter how prestigious the journal or numerous the citations. Straight-up replications are rare, but if a finding is real, other PIs will partially replicate and build upon it, typically as a smaller step in a related study. (E.g., a new finding about memory comes out, my field is emotion, I might do a new study looking at how emotion and your memory finding interact.) If the effect is replicable, it will end up used in other studies (subject to randomness and the file drawer effect, anyway). But if an effect is rarely mentioned in the literature afterwards...run far, FAR away, and don't base your research off it. A good advisor will be able to warn you off lost causes like this.
  • ChrisMarshallNY 9mo ago
    Check out the “Jick Study,” mentioned in Dopesick. https://en.wikipedia.org/wiki/Addiction_Rare_in_Patients_Treated_with_Narcotics https://en.wikipedia.org/wiki/Addiction_Rare_in_Patients_Tre...
    • NedF 9mo ago
      [dead]
• shiandow 9mo ago
  I appreciate the convenience of having the original text on hand, as opppsed to having to download it of Dropbox of all places. But if you're going to quote the whole thing it seems easier to just say so rather than quoting it bit by bit interspersed with "King continues" and annotating each I with [King].
• zahirbmirza 9mo ago
  Could you also provide your critical appraisal of the article so this can be more of a journal club for discussion vs just a paper link? I have no expertise in this field so would be good for some insights.
• indubioprorubik 9mo ago
  And thus all citing, have fatally flawed there paper if its central to the thesis, thus, he who proofs the root is rotten, should gain there funding from this point forward.
  • indubioprorubik 9mo ago
    I see this approach as a win win for science. Debunking bad science becomes a for profit enterprise, rigorous science becomes the only one sustainable, the paper churn gets reduced, as even producing a good one becomes a financial risk, when it becomes foundational and gets debunked later.
• motbus3 9mo ago
  I will not go into the details of the topic but the "What to do" is the most obvious thing. If a paper that is impactful cannot be backed by other works that should be a smell
• ChrisMarshallNY 9mo ago
  Sounds like the Watergate Scandal. The crime was one thing, but it was the cover-up that caused the most damage. Once something enters The Canon, it becomes “untouchable,” and no one wants to question it. Fairly classic human nature. > "The most erroneous stories are those we think we know best -and therefore never scrutinize or question." -Stephen Jay Gould
• steve-atx-7600 9mo ago
  The title alone is sus. I guess there are a lot of low quality papers out there in sciencey sounding fields.
  • rwmj 9mo ago
    The journal name ("Management Science") is a bit of a giveaway too.
    • ykonstant 9mo ago
      Join me in my new business endeavor where we found the Journal for Journal Science.
• pbhjpbhj 9mo ago
  Isn't at least part of the problem with replication that journals are businesses. They're selling in part based on limited human focus, and on desire to see something novel, to see progress in one's chosen field. Replications don't fit a commercial publications goals. Institutions could do something, surely. Require one-in-n papers be a replication. Only give prizes to replicated studies. Award prize monies split between the first two or three independent groups demonstrating a result. The 6k citations though ... I suspect most of those instances would just assert the result if a citation wasn't available.
  • arter45 9mo ago
    Not in academia myself, but I suspect the basic issue is simply that academics are judged by the number of papers they publish. They are pushed to publish a lot, which means journals have to review a lot of stuff (and they cannot replicate findings on their own). Once a paper is published on a decent journal, other researchers may not "waste time" replicating all findings, because they also want to publish a lot. The result is papers getting popular even if no one has actually bothered to replicate the results, especially if those papers are quoted by a lot of people and/or are written by otherwise reputable people or universities.
  • mike_hearn 9mo ago
    Journals aren't really businesses in the conventional sense. They're extensions of the universities: their primary customers and often only customers are university libraries, their primary service is creating a reputation economy for academics to decide promotions. If the flow of tax, student debt and philanthropic money were cut off, the journals would all be wiped out because there's no organic demand for what they're doing.
• Havoc 9mo ago
  Maybe that's why it gets cited? People starting with an answer and backfilling?
• loxodrome 9mo ago
  Do people actually take papers in "management science" seriously?
  • abanana 9mo ago
    Yes, that's the problem, many do, and they swear by these oversimplified ideas and one-liners that litter the field of popular management books, fully believing it's all "scientific" and they'll laugh at you for questioning it. It's nuts.
    • graemep 9mo ago
      There is a difference between popular management books and academic publications. For example there is a long history of studies of the relationship between working hours and productivity which is one of the few things that challenges the idea that longer hours means more output.
      • abanana 9mo ago
        Yes, but the books generally take their ideas from the academic publications. And the replication problems, and general incentives around academic publishing, show that all too often, the academic publications in the social sciences are unfortunately no more rigorous than the populist books.
        • graemep 9mo ago
          That is true, but the popular books both simplify and cherry pick which makes it a whole to worse.
  • malshe 9mo ago
    They do and there is nothing wrong with that. The papers published in this journal are peer-reviewed and go through multiple rounds of review. Also, note that Andrew King could carry out the replication because the data is publicly available.
• jokoon 9mo ago
  It's harder to do social/human science because it's just easier to make mistakes that leads to bias. It's harder to do in maths, physics, biology, medecine, astronomy, etc. I often say that "hard sciences" have often progressed much more than social/human sciences.
  • marginalia_nu 9mo ago
    Funny you say that, as medicine is one of the epicenters of the replication crisis[1]. [1] https://en.wikipedia.org/wiki/Replication_crisis#In_medicine https://en.wikipedia.org/wiki/Replication_crisis#In_medicine
    • QuadmasterXLII 9mo ago
      you get a replication crisis on the bleeding edge between replication being possible and impossible. There’s never going to be a replication crisis in linear algebra, there’s never going to be a replication crisis in theology, there definitely was a replication crisis in psych and a replication crisis in nutrition science is distinctly plausible and would be extremely good news for the field as it moves through the edge.
      • nickpsecurity 9mo ago
        Leslie Lamport came up with a structured method to find errors in proof. Testing it on a batch, he found most of them had errors. Peter Guttman's paper on formal verification likewise showed many "proven" or "verified" works had errors that were spottes quickly upon informal review or testing. We've also see important theories in math and physics change over time with new information. With the above, I think we've empirically proven that we can't trust mathmeticians more than any other humans We should still rigorously verify their work with diverse, logical, and empirical methods. Also, build ground up on solid ideas that are highly vetted. (Which linear algebra actually does.) The other approach people are taking are foundational, machine-checked, proof assistants. These use a vetted logic whose assistant produces a series of steps that can be checked by a tiny, highly-verified checker. They'll also oftne use a reliable formalism to check other formalisms. The people doing this have been making everything from proof checkers to compilers to assembly languages to code extraction in those tools so they are highly trustworthy. But, we still need people to look at the specs of all that to see if there are spec errors. There's fewer people who can vet the specs than can check the original English and code combos. So, are they more trustworthy? (Who knows except when tested empirically on many programs or proofs, like CompCert was.)
• tgv 9mo ago
  The root of the problem is referred to implicitly: publish or perish. To get tenure, you need publications, preferably highly cited, and money, which comes from grants that your peers (mostly from other institutions) decide on. So the mutual back scratching begins, and the publication mill keeps churning out papers whose main value is the career of the author and --through citation-- influential peers, truth be damned.
  • zero_shift 9mo ago
    something something Goodhart's Law
    • te7447 9mo ago
      Something "systems that are attacked by entities that adapt often need to be defended by entities that adapt".
  • bicepjai 9mo ago
    The same dynamics from school carry over into adulthood: early on it’s about grades and whether you get into a “good” school; later it becomes the adult version of that treadmill : publish or perish.
  • strangattractor 9mo ago
    Citations being the only metric is one problem. Maybe an improved rating/ranking system would be helpful. Ranking 1 to 3 - 1 being the best - 3 the bare minimum for publication. 3. Citations only 2. Citations + full disclosure of data. 1. Citations + full disclosure of data + replicated
    • nick486 9mo ago
      this will arguably be worse. you'll just get replication rings in addition to citation rings. People who cheat in their papers will have no issues cheating in their replication studies too. All this does, is give them a new tool to attack papers they don't like by faking a failed replication.
• dist-epoch 9mo ago
  In the past the elite would rule the plebs by saying "God says so, so you must do this". Today the elites rule the plebs by saying "Science sasy so, so you must do this". Author doesn't seem to understand this, the purpose of research papers is to be gospel, something to be believed, not scrutinized.
  • abanana 9mo ago
    That's a very good point. Some of what's called "science" today, in popular media and coming from governments, is religion. "We know all, do not question us." It's the common problem of headlines along the lines of "scientists say" or "The Science says", which should always be a red flag - but the majority of people believe it.
  • graemep 9mo ago
    In fact, religious ideas (at least in Europe) were often in opposition to the ruling elite (and still are) and even inspired rebellion: https://en.wikipedia.org/wiki/John_Ball_(priest) https://en.wikipedia.org/wiki/John_Ball_(priest) There is a reason scriptures were kept away from the oppressed, or only made available to them in a heavily censored form (e.g. the Slaves Bible).
  • Throaway1982 9mo ago
    A little more complicated than that. In the past, the elites said "don't read the religious texts, WE will tell you what's in them."
    • mike_hearn 9mo ago
      Scientists say that today too, it's a standard response if people outside of academia critique their work. "That person is not an expert" - totally normal response, it's taken to be a killer rebuttal by journalists and politicians.
      • Throaway1982 9mo ago
        Not exactly...in the past the Bible was literally not allowed to be translated from Latin into local languages. Ordinary people were 100% reliant on the elites to tell them what was in it.
• poemxo 9mo ago
  > They intended to type “not significant” but omitted the word “not.” This one is pretty egregious.
  • B1FIDO 9mo ago
    Once, back around 2011 or 2012, I was using Google Translate for a speech I was to deliver in church. It was shorter than one page printed out. I only needed the Spanish translation. Now I am proficient in spoken and written Spanish, and I can perfectly understand what is said, and yet I still ran the English through Google Translate and printed it out without really checking through it. I got to the podium and there was a line where I said "electricity is in the air" (a metaphor, obviously) and the Spanish translation said "electricidad no está en el aire" and I was able to correct that on-the-fly, but I was pissed at Translate, and I badmouthed it for months. And sure, it was my fault for not proofing and vetting the entire output, but come on!
    • aidenn0 9mo ago
      Similar timeframe, I used it for translating some German into English. I'm a native English speaker who has spent some time in Germany (but had not spoken any German in over a decade at this point) and quickly noticed some things were off. After reviewing the original text I realized that every single separable verb[1] that was not in infinitive form was mistranslated. This is an astoundingly bad systematic error for a machine translation program to have. 1: https://en.wikipedia.org/wiki/Separable_verb https://en.wikipedia.org/wiki/Separable_verb
• dev_l1x_be 9mo ago
  There is a surprisingly large amount of bad science out there. And we know it. One of my favourite writeup on the subject: John P. A. Ioannidis: Why Most Published Research Findings Are False https://pmc.ncbi.nlm.nih.gov/articles/PMC1182327/pdf/pmed.0020124.pdf https://pmc.ncbi.nlm.nih.gov/articles/PMC1182327/pdf/pmed.00...
  • FabHK 9mo ago
    John Ioannidis is a weird case. His work on the replication crisis across many domains was seminal and important. His contrarian, even conspiratorial take on COVID-19 not so much.
    • raddan 9mo ago
      Ugh, wow, somehow I missed all this. I guess he joins the ranks of the scientists who made important contributions and then leveraged that recognition into a platform for unhinged diatribes.
      • kelipso 9mo ago
        What’s happening here? “Most Published Research Findings Are False” —> “Most Published COVID-19 Research Findings Are False” -> “Uh oh, I did a wrongthink, let’s backtrack at bit”. Is that it?
        • mike_hearn 9mo ago
          Yes, sort of. Ioannidis published a serosurvey during COVID that computed a lower fatality rate than the prior estimates. Serosurveys are a better way to compute this value because they capture a lot of cases which were so mild people didn't know they were infected, or thought it wasn't COVID. The public health establishment wanted to use an IFR as high as possible e.g. the ridiculous Verity et al estimates from Jan 2020 of a 1% IFR were still in use more than a year later despite there being almost no data in Jan 2020, because high IFR = COVID is more important = more power for public health. If IFR is low then a lot of the assumptions that justified lockdowns are invalidated (the models and assumptions were wrong anyway for other reasons, but IFR is just another). So Ioannidis was a bit of a class traitor in that regard and got hammered a lot. The claim he's a conspiracy theorist isn't supported, it's just the usual ad hominem nonsense (not that there's anything wrong with pointing out genuine conspiracies against the public! That's usually called journalism!). Wikipedia gives four citations for this claim and none of them show him proposing a conspiracy, just arguing that when used properly data showed COVID was less serious than others were claiming. One of the citations is actually of an article written by Ioannidis himself. So Wikipedia is corrupt as per usual. Grokipedia's article is significantly less biased and more accurate.
• gus_massa 9mo ago
  The webpage of the journal [1] only says 109 citations of the original article, this count only "indexed" journals, that are not guaranty to be ultra high quality but at least filter the worse "pay us to publish crap" journals. ResearchGate says 3936 citations. I'm not sure what they are counting, probably all the pdf uploaded to ResearchGate I'm not sure how they count 6000 citations, but I guess they are counting everything, including quotes by the vicepresident. Probably 6001 after my comment. Quoted in the article: >> 1. Journals should disclose comments, complaints, corrections, and retraction requests. Universities should report research integrity complaints and outcomes. All comments, complaints, corrections, and retraction requests? Unmoderated? Einstein articles will be full of comments explaining why he is wrong, from racist to people that can spell Minkowski to save their lives. In /newest there is like one post per week from someone that discover a new physics theory with the help of ChatGPT. Sometimes it's the same guy, sometimes it's a new one. [1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1964011 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1964011 [2] https://www.researchgate.net/publication/279944386_The_Impact_of_Corporate_Sustainability_on_Organizational_Processes_and_Performance https://www.researchgate.net/publication/279944386_The_Impac...
  • Calavar 9mo ago
    > All comments, complaints, corrections, and retraction requests? Unmoderated? Einstein articles will be full of comments explaining why he is wrong, from racist to people that can spell Minkowski to save their lives. In /newest there is like one post per week from someone that discover a new physics theory with the help of ChatGPT. Sometimes it's the same guy, sometimes it's a new one. Judging from PubPeer, which allows people to post all of the above anonymously and with minimal moderation, this is not an issue in practice.
    • bee_rider 9mo ago
      They mentioned a famous work, which will naturally attract cranks to comment on it. I’d also expect to get weird comments on works with high political relevance.
    • gus_massa 9mo ago
• chmod775 9mo ago
  Pretty much all fields have shit papers, but if you ever feel the need to develop a superiority complex, take a vacation from your STEM field and have a look at what your university offers under the "business"-anything label. If anyone in those fields manages to produce anything of quality, they're defying the odds and should be considered one of the greats along the line of Euclid, Galileo Galilei, or Isaac Newton - because they surely didn't have many shoulders to stand on either.
  • HPsquared 9mo ago
    I suppose it's to be expected, the business department is built around the art of generating profit from cheap inputs. It's business thinking in action!
  • lordnacho 9mo ago
    This is exactly how I felt when studying management as part of ostensibly an Engineering / Econ / Management degree. When you added it up, most of the hard parts were Engineering, and a bit Econ. You would really struggle to work through tough questions in engineering, spend a lot of time on economic theory, and then read the management stuff like you were reading a newspaper. Management you could spot a mile away as being soft. There's certainly some interesting ideas, but even as students we could smell it was lacking something. It's just a bit too much like a History Channel documentary. Entertaining, certainly, but it felt like false enlightenment.
    • seec 9mo ago
      Econ is the only social science that isn't completely bogus. The replication rate isn't too bad, even though it is still worse than STEM of course. Everything else is basically like rolling a dice or even worse. Special mention to "pedagogy," which manages to be systematically worse than random; in other words, they only produce bullshit and not much else.
• Dumblydorr 9mo ago
  Does it bug anyone else when your article has so many quotes it’s practically all italics? Change the formatting style so we don’t have to read pages of italic quotes
  • shusaku 9mo ago
    This drove me nuts, but also the authors should like get to the point about what was wrong instead of dancing around it for page after page.
• deleted 9mo ago
  [deleted]
• bronlund 9mo ago
  This likely represents only a fragment of a larger pattern. Research contradicting prevailing political narratives faces significant professional obstacles, and as this article shows, so does critiques of research that don't.
• Biologist123 9mo ago
  Not enough is understood about the replication crisis in the social sciences. Or indeed in the hard sciences. I do wonder whether this is something that AI will rectify.
  • moolcool 9mo ago
    How would AI do anything to rectify it?
    • Levitz 9mo ago
      The same way it would correct typos in a text. It's just a tool, you tell it to find inconsistencies, see what results that yields, and optimize it for verification of claims.
  • buckle8017 9mo ago
    it will not, ai reads and "believes" the heavily cited but incorrect papers.
• gdevenyi 9mo ago
  Welcome Ideological science published to support the regime. There's a lot more where this came from .
• bluecalm 9mo ago
  The problem with academia is that it's often more about politics and reputation than seeking the truth. There are multiple examples of researchers making a career out flawed papers and never retracking or even admitting a mistake. All the talks they were invited to give, all the followers they had, all the courses they sold and impact factor they have built. They are not going to came forward and say "I misinterpreted the data and made long reaching conclusions that are nonsense, sorry for misleading you and thousands of others". The process protects them as well. Someone can publish another paper, make different conclusions. There is 0 effort get to the truth, to tell people what is and what isn't current consensus and what is reasonable to believe. Even if it's clear for anyone who digs a bit deeper it will not be communicated to the audience the academia is supposed to serve. The consensus will just quietly shift while the heavily quoted paper is still there. The talks are still out there, the false information is still propagated while the author enjoys all the benefits and suffers non of the negative consequences. If it functions like that I don't think it's fair that tax payer funds it. It's there to serve the population not to exist in its own world and play its own politics and power games.
• throwaway150 9mo ago
  > I’ve been in the car with some drunk drivers, some dangerous drivers, who could easily have killed people: that’s a bad thing to do, but I wouldn’t say these were bad people. If this isn't bad people, then who can ever be called bad people? The word "bad" loses its meaning if you explain away every bad deed by such people as something else. Putting other people's lives at risk by deciding to drive when you are drunk sounds like very bad people to me. > They’re living in a world in which doing the bad thing–covering up error, refusing to admit they don’t have the evidence to back up their conclusions–is easy, whereas doing the good thing is hard. I don't understand this line of reasoning. So if people do bad things because they know they can get away with it, they aren't bad people? How does this make sense? > As researchers they’ve been trained to never back down, to dodge all criticism. Exactly the opposite is taught. These people are deciding not to back down and admit wrong doing out of their own accord. Not because of some "training".
  • brabel 9mo ago
    When everyone else does it, it's extremely hard to be righteous. I did it long ago... everyone did it back then. We knew the danger and thought we were different, we thought we could drive safely no matter our state. Lots of tragedies happen because people disastrously misjudge their own abilities, and when alcohol is involved doubly so. They are not bad people, they're people who live in a flawed culture where alcohol is seen as acceptable and who cannot avoid falling for the many human fallacies... in this case caused by the Dunning Kruger effect. If you think people who fall for fallacies are bad, then being human is inherently bad in your opinion.
    • throwaway150 9mo ago
      I don't think being human is inherently bad. But you have to draw the line to consider someone as "bad" somewhere, right? If you don't draw a line, then nobody in the world is a bad person. So my question is where exactly is that line? You guys are saying that drink driving does not make someone a bad person. Ok. Let's say I grant you that. Where do you draw the line for someone being a bad person? I mean with this line of reasoning you can "explain way" every bad deed and then nobody is a bad person. So do you guys consider someone to be actually a bad person and what did they have to do to cross that line where you can't explain away their bad deed anymore and you really consider them to be bad?
• striking 9mo ago
  This link may be blogspam of https://www.linkedin.com/pulse/how-institutional-failures-undermine-trust-science-andrew-king-2d2ue https://www.linkedin.com/pulse/how-institutional-failures-un...
• learingsci 9mo ago
  I was young once too. “Your email is too long.” This whole thing is filled with “yeah, no s**” and lmao. More seriously, pretty sure the whole ESG thing has been debunked already, and those who care to know the truth already know it. A good rule of thumb is to be skeptical of results that make you feel good because they “prove” what you want them to.
• SeanLuke 9mo ago
  I developed and maintain a large and very widely used open source agent-based modeling toolkit. It's designed to be very highly efficient: that's its calling card. But it's old: I released its first version around 2003 and have been updating it ever since. Recently I was made aware by colleagues of a publication by authors of a new agent-based modeling toolkit in a different, hipper programming language. They compared their system to others, including mine, and made kind of a big checklist of who's better in what, and no surprise, theirs came out on top. But digging deeper, it quickly became clear that they didn't understand how to run my software correctly; and in many other places they bent over backwards to cherry-pick, and made a lot of bold and completely wrong claims. Correcting the record would place their software far below mine. Mind you, I'm VERY happy to see newer toolkits which are better than mine -- I wrote this thing over 20 years ago after all, and have since moved on. But several colleagues demanded I do so. After a lot of back-and-forth however, it became clear that the journal's editor was too embarrassed and didn't want to require a retraction or revision. And the authors kept coming up with excuses for their errors. So the journal quietly dropped the complaint. I'm afraid that this is very common.
  • bargle0 9mo ago
    If you’re the same Sean Luke I’m thinking of: I was an undergraduate at the University of Maryland when you were a graduate student there in the mid nineties. A lot of what you had to say shaped the way I think about computer science. Thank you.
    • domoregood 9mo ago
      Comments like this are the best part HN.
    • deleted 9mo ago
      [deleted]
    • sizzle 9mo ago
      Imagine if you did a bootcamp instead
  • deleted 9mo ago
• ungreased0675 9mo ago
  The paper publishing industry has a tragedy of the commons problem. Individual authors benefit from fake or misrepresented research. Over time more and more people roll their eyes when they hear “a study found…” Over a long period it depreciates science and elevates superstition. For example, look at how people interact with LLMs. Lots of superstition (take a deep breath) not much reading about the underlying architecture.
• MORPHOICES 9mo ago
  [dead]
• nickpsecurity 9mo ago
  I think what these papers prove is my newer theory that organized science isn't scientific at all. It's mostly unverified claims by people rewarded for throwing papers out that look scientific, have novelty, and achieve policy goals of specific groups. There's also little review with dissent banned in many places. We've been calling it scientism since it's like a self-reinforcing religion. We need to throw all of this out by default. From public policy to courtrooms, we need to treat it like any other eyewitness claim. We shouldn't beleive anything unless it has strong arguments or data backing it. For science, we need the scientific method applied with skeptical review and/or replication. Our tools, like statistical methods and programs, must be vetted. Like with logic, we shouldn't allow them to go beyond what's proven in this way. So, only the vetted claims are allowed as building blocks (premises) in newly-vetted work. The premises must be used how they were used before. If not, they are re-checked for the new circumstances. Then, the conclusions are stated with their preconditions and limitations to only he applied that way. I imagine many non-scientists and taxpayers assumed what I described is how all these "scientific facts" and "consensus" vlaims were done. The opposite was true in most cases. So, we need to not onoy redo it but apply scientific method to the institutions themselves assessing their reliability. If they don't get reliable, they loose their funding and quickly. (Note: There are groups in many fields doing real research and experimental science. We should highlight them as exemplars. Maybe let them take the lead in consulting for how to fix these problems.)
  • esseph 9mo ago
    I have a Growing Concern with our legal systems. > We need to throw all of this out by default. From public policy to courtrooms, we need to treat it like any other eyewitness claim. If you can't trust eyewitness claims, if you can't trust video or photographic or audio evidence, then how does one Find Truth? Nobody really seems to have a solid answer to this.
    • nickpsecurity 9mo ago
      It's specific segments of people saying we can't trust eyewitness claims. They actually work well enough that we run on them from childhood to adulthood. Accepting that truth is the first step. Next, we need to understand why that is, which should be trusted, and which can't be. Also, what methods to use in what contexts. We need to develop education for people about how humanity actually works. We can improve steadily over time. On my end, I've been collecting resources that might be helpful. That includes Christ-centered theology with real-world application, philosophies of knowledge with guides on each one, differences between real vs organized science, biological impact on these, dealing with media bias (eg AllSides), worldview analyses, critical thinking (logic), statistical analyses (esp error spotting), writing correct code, and so on. One day, I might try to put it together into a series that equips people to navigate all of this stuff. For right now, I'm using it as a refresher to improve my own abilities ahead of entering the Data Science field.
• glitchc 9mo ago
  This is simply a case of appeal to authority. No reviewer or editor would reject a paper from either HBS or LBS, let alone a joint paper between the two. Doing so would be akin to career suicide. And therein lies the uncomfortable truth: Collaborative opportunities take priority over veracity in publications every time.
  • cloud-oak 9mo ago
    That's why double-blind review shohld be the norm. It's wild to me that single-blind is still the norm in kost disciplines.
• drob518 9mo ago
  We’ve developed a “leaning tower of science.” Someday, it’s going to fall.
• deleted 9mo ago
  [deleted]
• efitz 9mo ago
  The paper touches on a point (“sustainability “) that is a sacred cow for many people. Even if you support sustainability, criticizing the paper will be treated as heresy by many. Despite our idealistic vision of Science(tm), it is a human process done by humans with human motivations and human weaknesses. From Galileo to today, we have repeatedly seen the enthusiastic willingness by majorities of scientists to crucify heretics (or sit by in silence) and to set aside scientific thinking and scientific process when it clashes against belief or orthodoxy or when it makes the difference whether you get tenure or publication.
• petesergeant 9mo ago
  I studied a Masters from Cambridge Judge Business School, and my takeaway is that “Management Science” is to Science what “Software Engineering” is to Engineering.
  • gyulai 9mo ago
    I expect that over the next 10 years, one of two things is going to happen: Either Software Engineering is going to reinvent itself as an actual engineering discipline, or Civil Engineering is going to cease to be one and we'll be driving over vibe-constructed bridges (and plunging to our certain deaths, in case the sarcasm wasn't clear).
• tokai 9mo ago
  Google Scholar citation numbers are unreliable and and cannot be used in bibliometric evaluation. They are auto generated and are not limited to the journal literature. This critique is completely unserious. At the same time bad papers also tend to get more citations on average than middling papers, because they are cited in critiques. This effect should be even larger in a dataset that includes more than the citations from journal papers. This blog post will in time also add to the Google Scholar citation count. Citation studies are problematic and can and their use should be criticized. But this here is just warm air build on a fundamental misunderstanding of how to measure and interpret citation data.
• jackconsidine 9mo ago
  Anyone know the VP who referenced the paper? Doesn't seem to be mentioned. My best guess is Gore. Living VPs Joe Biden — VP 2009–2017 (became President in 2021; after that he’s called a former VP and former president) Not likely the one referenced after 2017 because he became president in 2021, so later citations would likely call him a former president instead of former VP. Dan Quayle — VP 1989–1993, alive through 2026 Al Gore — VP 1993–2001, alive through 2026 Mike Pence — VP 2017–2021, alive through 2026 Kamala Harris — VP 2021–2025, alive through 2026 J.D. Vance — VP 2025–present (as of 2026)
• kittikitti 9mo ago
  The gatekeepers were able to convince the American public of such heinous things like circumcision at birth based on "science" and now they're having to deal with the corruption. People like RFK Jr. are able to be put into top positions because what they're spewing has no less scientific merit than what's accepted and recommended. The state of scientific literature is incredibly sad and mainly a factor of politics and money than of scientific evidence.
• cloche 9mo ago
  > Because published articles frequently omit key details This is a frustrating aspect of studies. You have to contact the authors for full datasets. I can see why it would not be possible to publish them in the past due to limited space in printed publications. In today's world though every paper should be required to have their full datasets published to a website for others to have access to in order to verify and replicate.
• lbcadden3 9mo ago
  >There’s a horrible sort of comfort in thinking that whatever you’ve published is already written and can’t be changed. Sometimes this is viewed as a forward-looking stance, but science that can’t be fixed isn’t past science; it’s dead science. Actually it’s not science at all.
• mwkaufma 9mo ago
  Conservatives very concerned about academic reproducability* (*except when the paper helps their agenda)
• SegfaultSeagull 9mo ago
  For all the outrage at Trump, RFK, and their Know-Nothing posture toward the world, we should recognize that the ground for their rise was fertilized by manure produced in academia.
• recursivecaveat 9mo ago
  I don't understand why it has been acceptable to not upload a tarball of your data with the paper in the internet age. Maybe the Asset4 database is only available with license and they can't publish too much. However, the key concern with the method is a pairwise matching of companies which is an invention of the paper authors and should be totally clear to publish. The number of stories I've heard from people forensically investigating PDF plots to uncover key data from a paper is absurd. Of course doing so is not free and it takes time. A paper represents at least months of work in data collection, analysis, writing, and editing though. A tarball seems like a relatively small amount of effort to provide an huge increase in confidence for the result.
  • bradley13 9mo ago
    This. I did my dissertation in the early '90s, so very early days of the internet. All of my data and code was online. IMHO this should be expected for any, literally any publication. If you have secrets, or proprietary information, fine - but then, you don't get to publish.
• deleted 9mo ago
  [deleted]
• platz 9mo ago
  What exactly is 'sustainability'
• bradley13 9mo ago
  "We should distinguish the person from the deed" No, we shouldn't. Research fraud is committed by people, who must be held accountable. In this specific case, if the issues had truly been accidental, the author's would have responded and revised their paper. They did not, ergo their false claims were likely deliberate. That the school and the journal show no interest - equally bad, and deserving of public shaming. Of course, this is also a consequence of "publish or perish."
• burgh 9mo ago
  [dead]
• slow_typist 9mo ago
  The problem is in parts, how confirmatory statistics work, and how journals work. Most journals wouldn’t publish „we really tried very hard to get significance that x causes y but found nothing. Probably, and contrary to our prior beliefs, y is completely independent of x.“ Even if nobody would cheat and massage data, we would still have studies that do not replicate on new data. 95 % confidence means that one in twenty surveys finds an effect that is only noise. The reporting of failed hypothesis testing would really help to find these cases. So pre-registration helps, and it would also help to establish the standard that everything needed to replicate must be published, if not in the article itself, then in an accompanying repository. But in the brutal fight for promotion and resources, of course labs won’t share all their tricks and process knowledge. Same problem if there is an interest in using the results commercially. E.g. in EE often the method is described in general but crucial parts of the code or circuit design are held back.
  • niccl 9mo ago
    obligatory xkcd https://xkcd.com/882/ https://xkcd.com/882/
    • slow_typist 9mo ago
      Haha yeah pretty much nails it.
• burgen 9mo ago
  The discussion has mostly revolved around the scientific system (it definitely has plenty of problems), but how about ethics? The paper in question shows - credibly or not - that companies focusing on sustainability perform better in a variety of metrics, including generating revenue. In other words: Not only can you have companies that do less harm, but these ethically superior companies also make more money. You can have your cake and eat it too. It likely has given many people a way to align their moral compass with their need to gain status and perform well within our system. Even if the paper is a completely fabrication, I'm convinced it has made the world a better a place. I can't help but wonder if Gelman and King paused to consider the possible repercussions of their actions, and of what kinds of motivations they might have had. The linked post briefly dips into ethics, benevolently proclaiming that the original authors of the paper are not necessarily bad people. Which feels ironic, as it seems to me that Gelman and King are doing the wrong here.
• tejtm 9mo ago
  It has been a viable strategy at least since Taylor 1911
• j45 9mo ago
  Creators of Studies reflect their own human flaws and shortcomings. This can directly undermine the scientific process. There has to be a better path forward.
• wtcactus 9mo ago
  There’s no such thing as management “science”. Social “sciences” are completely bastardizing the word science. Then, they come complaining that “society doesn’t trust science anymore”. They, the social “scientists”, the ones responsible for removing all meaning from the word science,
  • gyulai 9mo ago
    I came to this discussion, specifically looking for the term "management 'science'", with the quotation marks where they belong, and I found it right here, so thanks for that :-) ...I don't think I'd be capable of letting the term roll off my tongue without doing the airquotes either.
• invig 9mo ago
  We tried to scale "University". Turns out it doesn't scale well.
• thayne 9mo ago
  > and that replicators should tread very lightly That is not at all how science is supposed to work. If a result can't be replicated, it is useless. Replicators should not be told to "tread lightly", they should be encouraged. And replication papers should be published, regardless of the result (assuming they are good quality).
• wisty 9mo ago
  So 6000 people cited a paper, and either didn’t properly read it (IMO that's academic dishonesty) or weren't able to determine that the methdology was infeasible. No real surprise. I'm pretty sure most academics spend little time critically reading sources and just scan to see if it broadly supports their point (like an undergrad would). Or just cite a source if another paper says it supports a point. I've heard the most brutal thing an examiner can do in a viva vocce is to ask what a cited paper is about, lol.
• globalnode 9mo ago
  non researcher here -- how does one go about checking if a paper or article has been reproduced? just google it?
• f30e3dfed1c9 9mo ago
  Without looking, first thought was "Are the authors from Harvard Business School?" Sure enough, two out of three are. Something's gone really wrong at that place, they just keep churning out horseshit.
• spwa4 9mo ago
  The problem with psychology and the social sciences in general is that they're not neutral. The original justification for having management at all is something called "scientific management". The argument is that if you have a company, in the original meaning of a group of people working together with people individually paid per item they produce, where "new employees" (between quotes because they're not paid at that point) essentially train with more experienced employees to start producing more. The idea of management, introduced by Frederick Winslow Taylor, is to have people specifically dedicated to studying and improving the workflow of people, to become experts at making the workflow better, now known as "Taylorism". That justified middle management, in that this optimization would increase a company's productivity, and that lead to people doing what middle management still does. Before Taylorism, outside of the company owners, employees competed for wages in a competition like in "Monsters, Inc", with no-one reporting to anyone. There's a slight issue: Frederick Winslow Taylor was a con man. The experiment, introducing management, in reality lowered productivity by about 20%. It did not raise it. He kept "scientific records", measurements of productivity in a notebook and that notebook was presented to the owners of the railway. Turns out, he faked the numbers, both directly by just presenting fake numbers and by paying the company (as in the individual workers) more by faking accounts, resulting in a temporary boost in productivity. Oops. Repeated experiments showed the same. Having everyone in a company directly responsible for the functioning of the company as a whole, by being held responsible, financially, for their own work, works ... better than having management layers, according to the experiments done on the subject. You will find the social sciences defend an entirely different view. Oops. Has psychology or social sciences changed social sciences (specifically organizational psychology's) view on either Taylor or Scientific Management? No. They used it as one of the bases of the rest of psychology, of the rest of social sciences as if it was good science. This was not the first, not the last, and certainly not the most serious problem in psychology or social sciences. Some other famous problematic science. The Stanford prison experiment was faked [1]. Oops. No, that is not why people attack each other, it turns out it works far more direct. The Freudian view of psychology is not only thoroughly discredited, it is now strongly suspected that Sigmund Freud deliberately created this view to allow raping of women [2] (Freud is the person that created modern psychoanalysis, and he earned the equivalent of billions of dollars for getting rapists of the hook in court, he even had a few "successes", cases were rape victims got imprisoned, by order of a court that knew they were rape victims, in cases were the rapist was on trial. He got paid the very big Guilders for that). With that, of course, comes the reality that Freud was not an innocent scientist that came up with a wrong conclusion but a con man who caused incredible suffering for thousands of women, and hundreds of men (usually girls and boys that got raped). Autism is not an explanation of a condition of the human mind but, in the words of the creator/discoverer of Autism, Hans Asperger "serves to purify the genes of the noble Aryan race" [3] (note: yes, Autism's purpose was to purify genes by executing children). We know that being the victim of a crime raises the odds of the victim later imitating the perpetrator and committing the same crime, and to make matters worse this is a strong effect in unrelated adults, but it's stronger in adolescents, and also again stronger within families compared to between unrelated people. Note: this is not revenge, it's imitation. Victims commit the crime they were victimized by against other people, NOT the perpetrator (although if you look at it game theoretically it explains why human societies choose revenge punishments). Oops. Psychology and social sciences are not positivist sciences. The purpose is not to explain the human mind, but to justify predetermined outcomes. Especially the "discovery of Autism" illustrates this perfectly. Autism does not explain the behavior of some children, not back then, not now, and back then it justified the locking up and even executions of undesirable children, something the political climate between the two world wars really wanted to happen. Yes, you see the reverse now, but only because the political climate has changed again, not because the attitude of those sciences has changed, and you should not be surprised that if Trump stays and, say, Le Pen gets into power in France, new "psychological discoveries" will ... suddenly turn out to justify what ICE is doing, and no doubt, worse. In fact I'd argue that's exactly what's starting to happen [4]. [1] https://pubmed.ncbi.nlm.nih.gov/31380664/ https://pubmed.ncbi.nlm.nih.gov/31380664/ [2] https://en.wikipedia.org/wiki/Freudian_Coverup https://en.wikipedia.org/wiki/Freudian_Coverup (although frankly, look up what Freud's theory actually says and do you really need someone to tell you that is bullshit? Freud claims the only source of motivation for men and boys is to kill their father and rape their mother. And the only source of motivation for women and girls is to seduce men to rape them, preferably their own family. The point of Freud's theories, according to Freud's colleagues, is that it played really well in court: if a father raped his daughters or granddaughters or nieces or ... then he could not help it, it is human nature, and those daughters and nieces (and occasionally sons and nephews) really were really behind getting him to do it. Hence how does it make sense to punish him? Oh and Freud also offered services to treat/imprison those children/girls/women, of course at very high prices) [3] https://www.nature.com/articles/d41586-018-05112-1 https://www.nature.com/articles/d41586-018-05112-1 [4] https://www.nytimes.com/2026/01/24/us/children-genetics-race-science.html https://www.nytimes.com/2026/01/24/us/children-genetics-race...