16 ms·

Cracking down on research fraud

331 points by apsec112 6y ago

deleted 6y ago
[deleted]
RcouF1uZ4gsC 6y ago
I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm. In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper. Then the entirety of raw data would be sent off to a third party for data analysis and they would write the results part of the paper. The original team would then write the discussion part discussing the implications of the study. All papers would be required to be made public. The idea is that there would be specialized firms that do the analysis on the raw data for everyone. These would be carefully audited and certified by the government. They have no incentive to play statistical games. If they get caught cheating, then they have to pay for all the analysis on all the papers to be done again by a competitor and if any errors are found in a paper analysis it is automatically retracted. While the this re-analysis is going on, all papers would be quarantined with a note stating the paper is having its analysis redone. This incentivizes the analysis firm to be ethical, as well as incentivizes researchers to pick ethical analysis firms. Separating data collection from data analysis would help align incentives better.
- wjn0 6y ago
  That's an interesting idea that could stop very specific types of fraud, certainly in the life sciences. But it's not feasible for all kinds of research, and in fact could hinder lots of research. > All papers would be required to be made public. This is more universally feasible. Publicizing the data and analysis tools (scripts, software) falls into the same category, and would go a long way to help without the need for such strong separation.
- OminousWeapons 6y ago
  You would just get people massaging or generating the data to fit a conclusion prior to sending it out for statistical analysis. If people want to cheat they can find a way. The only way to really do what you are getting at is to construct core labs or CROs to run all experiments on behalf of the investigators. This is not infeasible in many cases (and is already done in narrow ways) but it requires hiring staff scientists to run every experiment rather than grad students or post docs and costs / complexity will explode. The real way to defend against a lot of fraud is to force people to submit actually detailed methods sections so experiments are legitimately reproducible (they largely aren't now). This would catch a lot more fraud quickly, although even this won't fully work as some experiments are simply too costly to reproduce for validation purposes (eg animal studies).
  - guerby 6y ago
    This. And also actually fund researchers who do reproducibility work. May be even fund specialized teams that do only reproducibility work.
woofie11 6y ago
Basic set of checks-and-balances: * Preregistration and adoption of open science practices * Public access to research results, methods, and data, with some exceptions (such as PII) * Federally-funded universities can't use NDAs or non-disparage agreements * Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs) * Federally-funded universities must adopt transparent governance * Salary caps at federally-funded universities and affiliated organizations * Conflict-of-interest laws with hard enforcement * Federally-funded universities must publicly publish research misconduct and alleged research misconduct. The latter is tricky, since you don't want to smear the researcher without proof, but you also don't want to trust results. This really needs reform.
- seibelj 6y ago
  Salary caps? So the best people go into the private sector where they can actually get paid?
  - wjn0 6y ago
    Agree, I don't think salary caps have much to do with the poor alignment of incentives under discussion here.
  - woofie11 6y ago
    Don't think $90k. Think $300k. There are plenty of top people glad to be academics for $150-$200k, and beyond that, you don't get an increase in quality. On the other hand, you avoid a lot of problems with misaligned incentives at the top.
- wjn0 6y ago
  > * Public access to research results, methods, and data, with some exceptions (such as PII) You mentioned PII, so I'm assuming some familiarity with the health field. I'm curious about your thoughts on the position that one should not be required to immediately publicize their data, because there needs to be an expectation that a researcher can translate the capital (both time and money) they expend to acquire quality data into academic and institutional capital (in the form of research output, i.e. papers). The fear being, there might be insufficient motivation to conduct large data collection-oriented studies due to another researcher beating the data collector to the punch in terms of publishing certain findings.
logicslave 6y ago
The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man for the job." "You got a PhD in ML by making some incremental improvement on some already existing model and then doing massive hyper parameter tuning? Here, become director of research at this big corporation." Research will never be fully productive in this system, there are too many people who have too much to gain from gaming the publication system.
- Something1234 6y ago
  Thanks for pointing out the problem with credentials. Name a better indicator then for being competent.
  - ktaylora 6y ago
    Industry kinda already has. Products launched. Teams managed. A candidate's publication record is a conversation starter during interviews, but I feel like years of experience at a company working on launching a successful project is a much more valuable currency.
    - saagarjha 6y ago
      For getting a job, maybe…industry isn't everything.
      - nomel 6y ago
        I think his point is that practical experience and a track record of proven, successful, outcomes are more useful for predicting future outcome.
    - throwaway2048 6y ago
      These metrics can just as easily be gamed, see how all the glory at Google goes towards launching exciting projects, that then immediately get put into stasis, then canceled.
viburnum 6y ago
A friend of mine was doing a chemistry phd when he discovered his supervisor was falsifying data. If he had blown the whistle it would have ended his career, but if he had played along his dissertation would have been based on false data. Both options were bad so he just quit. https://www.statnews.com/2016/11/25/postdocs-grad-students-fraud/ https://www.statnews.com/2016/11/25/postdocs-grad-students-f...
- kovac 6y ago
  That's some guts to do the right thing. Respect.
  - ramraj07 6y ago
    I was not faced with such a shitty situation, and I probably would have done the same as well, but the right thing would have been to report it.
    - viburnum 6y ago
      If you report it then the grant money is gone and everybody in the lab loses their job and all the PhD students are derailed.
      - chrstphrknwtn 6y ago
        That’s the fault of the person falsifying data, not the person reporting the fraud.
        jjoonathan 6y ago
        Yeah but it still happens, and to people you care about who didn't deserve it.
        henriquemaia 6y ago
        That's true. Several people will probably lose their jobs. But that is the kind of thinking that condemns whistleblowing for the wrong reasons. To put it differently, people rather prefer to keep on pretending if that keeps their jobs than to do the right thing. And that makes everyone a fraudster.
cblconfederate 6y ago
Equalize the field, radically remove research hierarchies. Liberate research funding from pointless bureaucracies and 4-year grants and instead pay scientists to work on whatever they like most. Let science be fun again I always return to this article from David Hubel (of Hubel&Wiesel): https://www.cell.com/neuron/fulltext/S0896-6273(09)00733-8 https://www.cell.com/neuron/fulltext/S0896-6273(09)00733-8 I arrived at his lab around noon and found him working alone, recording from an anesthetized macaque monkey. I asked him when he had started the experiment, and he answered “in the morning”, which I finally realized was the morning of the day before. So he had worked, by himself, all the day before, all that night, and that day until noon. What was typical, in that era, was not only the long hours but the fact that the project was done by one person, single handedly. The major papers were either by Mountcastle alone or in partnership with one other person. The leader of the physiology department was Philip Bard, but the idea that Bard should have asked to have his name on any of Vernon's papers surely never occurred to anyone.
- rumanator 6y ago
  > Liberate research funding from pointless bureaucracies and 4-year grants and instead pay scientists to work on whatever they like most. Bureaucracy exists to ensure that funds are indeed spent on research instead of jet skis and shiny toys, and hierarchies are established to focus work on specialized topics pertaining to the state of the art of a field of research. All your suggestions do is ensure that less funds are spent on research, and the few work that is performed cannot be any form of deep dive or continuous effort on a topic, thus resulting only in low-hanging fruits. And by the way, the minute a researcher complains he needs help with his research is the moment you get a hierarchy.
  - cblconfederate 6y ago
    bureaucracy cannot guarantee the quality of research beyond the jet ski bar hierarchies are established to ensure that the PI gets citations from the work of postdocs and postgrads typical grants encourage shallow trend-chasing or incremental continuation of the research of - often disinterested - senior PIs who simply get grants because they have gotten a lot of grants. The chase of the state of the art is just another incremental step - that 's shallow a research partner is not necessarily a subordinate
DanielleMolloy 6y ago
I've read that before 2003, the whole of humanity has published as many scientific papers as from 2003 to 2016. So what happened around 2000? Who has turned the scientific mission into a blind competition for superficial metrics? So many people in science I meet (apart from the few who benefited from this system, and therefore were selected by it) are frustrated by publishing for the sake of publishing (not science) and the bad incentives this system creates. Who has thought that these superficial metrics would improve anything about science and why?
- sgillen 6y ago
  I think it’s mostly a numbers game, more PhDs, roughly the same number of research position. This means most scientists don’t know most other scientists in their field, and so the metrics people use to select become more important. That and the increased competition, means a bigger rat race, more pressure to publish, etc.
- lrnStats 6y ago
  My guess is the education bubble. Loans are subsidized, universities have bigger budgets and want more studies done.
- nitrogen 6y ago
  With any exponential growth curve, you can point to some semi-recent point on the curve and say 50% is on the right side of that point. If the population is growing exponentially (it is), and the percentage of the population in academia is consistent or growing (that I'm not sure), then you could reasonably expect the quantity of research to be growing exponentially as well. Maybe it's just a curve with a doubling period of 13 years.
- amcoastal 6y ago
  I imagine the internet has done a lot to increase the speed of research, communication, and writing. I imagine it also speeds up peer review and has expanded the amount of available places to publish your work. I agree though, it would be nice to spend more time sciencing and less time writing about it.
- 6y ago
tlb 6y ago
Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much. Possible arguments include: - Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, at least.) - The 1% is wrong in a more harmful way than the 80%. Perhaps falsifying data is worse than hand-waving conclusions. So if the maximum upside is 1% of wrong research removed, and the downside is quenching some fraction of the good 19%, it's probably better to leave it alone.
- deleted 6y ago
  [deleted]
- ameliaquining 6y ago
  The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.
  - opportune 6y ago
    I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.
- ketzo 6y ago
  I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.
  - anonthrowaway3 6y ago
    Why is that critical to maintain? No one in the public should trust academic science given what a complete cesspool academia is like.
  - galimaufry 6y ago
jcahill 6y ago
Certain groups have been serious the replication crisis for 10-15y, but academic culture at large is simply not cut out to discuss fraud in a 'street epistemology' sort of way, such as the way security researchers might discuss cybercrime. There's a wild amount of pushback to any amount of meta-criticism. But once you get past that point, many roadblocks remain. In particular, there's extreme bias for meta-statistical methodologies that infer QRPs over other methods of investigation. Often, these methods aren't strictly necessary in context, and afford the opportunity to turn the metasci discourse into endless bikeshedding about the meta-framework rather than the object of dispute. Many interested parties will participate/perform in this discourse, but few will sit down and really look at things as simple as the logical structure of the paper's claims, or even simpler problems with its content. In the case of social psych, for instance, many problems lie with (a) stimuli and (b) unexamined assumptions on the part of the researchers that a certain manipulation holds, so these are important to assess. But extending critique to these things is seen as "reviewer 2" behavior — uncollegial, unfair, sniping, waaah etc. Since "researcher x produces bad research, but evidence of fraud is only circumstantial" isn't sufficient grounds for doing much of anything, little comes of these efforts. When an investigation does nail a fraudster to the wall, well, so? The papers remain, the poisoned citation tree remains, the culture remains. More than anything, the pushback in the form of tone-policing is what gets me. "Methodological terrorism" this, "reviewer 2" that. It shows how far from consequences the gatekeepers are. If you're a young person, whole branches of the academy have been pre-bankrupted for you. There's no hope there, short of a research path that manages to avoid citing any prior literature. But don't get tetchy with the grantlords! Meanwhile, all around the US, real effects of this dogshit excuse for scientific inquiry can be seen every single day. Police departments are adopting new policies around known-bad implicit bias papers. These won't work. We know they won't work. We've known this for years. What would make it stop? Every time cops kill an unarmed person, academics who still haven't retracted their implicit bias papers get fined? There's the rub. You can't do much to force the point. Effective, timely measures would be fairly brutal ones, and academics aren't ready to admit this to themselves. In some ways, COVID-19 may end up being one of those measures, though it will disproportionately affect younger researchers.
OminousWeapons 6y ago
I'd argue that a lot of what the authors describe isn't actually "fraud", it's more exploitation of an intentionally broken system. I've definitely had authors on my papers who didn't do work. I've definitely written papers for people who didn't do work. I've definitely done peer reviews on behalf of PIs. Why do people do this? Because the regulators allow it and they want the system that way. Why should who wrote the paper have any impact on review? Why should it matter who the journal editors are? Why should it matter where the paper is from? Etc...
- MattGaiser 6y ago
  Because we use heuristics instead of actual analysis. How many people citing a paper or reading it actually have time to think that deeply about it? The alternative to a world of trust and heuristics is a world where we are all bogged down trying to make decisions. This is heavily demonstrated in recruiting. Resume reading is about 7 seconds a person. How long would it take if they spent a minute for each?
lettergram 6y ago
I think there is outright fraud and subtle fraud. For instance, most mouse studies specifically as it relates to aging, drug safety, and cancer research should be thrown out. This is well covered here: https://m.youtube.com/watch?v=pRCzZp1J0v0 https://m.youtube.com/watch?v=pRCzZp1J0v0 That’s not the only issue, but it’s a known issue, everyone is ignoring as it relates to all studies with the most common mice
- Gatsky 6y ago
  This Weinstein mouse model thing is blown completely out of proportion, and is only so prominent because his own brother Eric put it on The Portal. The point is that everyone knows mouse models are imperfect, but they are still useful. Pointing out one of the ways they are imperfect is fine. Pretending this is a nobel prize winning earth shattering discovery that is suppressed by the establishment is just annoying. Research about how mouse models are imperfect comes out all the time, eg there was a paper about how the relatively sterile lab environment changes the murine microbiome and immune system. Why aren't these researchers on Joe Rogan's podcast?
  - aaron695 6y ago
    He specifically mentions comments like yours are not helpful. This is the researcher fraud that is the problem in science that we are talking about, issues at hand are ignored or twisted so the paper mill can continue. The concept is all mice in the USA come from one laboratory and have a common attribute that's different from wild mice and European suppliers. And it's spelt out how in theory this different attribute could change medical results. To come back with we don't care, without proof it's not a issue, is a deeply broken system, but is exactly how current universities work. Here's the actual segment (18 mins) - https://www.youtube.com/watch?v=ve4q-1D_Ajo https://www.youtube.com/watch?v=ve4q-1D_Ajo Related (12 mins) - https://www.youtube.com/watch?v=8ygLNOt43So https://www.youtube.com/watch?v=8ygLNOt43So
    - Gatsky 6y ago
      I've heard everything he said. I'm well aware of the concept. Again, everyone knows that all mice come from the same place (because it's written in the papers), and that they are very different to wild mice or mice from elsewhere. I also didn't say we don't have proof that it is a problem. Again, all models are imperfect, but some are useful. I am telling you what I see as the significance of Weinstein's findings as someone that works in the field. It is nonsesnical to me that this particular concept is suddenly so prominent because of Joe Rogan, whereas a multitude of other issues with research and human translation are not talked about at all. To me what this is really about is Weinstein's perception of the significance of his findings. He, like a lot of scientists, thinks his work is hugely important and he deserves more credit. Others do not. This happens all the time. The only difference is that he has found a way to disseminate his findings to a non-specialist audience.
IceCreamJonsey 6y ago
Coincidence for me to see this here, as a book by Dr. Stuart Ritchie was just published about this very thing. https://www.amazon.com/Science-Fictions-Negligence-Undermine-Search/dp/1250222699 https://www.amazon.com/Science-Fictions-Negligence-Undermine...
lrnStats 6y ago
Similar to research fraud, I want the medical field examined for anti-science fraud. I caught my Physician recommending an expensive and dangerous surgery that could be done by a dentist or surgeon. I asked if there was data, she said yes. There was no data. And the trend was using lasers rather than surgery since it's safer. I confronted her and she said- "If you ask a physician, they will recommend a physician. If you ask a dentist, they will recommend a dentist." This physician used factionalism rather than science. I imagine this has happened on a massive scale.
- bmunoz11 6y ago
  ha
Gatsky 6y ago
Research fraud gets a lot of attention because it is so black and white. But it is a symptom of larger problems. One issue is that the pace of progress is slowing, and as a result incremental gains are more prominent. This is fertile ground for fraudsters, as they can produce results which are plausible enough while seeming to be an important contribution to the field. All fraud fits into this category. Nobody makes a new grand unified theory of everything which they know is bogus. That would be too much work for a start. The other issue is the huge expansion in university size. Most of the fraud I've seen or heard about all happens in university research departments. This shows you the importance of their incentive structure. One can make things up and not only succeed, but do better than your competitors in this research setting, AND get a tenured position with life-long security. All competitive fields where achievement recieves external and highly persistent rewards suffer from this problem, whether it be sport and performance enhancing drugs or Ivy league univesrity admissions or even venture captial funding (Theranos). The natural response is to ask for more regulation and structural change in how research is conducted eg pre-registration, different statistical standards etc. But this has the major disadvantage of making life harder for the honest people. It also requires the creation of some parallel work force to handle all the checking. Research is already so difficult. Paradoxical effects, where such measures increase fraud, are definitely possible. There will never be zero fraud. The aim should be to change how research is done to make the experience more humane, train and mentor young scientists carefully and avoid perverse incentives. As far as I can tell, nobody has any idea how to do this. Instead they want to create investigatory bodies which will siphon off money that could be used for research, and then ruin lives pursuing some key performance indicator like N successful fraud cases per quarter. This experiment was already run in the USA with the Office of Research Integrity, and it failed. Malcolm Gladwell, who I am not a fan of in general, has a good podcast about it [1]. [1] http://revisionisthistory.com/episodes/28-the-imaginary-crimes-of-margit-hamosh http://revisionisthistory.com/episodes/28-the-imaginary-crim...
DrNuke 6y ago
Bluntly put, peer review is an overwhelming, unpaid, unsatisfying, time-expensive task. The only step forward is to force the release of both data and implementation, at least for the highest ranked journals, this possibly opening more unpredictable cans of worms. It is the elitist academy model on one side, not working any more, against the democratisation of research, still and just a torrential flow of noise.
bmunoz11 6y ago
hi i am new form this app or sum but can someone tell me about it come on.-_-
deleted 6y ago
[deleted]
gwerbret 6y ago
These issues of research fraud come up often, but the root of the problem is a bit more subtle. In North America at least, biomedical research labs operate largely as fiefdoms of the individual principal investigators (PIs). The actual research work falls almost entirely on the backs of grad students and postdoctoral fellows. The grad students need to generate "good" data in order to graduate, the postdocs need the same in order to gain real employment (with only about 10% gaining faculty positions themselves after many years of postdoctoral training). The PIs need such "productivity" from their trainees in order to gain the funding that keeps the labs going. The PIs themselves face success rates in grant applications that are often 10% or lower and, particularly early in their careers, their job security depends almost entirely on their ability to secure grant funding. These competitive pressures create enormous incentives for otherwise conscientious people, all the way along the hierarchy described above, to fudge their research data. Research fraud is thus a direct outcome of a fundamentally-broken approach to the structure of research funding. There are exceptions to that approach, however. The not-for-profit Howard Hughes Research Institute [1], and to an extent the intramural research programs of the NIH [2], offer funding for PIs to do what they do best, without the pressure of competing for scant funds. Coincidentally, some of the best science comes out of these sites. 1: https://www.hhmi.org/scientists https://www.hhmi.org/scientists 2: https://irp.nih.gov/about-us/what-is-the-irp https://irp.nih.gov/about-us/what-is-the-irp
- martincmartin 6y ago
  What do you suggest as an alternative?
  - cscurmudgeon 6y ago
    Research publication, funding, and jobs should not be tied to outcomes of research. They should be tied to how systematic, logical, and well-documented the research is. We need a system wide change from funding bodies, job committees to publishing criteria.
    - mlthoughts2018 6y ago
owenshen24 6y ago
Interesting notes from the paper mentioned in the article: https://journals.sagepub.com/doi/pdf/10.1177/1747016119898400 https://journals.sagepub.com/doi/pdf/10.1177/174701611989840... - "only 39 scientists from 7 countries have been subject to criminal sanctions between 1979 and 2015 (Oransky and Abritis, 2017)" That seems...very low. - "The Retraction Watch database—the largest of its kind—currently includes more than 18,500 retracted articles (Retraction Watch database, 2019). A recent analysis of 10,500 retracted papers up to 2016 showed that 0.04% of papers are retracted." This is once again a lower-bound; presumably if you account for additional authors and p-hacking the numbers go up a lot. Pushing for replication and improved methodology can help, but some of these issues seem to be related to scale. There are many more people outputting papers than there are people willing to vet them (outside of peer review). Furthermore, when you have many people researching hot fields, you should expect false positives and overestimates to dominate published results, even when everyone is trying to practice good statistical hygiene. (https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...)
grandridge 6y ago
But I thought we should "believe all scientists?!"
deleted 6y ago
[deleted]
jostmey 6y ago
I’ve seen lots of “questionable” research, so much, that I rarely read papers published by other researchers because chances are the data is junk, and I’m in academic biomedical research!!
deleted 6y ago
[deleted]
BurningFrog 6y ago
Starting to think it's time to close down the university system and start a new one...
- pas 6y ago
  Sure, but how would you organize the new one?
  - BurningFrog 6y ago
    I'd just hope new alternatives would emerge as the old system gets out of the way. Then we'll see what develops. Not a selling argument, I know :)
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
DoreenMichele 6y ago
So, some years ago, Temple Grandin wrote a set of standards and McDonald's adopted it and they buy so much beef that it became the de facto new standard for the beef industry. And it's a set of standards that helps beef producers succeed rather than a "gotcha" trying to find who is guilty. And that's the way you make the world a better place. Not by looking for new and creative ways to nail "bad guys" to the wall after you started from an assumption of guilt. I don't like this article. I don't like it at all. My feeling is that it was written as an emotional response to the pandemic and it is getting traction on HN for the exact same reason. People are stressed out and they are looking for a villain to go after. It won't fix the real problem -- the pandemic -- but that's how people tend to behave in a crisis. And it's a slippery slope towards a more draconian world. It doesn't make things better.
- _Microft 6y ago
  The McDonald's story reminds me of the Brussels effect [0] where legislation in the EU is extending (not by law but in its effect) to other parts of the world because it is easier or cheaper to comply with it for all customers than to treat EU costumers and others differently. [0] https://en.wikipedia.org/wiki/Brussels_effect https://en.wikipedia.org/wiki/Brussels_effect
  - deleted 6y ago
    [deleted]
  - haihaibye 6y ago
    So that's why I have to accept cookies all the time.
    - _Microft 6y ago
      No, this is either because they have no idea what they are doing, or because they like bossing people around in return for not getting their way regarding user data while pretending this behaviour is because of consumer-unfriendly EU regulations. It's perfectly fine to set cookies that are necessary for providing the service without having to ask the user. They only need to get one's permission if they want to do other things like selling data to advertisers.
hlfy_hn 6y ago
Makes me think of Hans Lehrach. Dunno why https://www.molgen.mpg.de/hans-lehrach https://www.molgen.mpg.de/hans-lehrach
cryptica 6y ago
The monetary system is founded on fraud. So it's not surprising to see that the system as a whole is tolerant of all other kinds of fraud. Those who have the power to fix it have no moral grounds to do so without turning into hypocrites. Nobody wants to be a hypocrite. At least if you keep your mouth shut, you can feel better about yourself; you're just doing what everyone else is doing.
deleted 6y ago
[deleted]
brownbat 6y ago
You can't really improve this until the lead author doesn't have a final say on how to treat outliers. In any given study, there are going to be hundreds of special cases in the data that you didn't anticipate, and you have to decide whether to include or exclude them. Any researcher will subconsciously be more sympathetic to arguments to exclude subjects that go against the principal theory, and less so to subjects that confirm it. And it's a battle of reasonable arguments, most of the cases aren't bright line fraud or misconduct, they're just humans finding some arguments more compelling and the impossibility of escaping our own biases. (And yes, sometimes it's fraud, but fraud is just the tip of the iceberg if we're talking about genuinely improving the reliability of scientific findings.) Prepublication is helpful, but more and more I'm convinced that the only way to do proper science would be to completely disaggregate study design from study execution.
AtOmXpLuS 6y ago
Everything in [AtOmXpLuS](https://www.atomxplus.com https://www.atomxplus.com)

331 points by apsec112 6y ago

deleted 6y ago

[deleted]

I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm. In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper. Then the entirety of raw data would be sent off to a third party for data analysis and they would write the results part of the paper. The original team would then write the discussion part discussing the implications of the study. All papers would be required to be made public. The idea is that there would be specialized firms that do the analysis on the raw data for everyone. These would be carefully audited and certified by the government. They have no incentive to play statistical games. If they get caught cheating, then they have to pay for all the analysis on all the papers to be done again by a competitor and if any errors are found in a paper analysis it is automatically retracted. While the this re-analysis is going on, all papers would be quarantined with a note stating the paper is having its analysis redone. This incentivizes the analysis firm to be ethical, as well as incentivizes researchers to pick ethical analysis firms. Separating data collection from data analysis would help align incentives better.

wjn0 6y ago
That's an interesting idea that could stop very specific types of fraud, certainly in the life sciences. But it's not feasible for all kinds of research, and in fact could hinder lots of research. > All papers would be required to be made public. This is more universally feasible. Publicizing the data and analysis tools (scripts, software) falls into the same category, and would go a long way to help without the need for such strong separation.
OminousWeapons 6y ago
You would just get people massaging or generating the data to fit a conclusion prior to sending it out for statistical analysis. If people want to cheat they can find a way. The only way to really do what you are getting at is to construct core labs or CROs to run all experiments on behalf of the investigators. This is not infeasible in many cases (and is already done in narrow ways) but it requires hiring staff scientists to run every experiment rather than grad students or post docs and costs / complexity will explode. The real way to defend against a lot of fraud is to force people to submit actually detailed methods sections so experiments are legitimately reproducible (they largely aren't now). This would catch a lot more fraud quickly, although even this won't fully work as some experiments are simply too costly to reproduce for validation purposes (eg animal studies).
- guerby 6y ago
  This. And also actually fund researchers who do reproducibility work. May be even fund specialized teams that do only reproducibility work.

woofie11 6y ago

Basic set of checks-and-balances: * Preregistration and adoption of open science practices * Public access to research results, methods, and data, with some exceptions (such as PII) * Federally-funded universities can't use NDAs or non-disparage agreements * Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs) * Federally-funded universities must adopt transparent governance * Salary caps at federally-funded universities and affiliated organizations * Conflict-of-interest laws with hard enforcement * Federally-funded universities must publicly publish research misconduct and alleged research misconduct. The latter is tricky, since you don't want to smear the researcher without proof, but you also don't want to trust results. This really needs reform.

seibelj 6y ago
Salary caps? So the best people go into the private sector where they can actually get paid?
- wjn0 6y ago
  Agree, I don't think salary caps have much to do with the poor alignment of incentives under discussion here.
- woofie11 6y ago
  Don't think $90k. Think $300k. There are plenty of top people glad to be academics for $150-$200k, and beyond that, you don't get an increase in quality. On the other hand, you avoid a lot of problems with misaligned incentives at the top.
wjn0 6y ago
> * Public access to research results, methods, and data, with some exceptions (such as PII) You mentioned PII, so I'm assuming some familiarity with the health field. I'm curious about your thoughts on the position that one should not be required to immediately publicize their data, because there needs to be an expectation that a researcher can translate the capital (both time and money) they expend to acquire quality data into academic and institutional capital (in the form of research output, i.e. papers). The fear being, there might be insufficient motivation to conduct large data collection-oriented studies due to another researcher beating the data collector to the punch in terms of publishing certain findings.

logicslave 6y ago

The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man for the job." "You got a PhD in ML by making some incremental improvement on some already existing model and then doing massive hyper parameter tuning? Here, become director of research at this big corporation." Research will never be fully productive in this system, there are too many people who have too much to gain from gaming the publication system.

Something1234 6y ago
Thanks for pointing out the problem with credentials. Name a better indicator then for being competent.
- ktaylora 6y ago
  Industry kinda already has. Products launched. Teams managed. A candidate's publication record is a conversation starter during interviews, but I feel like years of experience at a company working on launching a successful project is a much more valuable currency.
  - saagarjha 6y ago
    For getting a job, maybe…industry isn't everything.
    - nomel 6y ago
      I think his point is that practical experience and a track record of proven, successful, outcomes are more useful for predicting future outcome.
  - throwaway2048 6y ago
    These metrics can just as easily be gamed, see how all the glory at Google goes towards launching exciting projects, that then immediately get put into stasis, then canceled.

viburnum 6y ago

A friend of mine was doing a chemistry phd when he discovered his supervisor was falsifying data. If he had blown the whistle it would have ended his career, but if he had played along his dissertation would have been based on false data. Both options were bad so he just quit. https://www.statnews.com/2016/11/25/postdocs-grad-students-fraud/ https://www.statnews.com/2016/11/25/postdocs-grad-students-f...

kovac 6y ago
That's some guts to do the right thing. Respect.
- ramraj07 6y ago
  I was not faced with such a shitty situation, and I probably would have done the same as well, but the right thing would have been to report it.
  - viburnum 6y ago
    If you report it then the grant money is gone and everybody in the lab loses their job and all the PhD students are derailed.
    - chrstphrknwtn 6y ago
      That’s the fault of the person falsifying data, not the person reporting the fraud.
      - jjoonathan 6y ago
        Yeah but it still happens, and to people you care about who didn't deserve it.
        henriquemaia 6y ago
        That's true. Several people will probably lose their jobs. But that is the kind of thinking that condemns whistleblowing for the wrong reasons. To put it differently, people rather prefer to keep on pretending if that keeps their jobs than to do the right thing. And that makes everyone a fraudster.

cblconfederate 6y ago

Equalize the field, radically remove research hierarchies. Liberate research funding from pointless bureaucracies and 4-year grants and instead pay scientists to work on whatever they like most. Let science be fun again I always return to this article from David Hubel (of Hubel&Wiesel): https://www.cell.com/neuron/fulltext/S0896-6273(09)00733-8 https://www.cell.com/neuron/fulltext/S0896-6273(09)00733-8 I arrived at his lab around noon and found him working alone, recording from an anesthetized macaque monkey. I asked him when he had started the experiment, and he answered “in the morning”, which I finally realized was the morning of the day before. So he had worked, by himself, all the day before, all that night, and that day until noon. What was typical, in that era, was not only the long hours but the fact that the project was done by one person, single handedly. The major papers were either by Mountcastle alone or in partnership with one other person. The leader of the physiology department was Philip Bard, but the idea that Bard should have asked to have his name on any of Vernon's papers surely never occurred to anyone.

rumanator 6y ago
> Liberate research funding from pointless bureaucracies and 4-year grants and instead pay scientists to work on whatever they like most. Bureaucracy exists to ensure that funds are indeed spent on research instead of jet skis and shiny toys, and hierarchies are established to focus work on specialized topics pertaining to the state of the art of a field of research. All your suggestions do is ensure that less funds are spent on research, and the few work that is performed cannot be any form of deep dive or continuous effort on a topic, thus resulting only in low-hanging fruits. And by the way, the minute a researcher complains he needs help with his research is the moment you get a hierarchy.
- cblconfederate 6y ago
  bureaucracy cannot guarantee the quality of research beyond the jet ski bar hierarchies are established to ensure that the PI gets citations from the work of postdocs and postgrads typical grants encourage shallow trend-chasing or incremental continuation of the research of - often disinterested - senior PIs who simply get grants because they have gotten a lot of grants. The chase of the state of the art is just another incremental step - that 's shallow a research partner is not necessarily a subordinate

DanielleMolloy 6y ago

I've read that before 2003, the whole of humanity has published as many scientific papers as from 2003 to 2016. So what happened around 2000? Who has turned the scientific mission into a blind competition for superficial metrics? So many people in science I meet (apart from the few who benefited from this system, and therefore were selected by it) are frustrated by publishing for the sake of publishing (not science) and the bad incentives this system creates. Who has thought that these superficial metrics would improve anything about science and why?

sgillen 6y ago
I think it’s mostly a numbers game, more PhDs, roughly the same number of research position. This means most scientists don’t know most other scientists in their field, and so the metrics people use to select become more important. That and the increased competition, means a bigger rat race, more pressure to publish, etc.
lrnStats 6y ago
My guess is the education bubble. Loans are subsidized, universities have bigger budgets and want more studies done.
nitrogen 6y ago
With any exponential growth curve, you can point to some semi-recent point on the curve and say 50% is on the right side of that point. If the population is growing exponentially (it is), and the percentage of the population in academia is consistent or growing (that I'm not sure), then you could reasonably expect the quantity of research to be growing exponentially as well. Maybe it's just a curve with a doubling period of 13 years.
amcoastal 6y ago
I imagine the internet has done a lot to increase the speed of research, communication, and writing. I imagine it also speeds up peer review and has expanded the amount of available places to publish your work. I agree though, it would be nice to spend more time sciencing and less time writing about it.
6y ago

tlb 6y ago

Estimating from my own reading, 1% of research is fraud and 80% is worthless for other reasons. Numbers vary between fields. If that's the case, what's the argument for why we should spend time doing something about the 1%? Solving 100% of the 1% wouldn't change the overall situation much. Possible arguments include: - Fixing the 80% is hard, but fixing the 1% is satisfying (to the aggressively conventional-minded, at least.) - The 1% is wrong in a more harmful way than the 80%. Perhaps falsifying data is worse than hand-waving conclusions. So if the maximum upside is 1% of wrong research removed, and the downside is quenching some fraction of the good 19%, it's probably better to leave it alone.

deleted 6y ago
[deleted]
ameliaquining 6y ago
The author is arguing for an expansive definition of fraud that includes things like p-hacking. Probably a lot more than 1% of research is p-hacked; that general category of problem might even account for a majority of your 80%.
- opportune 6y ago
  I'm guessing throwing out "bad data" happens in a lot more than 1% of cases as well.
ketzo 6y ago
I think the disproportionate harm argument is a big deal. A single high-profile case of fraud undermines trust in the entire field, and that’s critical to maintain.
- anonthrowaway3 6y ago
  Why is that critical to maintain? No one in the public should trust academic science given what a complete cesspool academia is like.
- galimaufry 6y ago

jcahill 6y ago

Certain groups have been serious the replication crisis for 10-15y, but academic culture at large is simply not cut out to discuss fraud in a 'street epistemology' sort of way, such as the way security researchers might discuss cybercrime. There's a wild amount of pushback to any amount of meta-criticism. But once you get past that point, many roadblocks remain. In particular, there's extreme bias for meta-statistical methodologies that infer QRPs over other methods of investigation. Often, these methods aren't strictly necessary in context, and afford the opportunity to turn the metasci discourse into endless bikeshedding about the meta-framework rather than the object of dispute. Many interested parties will participate/perform in this discourse, but few will sit down and really look at things as simple as the logical structure of the paper's claims, or even simpler problems with its content. In the case of social psych, for instance, many problems lie with (a) stimuli and (b) unexamined assumptions on the part of the researchers that a certain manipulation holds, so these are important to assess. But extending critique to these things is seen as "reviewer 2" behavior — uncollegial, unfair, sniping, waaah etc. Since "researcher x produces bad research, but evidence of fraud is only circumstantial" isn't sufficient grounds for doing much of anything, little comes of these efforts. When an investigation does nail a fraudster to the wall, well, so? The papers remain, the poisoned citation tree remains, the culture remains. More than anything, the pushback in the form of tone-policing is what gets me. "Methodological terrorism" this, "reviewer 2" that. It shows how far from consequences the gatekeepers are. If you're a young person, whole branches of the academy have been pre-bankrupted for you. There's no hope there, short of a research path that manages to avoid citing any prior literature. But don't get tetchy with the grantlords! Meanwhile, all around the US, real effects of this dogshit excuse for scientific inquiry can be seen every single day. Police departments are adopting new policies around known-bad implicit bias papers. These won't work. We know they won't work. We've known this for years. What would make it stop? Every time cops kill an unarmed person, academics who still haven't retracted their implicit bias papers get fined? There's the rub. You can't do much to force the point. Effective, timely measures would be fairly brutal ones, and academics aren't ready to admit this to themselves. In some ways, COVID-19 may end up being one of those measures, though it will disproportionately affect younger researchers.

OminousWeapons 6y ago

I'd argue that a lot of what the authors describe isn't actually "fraud", it's more exploitation of an intentionally broken system. I've definitely had authors on my papers who didn't do work. I've definitely written papers for people who didn't do work. I've definitely done peer reviews on behalf of PIs. Why do people do this? Because the regulators allow it and they want the system that way. Why should who wrote the paper have any impact on review? Why should it matter who the journal editors are? Why should it matter where the paper is from? Etc...

MattGaiser 6y ago
Because we use heuristics instead of actual analysis. How many people citing a paper or reading it actually have time to think that deeply about it? The alternative to a world of trust and heuristics is a world where we are all bogged down trying to make decisions. This is heavily demonstrated in recruiting. Resume reading is about 7 seconds a person. How long would it take if they spent a minute for each?

lettergram 6y ago

I think there is outright fraud and subtle fraud. For instance, most mouse studies specifically as it relates to aging, drug safety, and cancer research should be thrown out. This is well covered here: https://m.youtube.com/watch?v=pRCzZp1J0v0 https://m.youtube.com/watch?v=pRCzZp1J0v0 That’s not the only issue, but it’s a known issue, everyone is ignoring as it relates to all studies with the most common mice

Gatsky 6y ago
This Weinstein mouse model thing is blown completely out of proportion, and is only so prominent because his own brother Eric put it on The Portal. The point is that everyone knows mouse models are imperfect, but they are still useful. Pointing out one of the ways they are imperfect is fine. Pretending this is a nobel prize winning earth shattering discovery that is suppressed by the establishment is just annoying. Research about how mouse models are imperfect comes out all the time, eg there was a paper about how the relatively sterile lab environment changes the murine microbiome and immune system. Why aren't these researchers on Joe Rogan's podcast?
- aaron695 6y ago
  He specifically mentions comments like yours are not helpful. This is the researcher fraud that is the problem in science that we are talking about, issues at hand are ignored or twisted so the paper mill can continue. The concept is all mice in the USA come from one laboratory and have a common attribute that's different from wild mice and European suppliers. And it's spelt out how in theory this different attribute could change medical results. To come back with we don't care, without proof it's not a issue, is a deeply broken system, but is exactly how current universities work. Here's the actual segment (18 mins) - https://www.youtube.com/watch?v=ve4q-1D_Ajo https://www.youtube.com/watch?v=ve4q-1D_Ajo Related (12 mins) - https://www.youtube.com/watch?v=8ygLNOt43So https://www.youtube.com/watch?v=8ygLNOt43So
  - Gatsky 6y ago
    I've heard everything he said. I'm well aware of the concept. Again, everyone knows that all mice come from the same place (because it's written in the papers), and that they are very different to wild mice or mice from elsewhere. I also didn't say we don't have proof that it is a problem. Again, all models are imperfect, but some are useful. I am telling you what I see as the significance of Weinstein's findings as someone that works in the field. It is nonsesnical to me that this particular concept is suddenly so prominent because of Joe Rogan, whereas a multitude of other issues with research and human translation are not talked about at all. To me what this is really about is Weinstein's perception of the significance of his findings. He, like a lot of scientists, thinks his work is hugely important and he deserves more credit. Others do not. This happens all the time. The only difference is that he has found a way to disseminate his findings to a non-specialist audience.

IceCreamJonsey 6y ago

Coincidence for me to see this here, as a book by Dr. Stuart Ritchie was just published about this very thing. https://www.amazon.com/Science-Fictions-Negligence-Undermine-Search/dp/1250222699 https://www.amazon.com/Science-Fictions-Negligence-Undermine...

lrnStats 6y ago

Similar to research fraud, I want the medical field examined for anti-science fraud. I caught my Physician recommending an expensive and dangerous surgery that could be done by a dentist or surgeon. I asked if there was data, she said yes. There was no data. And the trend was using lasers rather than surgery since it's safer. I confronted her and she said- "If you ask a physician, they will recommend a physician. If you ask a dentist, they will recommend a dentist." This physician used factionalism rather than science. I imagine this has happened on a massive scale.

bmunoz11 6y ago
ha

Gatsky 6y ago

Research fraud gets a lot of attention because it is so black and white. But it is a symptom of larger problems. One issue is that the pace of progress is slowing, and as a result incremental gains are more prominent. This is fertile ground for fraudsters, as they can produce results which are plausible enough while seeming to be an important contribution to the field. All fraud fits into this category. Nobody makes a new grand unified theory of everything which they know is bogus. That would be too much work for a start. The other issue is the huge expansion in university size. Most of the fraud I've seen or heard about all happens in university research departments. This shows you the importance of their incentive structure. One can make things up and not only succeed, but do better than your competitors in this research setting, AND get a tenured position with life-long security. All competitive fields where achievement recieves external and highly persistent rewards suffer from this problem, whether it be sport and performance enhancing drugs or Ivy league univesrity admissions or even venture captial funding (Theranos). The natural response is to ask for more regulation and structural change in how research is conducted eg pre-registration, different statistical standards etc. But this has the major disadvantage of making life harder for the honest people. It also requires the creation of some parallel work force to handle all the checking. Research is already so difficult. Paradoxical effects, where such measures increase fraud, are definitely possible. There will never be zero fraud. The aim should be to change how research is done to make the experience more humane, train and mentor young scientists carefully and avoid perverse incentives. As far as I can tell, nobody has any idea how to do this. Instead they want to create investigatory bodies which will siphon off money that could be used for research, and then ruin lives pursuing some key performance indicator like N successful fraud cases per quarter. This experiment was already run in the USA with the Office of Research Integrity, and it failed. Malcolm Gladwell, who I am not a fan of in general, has a good podcast about it [1]. [1] http://revisionisthistory.com/episodes/28-the-imaginary-crimes-of-margit-hamosh http://revisionisthistory.com/episodes/28-the-imaginary-crim...

DrNuke 6y ago

Bluntly put, peer review is an overwhelming, unpaid, unsatisfying, time-expensive task. The only step forward is to force the release of both data and implementation, at least for the highest ranked journals, this possibly opening more unpredictable cans of worms. It is the elitist academy model on one side, not working any more, against the democratisation of research, still and just a torrential flow of noise.

bmunoz11 6y ago

hi i am new form this app or sum but can someone tell me about it come on.-_-

deleted 6y ago

[deleted]

gwerbret 6y ago

These issues of research fraud come up often, but the root of the problem is a bit more subtle. In North America at least, biomedical research labs operate largely as fiefdoms of the individual principal investigators (PIs). The actual research work falls almost entirely on the backs of grad students and postdoctoral fellows. The grad students need to generate "good" data in order to graduate, the postdocs need the same in order to gain real employment (with only about 10% gaining faculty positions themselves after many years of postdoctoral training). The PIs need such "productivity" from their trainees in order to gain the funding that keeps the labs going. The PIs themselves face success rates in grant applications that are often 10% or lower and, particularly early in their careers, their job security depends almost entirely on their ability to secure grant funding. These competitive pressures create enormous incentives for otherwise conscientious people, all the way along the hierarchy described above, to fudge their research data. Research fraud is thus a direct outcome of a fundamentally-broken approach to the structure of research funding. There are exceptions to that approach, however. The not-for-profit Howard Hughes Research Institute [1], and to an extent the intramural research programs of the NIH [2], offer funding for PIs to do what they do best, without the pressure of competing for scant funds. Coincidentally, some of the best science comes out of these sites. 1: https://www.hhmi.org/scientists https://www.hhmi.org/scientists 2: https://irp.nih.gov/about-us/what-is-the-irp https://irp.nih.gov/about-us/what-is-the-irp

martincmartin 6y ago
What do you suggest as an alternative?
- cscurmudgeon 6y ago
  Research publication, funding, and jobs should not be tied to outcomes of research. They should be tied to how systematic, logical, and well-documented the research is. We need a system wide change from funding bodies, job committees to publishing criteria.
  - mlthoughts2018 6y ago

owenshen24 6y ago

Interesting notes from the paper mentioned in the article: https://journals.sagepub.com/doi/pdf/10.1177/1747016119898400 https://journals.sagepub.com/doi/pdf/10.1177/174701611989840... - "only 39 scientists from 7 countries have been subject to criminal sanctions between 1979 and 2015 (Oransky and Abritis, 2017)" That seems...very low. - "The Retraction Watch database—the largest of its kind—currently includes more than 18,500 retracted articles (Retraction Watch database, 2019). A recent analysis of 10,500 retracted papers up to 2016 showed that 0.04% of papers are retracted." This is once again a lower-bound; presumably if you account for additional authors and p-hacking the numbers go up a lot. Pushing for replication and improved methodology can help, but some of these issues seem to be related to scale. There are many more people outputting papers than there are people willing to vet them (outside of peer review). Furthermore, when you have many people researching hot fields, you should expect false positives and overestimates to dominate published results, even when everyone is trying to practice good statistical hygiene. (https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...)

grandridge 6y ago

But I thought we should "believe all scientists?!"

deleted 6y ago

[deleted]

jostmey 6y ago

I’ve seen lots of “questionable” research, so much, that I rarely read papers published by other researchers because chances are the data is junk, and I’m in academic biomedical research!!

deleted 6y ago

[deleted]

BurningFrog 6y ago

Starting to think it's time to close down the university system and start a new one...

pas 6y ago
Sure, but how would you organize the new one?
- BurningFrog 6y ago
  I'd just hope new alternatives would emerge as the old system gets out of the way. Then we'll see what develops. Not a selling argument, I know :)

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

DoreenMichele 6y ago

So, some years ago, Temple Grandin wrote a set of standards and McDonald's adopted it and they buy so much beef that it became the de facto new standard for the beef industry. And it's a set of standards that helps beef producers succeed rather than a "gotcha" trying to find who is guilty. And that's the way you make the world a better place. Not by looking for new and creative ways to nail "bad guys" to the wall after you started from an assumption of guilt. I don't like this article. I don't like it at all. My feeling is that it was written as an emotional response to the pandemic and it is getting traction on HN for the exact same reason. People are stressed out and they are looking for a villain to go after. It won't fix the real problem -- the pandemic -- but that's how people tend to behave in a crisis. And it's a slippery slope towards a more draconian world. It doesn't make things better.

_Microft 6y ago
The McDonald's story reminds me of the Brussels effect [0] where legislation in the EU is extending (not by law but in its effect) to other parts of the world because it is easier or cheaper to comply with it for all customers than to treat EU costumers and others differently. [0] https://en.wikipedia.org/wiki/Brussels_effect https://en.wikipedia.org/wiki/Brussels_effect
- deleted 6y ago
  [deleted]
- haihaibye 6y ago
  So that's why I have to accept cookies all the time.
  - _Microft 6y ago
    No, this is either because they have no idea what they are doing, or because they like bossing people around in return for not getting their way regarding user data while pretending this behaviour is because of consumer-unfriendly EU regulations. It's perfectly fine to set cookies that are necessary for providing the service without having to ask the user. They only need to get one's permission if they want to do other things like selling data to advertisers.

hlfy_hn 6y ago

Makes me think of Hans Lehrach. Dunno why https://www.molgen.mpg.de/hans-lehrach https://www.molgen.mpg.de/hans-lehrach

cryptica 6y ago

The monetary system is founded on fraud. So it's not surprising to see that the system as a whole is tolerant of all other kinds of fraud. Those who have the power to fix it have no moral grounds to do so without turning into hypocrites. Nobody wants to be a hypocrite. At least if you keep your mouth shut, you can feel better about yourself; you're just doing what everyone else is doing.

deleted 6y ago

[deleted]

brownbat 6y ago

You can't really improve this until the lead author doesn't have a final say on how to treat outliers. In any given study, there are going to be hundreds of special cases in the data that you didn't anticipate, and you have to decide whether to include or exclude them. Any researcher will subconsciously be more sympathetic to arguments to exclude subjects that go against the principal theory, and less so to subjects that confirm it. And it's a battle of reasonable arguments, most of the cases aren't bright line fraud or misconduct, they're just humans finding some arguments more compelling and the impossibility of escaping our own biases. (And yes, sometimes it's fraud, but fraud is just the tip of the iceberg if we're talking about genuinely improving the reliability of scientific findings.) Prepublication is helpful, but more and more I'm convinced that the only way to do proper science would be to completely disaggregate study design from study execution.

AtOmXpLuS 6y ago

Everything in [AtOmXpLuS](https://www.atomxplus.com https://www.atomxplus.com)