4 ms·
It does seem like its way too easy to do this kind of fraud. The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to pr
by bawolff 1mo ago
It does seem like its way too easy to do this kind of fraud.
The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?
- p-e-w 1mo ago> Is there some way to change incentives to get people to try and replicate? Sure. Pay them to do it.
- tempestn 1mo agoIf the culture shifted such that people were skeptical of original results but excited about replication, replication would be the big news and people would want to do it. How to get there, I don't know.
- ChrisMarshallNY 1mo agoI seem to remember a lot of hype about room-temperature superconductors, that croaked upon [lack of] replication. That wasn't too long ago. Before that, cold fusion. We still have the ion drive that doesn't require fuel. I don't think it has ever been successfully replicated, but I think that it also hasn't been successfully repudiated, either. This kind of thing has been going on for a long time. I suspect that not all of it is deliberate fraud, but rather, people wanting their preconceived bias to be satisfied.
- unloader6118 1mo agoReplicating superconductor experiment gives you the conductor and the skill. Both of them are useful as is. Replicating a social experiment, when successful, don't give you anything new.
- qsera 1mo agoThen it "replication" becomes a target and thus would be gamed instead. This is a constant tug of war..
- jasonfarnon 1mo agoNo, that would never happen. People would just be skeptical of most findings until they see replications (where we are headed in the social sciences, anyway). If people see replications the original discoverer will get all the accolades, not the (several) replications.
- vkazanov 1mo agoOr maybe we shouldnt call anything "science" until it assumes replication of foundational results. Physics solved this quite some time ago.
- edgyquant 1mo agoWhy do we even accept anything as true without replication by default?
- EA-3167 1mo agoWhy do people on this site love nutrition and self-medication fads? People are surprisingly thoughtless.
- argee 1mo agoOr as one might say, predictably irrational.
- vjulian 1mo agoBecause “science” doesn’t know everything, and people don’t want to wait until their options are scientifically-validated. Yoga, breathing exercises, meditation and the like were once considered fads and derided by your ‘science’.
- lostlogin 1mo agoHow do you decide whether something works or not?
- qsera 1mo agoYou don't. It is too costly and thus risky. Instead you do a charade and convince authorities (who just want to cover their asses citing some vetted studies) and people that your stuff work. It is "ass covering" all the way. Not "does it work" all the way...
- 028363922092 1mo agoAre you a vendor of orgone accumulators or a staunch believer and customer?
- lelanthran 1mo ago> Yoga, breathing exercises, meditation and the like were once considered fads and derided by your ‘science’. What do you mean "once"?
- dbrans 1mo agoReplication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.
- pinkmuffinere 1mo agoOn the other hand, this seems like a super interesting path for an (unglamorous) PhD -- replicate the N most important findings in a field, subject to some restrictions (maybe focus on the ones that are "easy", or haven't been replicated yet, or the ones that could be patched together to make an acceptable PhD thesis). This sort of stuff really appeals to me, I think I might end up doing something like that in 5-10 years.
- Tomte 1mo agoGood luck finding an advisor/a department that awards PhDs for that. “It’s not novel”.
- pinkmuffinere 1mo agoI think you're right that this will be the most common objection, but wrong that it will be exceedingly difficult to find an advisor that would support this. I assume that if you _can_ overturn some foundational result, you won't face criticism over novelty (potentially you can argue that overturning important accepted results is inherently novel). Then the rest is just math: Assume that n% of foundational studies are demonstrably incorrect if replication were attempted. Then if you replicate X foundational studies, there's a 1 - (1-n/100)^X chance that you'll show one foundational study to be incorrect. Then I think you just need to find an advisor who believes the incorrectness rate n, and the replication attempts X is high enough for you to revert a foundational study over the course of your PhD.
- fercircularbuf 1mo agoI share your enthusiasm for this. Creating a map of the most influential research across all the fields to identify the highest priority work, any work replicating that work or validating that work by building from it would be a good start. Ideally funded by governments, but institutions, companies, and individuals could put up bounties for replicating particular research.
- jasonfarnon 1mo ago"easy to replicate (no fancy equipment needed, just ask some people to proofread)" ? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.
- pinkmuffinere 1mo ago> The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now They were just putting it off.
- MarceliusK 1mo ago[dead]
- mike_hearn 1mo agoReplication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science: https://blog.plan99.net/replication-studies-cant-fix-science-0e195234a280 https://blog.plan99.net/replication-studies-cant-fix-science... N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern: https://www.fantasticanachronism.com/p/whats-wrong-with-social-science-and-how-to-fix-it https://www.fantasticanachronism.com/p/whats-wrong-with-soci... > Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems. So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper. The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there. So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.
- zug_zug 1mo agoMy proposal, once you complete your study, if you find a significant positive result you should be required (prior to publication) to describe your methodology in full, and have another lab (who you never have any contact with) blindly take your methodology, and repeat it (on video), and then do their own independent analysis of the data. It's okay if it's $200k or so to do each of these, the university should pay. The point is if this methodology were applied people would have incentive to cheat anymore. It would also serve as an immediate replication. Schools won't do this because they have a perverse incentive for big successes and quiet retractions. It's corruption all the way up.
- spacephysics 1mo agoWould be a great way for up and coming researchers to learn and apply skills before attempting to work in labs pursuing novel research. Maybe some joint effort across a large number of research schools which function as “proportional to number of students in your program, X number of validation studies must be performed” with maybe some monetary or access incentives aligned. Since we’ve had student loans all but guaranteed, and therefore cost of schooling skyrocketing, the flywheel of pursing higher profits and more bureaucratic cruft (looking at ratios of university staff to students), its no wonder that incentive has extended to research.
- ModernMech 1mo agoFund it. Unfortunately there's hardly any money for regular studies, let alone replication studies. If you want better science there has to be tax dollars behind it. Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
- morning-coffee 1mo ago> Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them. Well, that's the incentive mechanism that keeps the accountability for the results whether we like it or not. The government doesn't immediately go out of business if the research they fund is full of fraud or goes nowhere useful. Moral outrage at all the fraud in research associated with these Universities and government funded studies isn't a powerful enough incentive mechanism to correct that fraud, apparently.