3 ms·
The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the den
by tptacek 2y ago
The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator. There is a truly gargantuan amount of NIH-funded research happening; NIH funds over 30,000 PIs per year, and those grants cover years worth of research, most of which involve teams of 5-10 people.
I'm not saying research fraud doesn't matter or isn't worth the stories written about it. I'm saying that people commenting on it generally don't have any sense of its scale, and fill in some very weird blanks about it.
- monero-xmr 2y agoI think something like a very public, televised shaming of researchers found to be fraudulent. Furthermore far, far more done to catch fraud before it is published, like bounties paid out, and instant career destruction to the fraudsters. There was decades of “but everyone is doing it!” in the system, so much so that fraudsters became presidents of Ivy League institutions. The reckoning has not come.
- tptacek 2y agoThis is what I mean by how messageboardy this issue is.
- monero-xmr 2y agoWhat is your solution to prevent continued fraud and abuse in our research process, other than random people going through published papers?
- tptacek 2y agoThe optimal amount of fraud in any large system is almost never "zero".
- monero-xmr 2y agoYes I will stick by my opinion we should have scientific fraudsters go on a televised panel to be shamed publicly and mercilessly
- tptacek 2y agoSounds awesome, maybe MrBeast can host.
- dullcrisp 2y agoMrBeast has important things to do.
- fzeroracer 2y agoI hope the deep irony of someone whose named after and advertises a cryptocoin talking about fraudsters being shamed publicly on televised panels isn't lost on you.
- lamename 2y agoSure. As long as all the other fraudsters and science deniers that were just elected or waiting to be appointed go too
- cycrutchfield 2y ago[flagged]
- totallynothoney 2y agoMaoist struggle sessions, now with shitcoins!
- renewiltord 2y agoHave you read this post about p-value suspiciousness as a means of testing for widespread non-replicable results? https://www.cremieux.xyz/p/ranking-fields-by-p-value-suspiciousness https://www.cremieux.xyz/p/ranking-fields-by-p-value-suspici... And that's not to speak of things like biotech which is full of photoshopped images of fluorescence and whatnot.
- tptacek 2y agoI've been on a messageboard for more than a couple years, I am familiar with the concept of p-hacking, and it has nothing to do with what I just wrote.
- renewiltord 2y agoThe claim in the post is that the replication crisis is real. Of course, it's among those you consider messageboard folk so you can dismiss it on those grounds, but I think you are wrong to say it has nothing to do with what you wrote since it attempts to deal with both the numerator and the denominator by taking a sample across fields.
- tptacek 2y agoI don't know what point you're trying to make here. Is it that there are fraudulent research papers in journals? Everybody knows that's true. Is it that there are tens of thousands of them produced every year? That's made up. People are counting on readers not to know or take the time to learn how much research is happening, so that individual instances of the problem (in the United States) seem systemic.
- renewiltord 2y agoThe post makes the point that a large amount of science produced in the US is not trustworthy. Fraudulence implies intentional deception, but it is more likely that it is simply a response to incentives. I don't see why you can't understand what I'm saying, but in the hope that it's easier I'll put it in bullet points and try to be direct * The post is a sample of papers across many fields * It describes a method to determine whether widespread p-hacking is occurring in a field * Widespread p-hacking in a field will lead to unreproducible results (because the results are an artifact of the data rather than reflecting any underlying cause) * You said: The replication crisis stuff is mostly a message board fixation. The problem we have is that people unfamiliar with the field don't have a sense of what the denominator is, only the numerator. * The approach in the post uses a statistical method across many papers across many fields. It does not rely on the existence of some number of non-replicable results (in fact, one should expect some percentage of results to be non-replicable). * Therefore, it doesn't rely on "the numerator" so to speak, but a sample of all papers in a field to characterize the field's replicability. You're a smart guy. I follow your posts here and on Twitter. So I can only assume that either I've flipped some bozo bit or something in my communication style is abstruse. I don't understand why you're saying that p-hacking has nothing to do with the replication crisis (most would expect the two to be quite related), or that saying "the replication crisis is real" means I am claiming that individual papers exist that are non-replicable (which is fine, but not a replication crisis). In either case, we're clearly getting nothing out of talking to each other so I might as well leave off here.
- llm_trw 2y agoIn ml basically every paper from a Western university is either useless to industry or not replicable. It is very much a crisis and one that academia has no interest to solve. In a sane world we'd start with figuring out what the mystery meat in the version of pytorch used for training is and work up from there. There is zero funding for building up the required infrastructure for doing actual science instead of whatever the fuck you call the current mess. Basically anything on a more complex dataset than cifar is probably a statistical fluke because of hyper parameter tuning. To find this out you will need to spend three months building up all the bit rotten code that doesn't work from the paper. To draw the conclusion that everyone is full of shit you'll need to waste years of your life.
- deleted 2y ago[deleted]
- light_hue_1 2y agoI'm an AI/ML researcher. If this was true, then we would never have made progress in the first place. Instead, we're making astounding progress at a rate that no one really would have expected. Learning to replicate papers is a skill. It's just a matter of practice. You'll get it if you try.
- llm_trw 2y agoThe progress isn't happening at Western universities.
- 331c8c71 2y agoWhy you keep emphasihing the "Western" part? Non-western are any better?
- llm_trw 2y agoYeah, I can replicate work out of a paper quite easily if half or more of the contributors have Chinese, Indian or Russian email addresses. A lot of small to medium sized companies fighting for mind share also release one script deploy pipelines for their work, which are better than the academic releases. It's western universities and big tech that releases at best bit rotten code and at worse no code what so ever.
- jghn 2y agoNot directly tied to GP's complaints but another thing that often comes up are people complaining about the frivolity of the research without understanding how basic science works. "Can you believe we're paying money to understand fruit fly mating behavior!?!", well yes, because fruit flies are model organisms and developing an understanding of these very mundane sounding things are important before being able to make broader hypotheses
- ashoeafoot 2y agoIf you cant replicate foundational papers your faculty is a house of cards built on lies.