4 ms·

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and p

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.

mike_hearn 3y ago
You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.
- whatshisface 3y ago
  You don't need a science police - allocating one half of all funding to replicating existing experiments will solve most of the problem. Why would one scientist commit fraud to protect the reputation of someone they've never met?
  - mike_hearn 3y ago
    Oh boy. They do. They do it all the time. Too many of them see themselves as part of a "team", defending each other against outsiders. The Slack chats between virologists talking about the COVID lab leak hypothesis are a good example of this, they were very explicit about their aims there. Huge funding for replication studies is commonly suggested, but it won't work. I'm basing that view on experience of having read and reported invalid papers across several different fields, spent years following this story and its various ins and outs, and have written extensively about the problems with pseudo-science coming out of official institutions. Replication is the polite society way to talk about a whole range of problems. People say there's a replication crisis because it sounds a lot better than saying there's a large scale fraud and incompetence crisis. But it's not like there are papers out there that abstractly don't replicate, nobody could have known, and when someone tries and fails then there's suddenly a whole process that's followed to root cause why it failed and fix it. There isn't anything even close to that. What actually happens is that very obvious problems are ignored or kicked into the long grass as long as possible, often indefinitely, and sometimes scientists even argue they're being victimized by those reporting problems. This article is a reasonable place to start, if you're new to the problem space: https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/ https://fantasticanachronism.com/2020/09/11/whats-wrong-with... It raises points that are often overlooked due to the "replication crisis" framing, points like: • Many papers that can be replicated are only replicable because they're of little value (e.g. poor children have worse exam results). We don't want to be flooded with highly replicable science that tells us only what everyone already knew. • Many papers are replicable but their claims are still invalid because their logic or methodology is wrong. This category covers significant amounts of COVID science, for example. • It's often unclear exactly what the definition of replicable is. But zooming out from replicability for a moment, I wrote an article with about some of the more egregious problems here: https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 We're talking about a scientific culture that can't even stop obviously machine-generated text from being published on a massive scale. No other part of society has problems like this. Even talking about replication seems like a distraction whilst you have papers being published every day that contain obviously Photoshopped images, mathematically impossible numbers and AI generated text, yet nobody cares and getting even one case "resolved" (paper retracted) requires months or years of external pressure. Think about how big the trust problems are with journalism. How to fix trust in journalism is talked about a lot on the conference circuit, but it's a hard problem. And yet newspapers don't publish garbled gibberish and faked images on a daily basis! That's the scale of the challenge faced with science reform. And even if you somehow manage to drain that swamp, the first step before full replication is attempted is to get the original data, so the analysis steps can be replicated using the original numbers. Whole swathes of science fail here because academics refuse to reveal their data. This happens even when they've been made to sign a statement agreeing that they'll provide it on request. Again, their employers do nothing, so where's the pressure point? Only massive fiscal punishment could cause culture change here but the NSF has a single institutional goal of dishing out as much money as possible. To quote Alvaro de Menard, Why is the Replication Markets project funded by the Department of Defense? If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved"). Pusillanimous reports repeat tired clichés about "training", "transparency", and a "culture of openness" while downplaying the scale of the problem and ignoring the incentives. No serious actions have followed from their recommendations. It's not that they're trying and failing—they appear to be completely oblivious. We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything.

Or working from historical analogies, 3) find scientists with a public and obvious track record of replication, and maybe even a few of the fraud hunters, and put them in charge of the NSF.

mike_hearn 3y ago

You could try to build a science police but it won't work. Congress tried that in the 70s with the ORI. After a few failed prosecutions they basically gave up and are now forgotten - still there and they mount a few prosecutions a year but clearly it has no effect. The sheer scale of the problem cannot be fixed by trying to find people breaking "the rules" because there are no rules to begin with. Attempting to fix this within the framework of government would be incredibly distracting for the state, as it'd lead to endless fights and debates. For example, a reasonable person appointed to run the NSF might conclude they should just defund whole subfields of the social sciences, because there's no way to make them scientific at all, but good luck lasting in the job if you do that. A big part of the problem is that scientists are in hoc to a specific political ideology, so they will have powerful friends in Washington who want to see them stick around producing useful propaganda.

whatshisface 3y ago
You don't need a science police - allocating one half of all funding to replicating existing experiments will solve most of the problem. Why would one scientist commit fraud to protect the reputation of someone they've never met?
- mike_hearn 3y ago
  Oh boy. They do. They do it all the time. Too many of them see themselves as part of a "team", defending each other against outsiders. The Slack chats between virologists talking about the COVID lab leak hypothesis are a good example of this, they were very explicit about their aims there. Huge funding for replication studies is commonly suggested, but it won't work. I'm basing that view on experience of having read and reported invalid papers across several different fields, spent years following this story and its various ins and outs, and have written extensively about the problems with pseudo-science coming out of official institutions. Replication is the polite society way to talk about a whole range of problems. People say there's a replication crisis because it sounds a lot better than saying there's a large scale fraud and incompetence crisis. But it's not like there are papers out there that abstractly don't replicate, nobody could have known, and when someone tries and fails then there's suddenly a whole process that's followed to root cause why it failed and fix it. There isn't anything even close to that. What actually happens is that very obvious problems are ignored or kicked into the long grass as long as possible, often indefinitely, and sometimes scientists even argue they're being victimized by those reporting problems. This article is a reasonable place to start, if you're new to the problem space: https://fantasticanachronism.com/2020/09/11/whats-wrong-with-social-science-and-how-to-fix-it/ https://fantasticanachronism.com/2020/09/11/whats-wrong-with... It raises points that are often overlooked due to the "replication crisis" framing, points like: • Many papers that can be replicated are only replicable because they're of little value (e.g. poor children have worse exam results). We don't want to be flooded with highly replicable science that tells us only what everyone already knew. • Many papers are replicable but their claims are still invalid because their logic or methodology is wrong. This category covers significant amounts of COVID science, for example. • It's often unclear exactly what the definition of replicable is. But zooming out from replicability for a moment, I wrote an article with about some of the more egregious problems here: https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 We're talking about a scientific culture that can't even stop obviously machine-generated text from being published on a massive scale. No other part of society has problems like this. Even talking about replication seems like a distraction whilst you have papers being published every day that contain obviously Photoshopped images, mathematically impossible numbers and AI generated text, yet nobody cares and getting even one case "resolved" (paper retracted) requires months or years of external pressure. Think about how big the trust problems are with journalism. How to fix trust in journalism is talked about a lot on the conference circuit, but it's a hard problem. And yet newspapers don't publish garbled gibberish and faked images on a daily basis! That's the scale of the challenge faced with science reform. And even if you somehow manage to drain that swamp, the first step before full replication is attempted is to get the original data, so the analysis steps can be replicated using the original numbers. Whole swathes of science fail here because academics refuse to reveal their data. This happens even when they've been made to sign a statement agreeing that they'll provide it on request. Again, their employers do nothing, so where's the pressure point? Only massive fiscal punishment could cause culture change here but the NSF has a single institutional goal of dishing out as much money as possible. To quote Alvaro de Menard, Why is the Replication Markets project funded by the Department of Defense? If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved"). Pusillanimous reports repeat tired clichés about "training", "transparency", and a "culture of openness" while downplaying the scale of the problem and ignoring the incentives. No serious actions have followed from their recommendations. It's not that they're trying and failing—they appear to be completely oblivious. We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything.