4 ms·

> So if I’m understanding correctly: no structured sharing of scientific ideas until they’re replicated? No, you don't. I'm actually struggling to think of a m

by Kamq 3y ago

> So if I’m understanding correctly: no structured sharing of scientific ideas until they’re replicated? No, you don't. I'm actually struggling to think of a more extreme interpretation of my words. Trying something novel is fine. But trying something out, getting a single result, treating it as gospel, and not even trying to find the flaws in it isn't doing science. A lot of federal grant money has gone into institutions that do just that, and I think it should be re-directed elsewhere. To, y'know, institutions that are actually going to do science.

ethanbond 3y ago
Putting something in a journal is literally just publishing a result. It explicitly does not intrinsically carry notions of validity for the exact reasons you point out. How are scientists supposed to share results in a way that doesn’t run the risk of someone else taking it as gospel, or even more likely: of someone on the internet thinking that the scientific community is taking it as gospel (which they’re generally not… again for the exact reasons you mention).
- Kamq 3y ago
  > How are scientists supposed to share results in a way that doesn’t run the risk of someone else taking it as gospel Could not care less, but federal funding should be shifted to institutions that spend >X% (politically negotiable) of resources on replication. And as a consideration for federal funding, groups that get federal funding, should be required to give first crack to journals that devote >X% of space to replication. If people still want to only do novel studies, fine. I can't stop them. But we don't have to fund groups that don't go back and double check their work.
  - ethanbond 3y ago
    I agree with the target you’re shooting for here but I think it’s not nearly that simple to get there. For example, you’d just have labs doing extremely expensive replications of extremely simple or well-established experiments. Why not? IMO maybe an alternative solution: a government-run journal that 1) pays researchers handsome sums upon publication, 2) only publishes high-quality replications, and 3) only publishes replications that increase marginal confidence on a defensible important question. That way this journal could pick up “negative results” and other “not-blockbuster” results that might come about as a byproduct of pursuing other research goals, and there’s no incentive to replicate meaningless work, and there’s no central planning required of the scientific endeavor (of which there’s already too much due to the funding mechanisms).
    - Kamq 3y ago
      > For example, you’d just have labs doing extremely expensive replications of extremely simple or well-established experiments. Why not? You're making grants sound like a cost-plus contract. I don't think grants currently work like that. Why would they be changed? As to why that wouldn't happen under the current system. Because that lab would go under and before they published the replications (these things do get reviewed). And after a certain number of replications, you would (presumably) stop funding those replications. Maybe re-visit it once every decade or so in the softer sciences. You never know what's a product of the current moment in something like sociology or psychology. > IMO maybe an alternative solution: a government-run journal I'd worry about inertia in a government run program. You can just point funding at a different institution when you're only talking about government funding, but when something is government run, it tends to be harder to kill than just re-directing funding. I think you'd have to set up an entire ecosystem of government run journals, but that still makes it harder to pivot if one of the non-government run journals starts doing way better work.

by Kamq 3y ago

ethanbond 3y ago

Putting something in a journal is literally just publishing a result. It explicitly does not intrinsically carry notions of validity for the exact reasons you point out. How are scientists supposed to share results in a way that doesn’t run the risk of someone else taking it as gospel, or even more likely: of someone on the internet thinking that the scientific community is taking it as gospel (which they’re generally not… again for the exact reasons you mention).

Kamq 3y ago
> How are scientists supposed to share results in a way that doesn’t run the risk of someone else taking it as gospel Could not care less, but federal funding should be shifted to institutions that spend >X% (politically negotiable) of resources on replication. And as a consideration for federal funding, groups that get federal funding, should be required to give first crack to journals that devote >X% of space to replication. If people still want to only do novel studies, fine. I can't stop them. But we don't have to fund groups that don't go back and double check their work.
- ethanbond 3y ago
  I agree with the target you’re shooting for here but I think it’s not nearly that simple to get there. For example, you’d just have labs doing extremely expensive replications of extremely simple or well-established experiments. Why not? IMO maybe an alternative solution: a government-run journal that 1) pays researchers handsome sums upon publication, 2) only publishes high-quality replications, and 3) only publishes replications that increase marginal confidence on a defensible important question. That way this journal could pick up “negative results” and other “not-blockbuster” results that might come about as a byproduct of pursuing other research goals, and there’s no incentive to replicate meaningless work, and there’s no central planning required of the scientific endeavor (of which there’s already too much due to the funding mechanisms).
  - Kamq 3y ago
    > For example, you’d just have labs doing extremely expensive replications of extremely simple or well-established experiments. Why not? You're making grants sound like a cost-plus contract. I don't think grants currently work like that. Why would they be changed? As to why that wouldn't happen under the current system. Because that lab would go under and before they published the replications (these things do get reviewed). And after a certain number of replications, you would (presumably) stop funding those replications. Maybe re-visit it once every decade or so in the softer sciences. You never know what's a product of the current moment in something like sociology or psychology. > IMO maybe an alternative solution: a government-run journal I'd worry about inertia in a government run program. You can just point funding at a different institution when you're only talking about government funding, but when something is government run, it tends to be harder to kill than just re-directing funding. I think you'd have to set up an entire ecosystem of government run journals, but that still makes it harder to pivot if one of the non-government run journals starts doing way better work.