4 ms·

Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are

Preregistration does help with this issue a bit. It won't make published studies any better, but it still helps solve the file-drawer problem if the results are available somewhere. If you take a shotgun approach to getting significance out of null hypotheses, you presumably get 5% successes. (Sort of. Maybe. p-values really stink, as you point out.) But if 20 people all run variants of the same bunk study, preregistration means that instead of just seeing one success, we see one success on the cover of PNAS, and then at least have the option to look up the ~10 prior failures and decide not to accept the conclusion. (Well, one time in twenty it'll be the first try, but it's still progress.) Of course, that's still going to push researchers to change up their techniques for getting false positives. It rewards either covering unexplored domains (so there's no counterexample to your false positive), or exploiting other tricks to manipulate significance in preregistered work. I don't think existing journals have much hope of improving their practices, honestly. Between their ludicrous financial incentives and ideologues talking "methodological terrorism", that matter seems depressingly settled. But I wonder if there might be hope for creating a counterweight to this? What happens if Gelman and Khahneman get together to create the Very Prestigious Journal of Replications and the Somewhat Prestigious Journal of Rejected Hypotheses? The first one is neat because it rewards negative results, since they change the state of what's understood - at the very least it might encourage more people to start checking the most influential and most suspicious results. The second thing might be a harder sell, but it could reward comprehensive negative work on key topics (much like Alzheimer's research has heavily consisted of excluding possible pathways). Honestly, "create new prestigious journals from scratch" sounds far easier than "reform existing journals" or "tenure and reward professors for reasons other than publications".

08-15 8y ago
> Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.
- Bartweiss 8y ago
  > For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.
  - grigjd3 8y ago
    A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
    - Bartweiss 8y ago
      I wish I could find it, but there's a superb piece out there about the inherent dysfunction of lottery-style careers. The basic idea was that there are a handful of careers where success is highly rewarded, but job count and quality don't really respond to the market. In academia, there are however many tenured professorships with whatever salary, set by largely non-market forces. In Hollywood, there are only so many recognizable stars at a time who can guarantee sales, so their salaries don't really decline no matter how many people vie for their spots. And the same in pro spots, music, politics, Harvard admissions, etc. And the result is that without adding new jobs or dropping salaries, the people hiring balance the market somehow. Unpaid internships, selling expensive credentials, would-be singers selling hits to existing stars, massive university donations by parents, vicious hazing on football teams, horrifying abuse by casting agents; people who control these 1,000:1 (or higher) seeker-to-slot fields manage to exploit that eagerness to all kinds of horrible ends. [1] Academia is probably less bad than many of those fields, perhaps because the rewards at the top are less spectacular. But it's not a coincidence that Brian Wansink's big scandal was about pressuring a grad student to find results in null data, or that grad students and postdocs end up working miserable hours to be third author on their own research. I don't have an answer here, really. But I certainly agree that as long as there are endless ranks of people struggling to make their names, people are going to keep finding ways to get dramatic-looking results, data be damned. [1] I suspect this stuff really kicks in at several hundred to one or higher. Harvard doesn't reject this percentage, but note that it applies a time-and-money filter before the applications even arrive.

08-15 8y ago

> Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.

Bartweiss 8y ago
> For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.
- grigjd3 8y ago
  A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
  - Bartweiss 8y ago
    I wish I could find it, but there's a superb piece out there about the inherent dysfunction of lottery-style careers. The basic idea was that there are a handful of careers where success is highly rewarded, but job count and quality don't really respond to the market. In academia, there are however many tenured professorships with whatever salary, set by largely non-market forces. In Hollywood, there are only so many recognizable stars at a time who can guarantee sales, so their salaries don't really decline no matter how many people vie for their spots. And the same in pro spots, music, politics, Harvard admissions, etc. And the result is that without adding new jobs or dropping salaries, the people hiring balance the market somehow. Unpaid internships, selling expensive credentials, would-be singers selling hits to existing stars, massive university donations by parents, vicious hazing on football teams, horrifying abuse by casting agents; people who control these 1,000:1 (or higher) seeker-to-slot fields manage to exploit that eagerness to all kinds of horrible ends. [1] Academia is probably less bad than many of those fields, perhaps because the rewards at the top are less spectacular. But it's not a coincidence that Brian Wansink's big scandal was about pressuring a grad student to find results in null data, or that grad students and postdocs end up working miserable hours to be third author on their own research. I don't have an answer here, really. But I certainly agree that as long as there are endless ranks of people struggling to make their names, people are going to keep finding ways to get dramatic-looking results, data be damned. [1] I suspect this stuff really kicks in at several hundred to one or higher. Harvard doesn't reject this percentage, but note that it applies a time-and-money filter before the applications even arrive.