4 ms·

> Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly bec

by 08-15 8y ago

> Honestly, "create new prestigious journals from scratch" sounds far easier... What used to be "prestigious" has come to mean "high impact factor", mostly because funding agencies needed an "objective" way to grant funding, and being run by bureaucrats, they picked impact factor as a measure of scientific success. For a Journal Of Replications to have impact, researchers would have to cite papers about replications. But that's not expected; you cite the original paper, not the one that replicates a result, and you never hear about the one that failed to reproduce a result. I don't see that approach working, but I'd like to see someone try it.

Bartweiss 8y ago
> For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.
- grigjd3 8y ago
  A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
  - Bartweiss 8y ago
    I wish I could find it, but there's a superb piece out there about the inherent dysfunction of lottery-style careers. The basic idea was that there are a handful of careers where success is highly rewarded, but job count and quality don't really respond to the market. In academia, there are however many tenured professorships with whatever salary, set by largely non-market forces. In Hollywood, there are only so many recognizable stars at a time who can guarantee sales, so their salaries don't really decline no matter how many people vie for their spots. And the same in pro spots, music, politics, Harvard admissions, etc. And the result is that without adding new jobs or dropping salaries, the people hiring balance the market somehow. Unpaid internships, selling expensive credentials, would-be singers selling hits to existing stars, massive university donations by parents, vicious hazing on football teams, horrifying abuse by casting agents; people who control these 1,000:1 (or higher) seeker-to-slot fields manage to exploit that eagerness to all kinds of horrible ends. [1] Academia is probably less bad than many of those fields, perhaps because the rewards at the top are less spectacular. But it's not a coincidence that Brian Wansink's big scandal was about pressuring a grad student to find results in null data, or that grad students and postdocs end up working miserable hours to be third author on their own research. I don't have an answer here, really. But I certainly agree that as long as there are endless ranks of people struggling to make their names, people are going to keep finding ways to get dramatic-looking results, data be damned. [1] I suspect this stuff really kicks in at several hundred to one or higher. Harvard doesn't reject this percentage, but note that it applies a time-and-money filter before the applications even arrive.

by 08-15 8y ago

Bartweiss 8y ago

> For a Journal Of Replications to have impact, researchers would have to cite papers about replications. Yep, this is definitely the biggest weakness for the idea. A journal about replications would probably end up with low impact factor, and wouldn't be able to grab a groundbreaking result to boost its prestige the way some narrow-topic journals have. My only thought for handling that was alluded to with "Gelman and Kahneman"; maintaining a tolerable reputation and impact factor would probably require leaning on big names and people who are devoted to doing replication work regardless of journal. Gelman, for instance, showed up in PNAS with references when he criticized that godawful "boarding and air rage" study. If we're willing to abuse the process even further, there might be room to publish methodology-of-replication papers in statistical or procedural journals and create further citations that way. It's not a very good answer, and I don't think it's going to free anyone from "publish or perish". The best I can really hope for is creating a bit of space for talented, skeptical researchers to add replications to their other work without needing to be tenured and entrenched first.

grigjd3 8y ago
A big part of the problem is there are too many PhDs chasing too little funding. There's no room for negative results when the competition is that big.
- Bartweiss 8y ago
  I wish I could find it, but there's a superb piece out there about the inherent dysfunction of lottery-style careers. The basic idea was that there are a handful of careers where success is highly rewarded, but job count and quality don't really respond to the market. In academia, there are however many tenured professorships with whatever salary, set by largely non-market forces. In Hollywood, there are only so many recognizable stars at a time who can guarantee sales, so their salaries don't really decline no matter how many people vie for their spots. And the same in pro spots, music, politics, Harvard admissions, etc. And the result is that without adding new jobs or dropping salaries, the people hiring balance the market somehow. Unpaid internships, selling expensive credentials, would-be singers selling hits to existing stars, massive university donations by parents, vicious hazing on football teams, horrifying abuse by casting agents; people who control these 1,000:1 (or higher) seeker-to-slot fields manage to exploit that eagerness to all kinds of horrible ends. [1] Academia is probably less bad than many of those fields, perhaps because the rewards at the top are less spectacular. But it's not a coincidence that Brian Wansink's big scandal was about pressuring a grad student to find results in null data, or that grad students and postdocs end up working miserable hours to be third author on their own research. I don't have an answer here, really. But I certainly agree that as long as there are endless ranks of people struggling to make their names, people are going to keep finding ways to get dramatic-looking results, data be damned. [1] I suspect this stuff really kicks in at several hundred to one or higher. Harvard doesn't reject this percentage, but note that it applies a time-and-money filter before the applications even arrive.