15 ms·

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewe

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomization handled as well.) Replication has to be considered at the same value as original research. And maybe smaller faculties at R2s pivot to replication hubs. And maybe this is easier for some sections of biology, chemistry and psychology than it is for particle physics. We could start where cost of replication is relatively low and work out the details. It's completely doable in some cases. (It may never be doable in some areas either.)

Eddy_Viscosity2 3y ago
It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?
- ebiester 3y ago
  We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)
  - MichaelZuo 3y ago
    This isn't just not easy, it would probably be extremely political to change the structure of the NSF, National Labs, all universities and colleges, etc., so dramatically.
SkyMarshal 3y ago
> x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?
- nine_k 3y ago
  No, the idea is that the same researcher should produce k papers and n replications, instead of just k + n published papers. I'd argue that since replication is somehow faster than original research, the requirement would count a replication somewhat lower than an original paper (say, at 0.75).
  - ebiester 3y ago
    That is my idea... If we opened it up, there's probably more interesting iterations, such as requiring pre-registration for all papers, having papers with pre-registration count as some portion of a full paper even if they fail so long as the pre-registration passed scrutiny, having non-replicated papers count as some portion of a fully replicated paper, and having replication as a separate category such that there is a minimum k, a minimum n, and a minimum k+n. The non-easy part of this is once we start making changes to the criteria for tenure, this opens up people trying to stuff all the solutions for all of the problems that everyone knows already. (See Above.) Would some one try to stuff code-available for CS conference papers, for example? What does it mean for a poster session? At what point are papers released for pre-print? What does it mean for the tenure clock or the Ph.D clock? Does it mean that pre-tenure can't depend on studies that take time to replicate? What do we do with longitudinal studies? I think you're looking at a 50 year transition where you would have to start simple and iterate.
    - harimau777 3y ago
      Is tenure really as mechanical as "publish this many papers and you get it"? My impression was that it took into account things like impact factor and was much more subjective. If that were the case, then wouldn't you run into problems with whoever decides tenure paying lip service to counting replication or failed pre-registered papers but in practice being biased in favor of original research?
      - ebiester 3y ago
        It depends on where you are. The more prestigious the university, the more that impact factors start to matter. There is also the matter of teaching and service, but I'm not talking about the entire tenure system.
tnecniv 3y ago
Your proposal has a whole slew of issues. First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. Second, who works in their lab? Most of the people doing the leg work in a lab are PhD students, and, to graduate, they need to do something novel to write up in their dissertation. Thus, they can’t just replicate three experiments and get a doctorate. Third, you underestimate how specialized lab groups are — both in terms of the incredibly expensive equipment it is equipped with and the expertise within the lab. Even folks in the same subfield (or even in the same research group!) often don’t have much in common when it comes to interests, experience, and practical skills. For every lab doing new work, you’d basically need a clone of that lab to replicate their work.
- majormajor 3y ago
  > First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. If we're talking about weird incentives and academia you hit on one of the worst ones right here, I think, since nothing there is very closely connected to helping students learn. I know that's a dead horse, but it's VERY easy to find reasons that we shouldn't be too closely attached to the status quo. > For every lab doing new work, you’d basically need a clone of that lab to replicate their work. Hell, that's how startup funding works, or market economies in general. Top-down, non-redundant systems are way more fragile than distributed ecosystems. If you don't have the competition and the complete disconnection, you so much more easily fall into political games of "how do we get this published even if it ain't great" vs "how do we find shit that will survive the competition"
- ebiester 3y ago
  I was thinking that this would be more of a postdoc position, or that most labs would be hybrid: that is, they are doing some portion of original research and replication research. If they are replicating, they are made authors on the paper as a new position on the paper. They get credit, but a different type of credit. A Ph.D would be expected to perform some replication research as part of their package. Finally, we would start with the fields that are easy to replicate and move up. We couldn't replicate CERN if we tried. But we could implement this in psychology tomorrow, for example.
harimau777 3y ago
I think that there's also a lot of psychological/cultural/political issues that work also need to be worked out: If someone wins the Nobel Prize, do the people who replicated their work also win it? When the history books are written do the replicators get equal billing to the people who made the discovery? When selecting candidates for prestigious positions, are they really going to consider a replicator equal to an original researcher?
rapjr9 3y ago
Another approach I've seen actually used in Computer Science and Physics is to make replication a part of teaching to undergrads and masters candidates. The students learn how to do the science, and they get a paper out of replicating the work (which may or may not support the original results), and the field benefits from the replication.

Eddy_Viscosity2 3y ago

It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?

ebiester 3y ago
We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)
- MichaelZuo 3y ago
  This isn't just not easy, it would probably be extremely political to change the structure of the NSF, National Labs, all universities and colleges, etc., so dramatically.

SkyMarshal 3y ago

> x fewer papers but x number of replications, and you are expected to have x replications in your specialty. Could it be simplified it even further to say x number of papers, but they only count if they’re replicated by others in the field?

nine_k 3y ago
No, the idea is that the same researcher should produce k papers and n replications, instead of just k + n published papers. I'd argue that since replication is somehow faster than original research, the requirement would count a replication somewhat lower than an original paper (say, at 0.75).
- ebiester 3y ago
  That is my idea... If we opened it up, there's probably more interesting iterations, such as requiring pre-registration for all papers, having papers with pre-registration count as some portion of a full paper even if they fail so long as the pre-registration passed scrutiny, having non-replicated papers count as some portion of a fully replicated paper, and having replication as a separate category such that there is a minimum k, a minimum n, and a minimum k+n. The non-easy part of this is once we start making changes to the criteria for tenure, this opens up people trying to stuff all the solutions for all of the problems that everyone knows already. (See Above.) Would some one try to stuff code-available for CS conference papers, for example? What does it mean for a poster session? At what point are papers released for pre-print? What does it mean for the tenure clock or the Ph.D clock? Does it mean that pre-tenure can't depend on studies that take time to replicate? What do we do with longitudinal studies? I think you're looking at a 50 year transition where you would have to start simple and iterate.
  - harimau777 3y ago
    Is tenure really as mechanical as "publish this many papers and you get it"? My impression was that it took into account things like impact factor and was much more subjective. If that were the case, then wouldn't you run into problems with whoever decides tenure paying lip service to counting replication or failed pre-registered papers but in practice being biased in favor of original research?
    - ebiester 3y ago
      It depends on where you are. The more prestigious the university, the more that impact factors start to matter. There is also the matter of teaching and service, but I'm not talking about the entire tenure system.

tnecniv 3y ago

Your proposal has a whole slew of issues. First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. Second, who works in their lab? Most of the people doing the leg work in a lab are PhD students, and, to graduate, they need to do something novel to write up in their dissertation. Thus, they can’t just replicate three experiments and get a doctorate. Third, you underestimate how specialized lab groups are — both in terms of the incredibly expensive equipment it is equipped with and the expertise within the lab. Even folks in the same subfield (or even in the same research group!) often don’t have much in common when it comes to interests, experience, and practical skills. For every lab doing new work, you’d basically need a clone of that lab to replicate their work.

majormajor 3y ago
> First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. If we're talking about weird incentives and academia you hit on one of the worst ones right here, I think, since nothing there is very closely connected to helping students learn. I know that's a dead horse, but it's VERY easy to find reasons that we shouldn't be too closely attached to the status quo. > For every lab doing new work, you’d basically need a clone of that lab to replicate their work. Hell, that's how startup funding works, or market economies in general. Top-down, non-redundant systems are way more fragile than distributed ecosystems. If you don't have the competition and the complete disconnection, you so much more easily fall into political games of "how do we get this published even if it ain't great" vs "how do we find shit that will survive the competition"
ebiester 3y ago
I was thinking that this would be more of a postdoc position, or that most labs would be hybrid: that is, they are doing some portion of original research and replication research. If they are replicating, they are made authors on the paper as a new position on the paper. They get credit, but a different type of credit. A Ph.D would be expected to perform some replication research as part of their package. Finally, we would start with the fields that are easy to replicate and move up. We couldn't replicate CERN if we tried. But we could implement this in psychology tomorrow, for example.

harimau777 3y ago

I think that there's also a lot of psychological/cultural/political issues that work also need to be worked out: If someone wins the Nobel Prize, do the people who replicated their work also win it? When the history books are written do the replicators get equal billing to the people who made the discovery? When selecting candidates for prestigious positions, are they really going to consider a replicator equal to an original researcher?

rapjr9 3y ago

Another approach I've seen actually used in Computer Science and Physics is to make replication a part of teaching to undergrads and masters candidates. The students learn how to do the science, and they get a paper out of replicating the work (which may or may not support the original results), and the field benefits from the replication.