6 ms·

For scholarly publishing, the secret sauce - the essential thing - is a mechanism for review I’d venture to say that’s close but not phrased quite right. The s

For scholarly publishing, the secret sauce - the essential thing - is a mechanism for review I’d venture to say that’s close but not phrased quite right. The secret sauce of academic journal publishing is that we get young academics hired. If you can figure out a way for a young junior faculty member to get tenure without publishing in a traditional journal then you’re golden. But as it stands today, the name of the journal is the currency on your CV. Peer review is part of the mechanics of how that works, but it’s the prestige that’s the key part, not the review mechanism itself. Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. They need some form of proxy. Today that’s the impact factor and reputation of the journal title, almost exclusively. It’s entirely possible that that will change and something else (or most likely a variety of something elses) will take its place. There are a lot of people working on alternatives right now. I don’t know what the winners are going to be, but it’s not going to be number of tweets or Facebook likes. If you want to break the existing system you have to figure out how to get a young academic without an established reputation noticed in her field and hired. If you can do that without the need for a journal at all then you can put a crack in the foundation of the industry.

ngoldbaum 12y ago
Astronomy is a bit unique in that there is no real hierarchy of journals in our field. There are really only four main "bread and butter" journals: Astronomical Journal, Astrophyiscal Journal, Monthly Notices of the Royal Astronomical Society, and Astronomy & Astrophysics. A paper published in any one of these journals has roughly equal weight and "impact". I think this unique status will help OJA succeed - astronomers are already used to judging papers from all four journals with roughly equal weight. If some senior people get behind it and start making big, important papers available via OJA, I think it really might take off. Page charges can be seriously expensive. I think many academics would be happy to avoid the whole mess and just post to the arxiv and get peer review via OJA. It's already more or less standard practice to post to the arxiv either upon submission or acceptance, this just formalizes that practice and adds an element of certification of the result.
- seanmcdirmid 12y ago
  That doesn't help with tenure committees. I guess in Astronomy, this is less of a problem, but in computer science, different fields of computer science are so different that those on your committee will have probably never have read what journal you are publishing to. They go by reputation alone.
001sky 12y ago
Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. They need some form of proxy. This is a completely unacceptable situation, it seems.
gioele 12y ago
> Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. I once discussed this with the dean of a humanities faculty in Norway. He said they did not take h-index and similar metrics into account while debating tenure. So I asked: "How do you evaluate the applicants, then?". "We read their papers" he replied. A room full of professors laughed loud. He then went on telling us about how they shortlist the candidates and how many paper each person in the committee has to read. The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. To me it does not seem a great amount of work for somebody that is about to appoint another scholar as their peer. It also seems to me as a very decent way to treat applicants: as people that did contribute interesting knowledge, not data points.
- p4bl0 12y ago
  > The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. This should be the rule, not the exception. I hope we're moving towards that in the post-journal and open peer review era.
  - seanmcdirmid 12y ago
    As a commenter already pointed put, you have to solicit reviews from experts in the field, and there is unlikely to be another one of those persons in your school. Open peer review is a nice idea, but when the people that can review your work is roughly 50 or persons whom you met already...
    - p4bl0 12y ago
      > the people that can review your work is roughly 50 or persons whom you met already... I don't see the problem with this. If the reviews are signed, public, and citable, then it doesn't matter if it is your best friend who reviewed your paper. What he says in the review is public. He engages his name as much as if he was writing a paper, he can't just say "this is good" without being sure that it is and arguing why it is. Just as people would not be able to just say "this is bad" without arguing and pointing out mistakes. I'm pretty sure that such a system would indeed improve the quality of the reviews, and thus the quality of the papers.
pizza_boy 12y ago
I absolutely agree with Doug. The key is prestige. Our approach to that is to turn peer review itself into a measurable research output -- see our recent partnership with PeerJ for an example (http://blog.publons.com/post/85660504608/publons-partners-with-peerj-reviewers-get-more-credit http://blog.publons.com/post/85660504608/publons-partners-wi...)

ngoldbaum 12y ago

Astronomy is a bit unique in that there is no real hierarchy of journals in our field. There are really only four main "bread and butter" journals: Astronomical Journal, Astrophyiscal Journal, Monthly Notices of the Royal Astronomical Society, and Astronomy & Astrophysics. A paper published in any one of these journals has roughly equal weight and "impact". I think this unique status will help OJA succeed - astronomers are already used to judging papers from all four journals with roughly equal weight. If some senior people get behind it and start making big, important papers available via OJA, I think it really might take off. Page charges can be seriously expensive. I think many academics would be happy to avoid the whole mess and just post to the arxiv and get peer review via OJA. It's already more or less standard practice to post to the arxiv either upon submission or acceptance, this just formalizes that practice and adds an element of certification of the result.

seanmcdirmid 12y ago
That doesn't help with tenure committees. I guess in Astronomy, this is less of a problem, but in computer science, different fields of computer science are so different that those on your committee will have probably never have read what journal you are publishing to. They go by reputation alone.

001sky 12y ago

Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. They need some form of proxy. This is a completely unacceptable situation, it seems.

gioele 12y ago

> Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. I once discussed this with the dean of a humanities faculty in Norway. He said they did not take h-index and similar metrics into account while debating tenure. So I asked: "How do you evaluate the applicants, then?". "We read their papers" he replied. A room full of professors laughed loud. He then went on telling us about how they shortlist the candidates and how many paper each person in the committee has to read. The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. To me it does not seem a great amount of work for somebody that is about to appoint another scholar as their peer. It also seems to me as a very decent way to treat applicants: as people that did contribute interesting knowledge, not data points.

p4bl0 12y ago
> The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. This should be the rule, not the exception. I hope we're moving towards that in the post-journal and open peer review era.
- seanmcdirmid 12y ago
  As a commenter already pointed put, you have to solicit reviews from experts in the field, and there is unlikely to be another one of those persons in your school. Open peer review is a nice idea, but when the people that can review your work is roughly 50 or persons whom you met already...
  - p4bl0 12y ago
    > the people that can review your work is roughly 50 or persons whom you met already... I don't see the problem with this. If the reviews are signed, public, and citable, then it doesn't matter if it is your best friend who reviewed your paper. What he says in the review is public. He engages his name as much as if he was writing a paper, he can't just say "this is good" without being sure that it is and arguing why it is. Just as people would not be able to just say "this is bad" without arguing and pointing out mistakes. I'm pretty sure that such a system would indeed improve the quality of the reviews, and thus the quality of the papers.

pizza_boy 12y ago

I absolutely agree with Doug. The key is prestige. Our approach to that is to turn peer review itself into a measurable research output -- see our recent partnership with PeerJ for an example (http://blog.publons.com/post/85660504608/publons-partners-with-peerj-reviewers-get-more-credit http://blog.publons.com/post/85660504608/publons-partners-wi...)