5 ms·

> Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. I once discussed this with the dean o

by gioele 12y ago

> Tenure committees can’t read the actual papers of all the applicants to form an opinion on the quality of the research. I once discussed this with the dean of a humanities faculty in Norway. He said they did not take h-index and similar metrics into account while debating tenure. So I asked: "How do you evaluate the applicants, then?". "We read their papers" he replied. A room full of professors laughed loud. He then went on telling us about how they shortlist the candidates and how many paper each person in the committee has to read. The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. To me it does not seem a great amount of work for somebody that is about to appoint another scholar as their peer. It also seems to me as a very decent way to treat applicants: as people that did contribute interesting knowledge, not data points.

p4bl0 12y ago
> The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. This should be the rule, not the exception. I hope we're moving towards that in the post-journal and open peer review era.
- seanmcdirmid 12y ago
  As a commenter already pointed put, you have to solicit reviews from experts in the field, and there is unlikely to be another one of those persons in your school. Open peer review is a nice idea, but when the people that can review your work is roughly 50 or persons whom you met already...
  - p4bl0 12y ago
    > the people that can review your work is roughly 50 or persons whom you met already... I don't see the problem with this. If the reviews are signed, public, and citable, then it doesn't matter if it is your best friend who reviewed your paper. What he says in the review is public. He engages his name as much as if he was writing a paper, he can't just say "this is good" without being sure that it is and arguing why it is. Just as people would not be able to just say "this is bad" without arguing and pointing out mistakes. I'm pretty sure that such a system would indeed improve the quality of the reviews, and thus the quality of the papers.
    - gioele 12y ago
      > If the reviews are signed, public, and citable If the reviews were signed, public and citable there would be much fewer critical reviews, especially in small fields. Cue all arguments in favour of anonymous participation on the net.
      - p4bl0 12y ago
        > there would be much fewer critical reviews I'm not entirely sure about that. But even if it is the case, so what? People who know each other could continue to send everything which could have been in a critical review via private email as they do today to help improve the papers. If what you assume is true, then it would mean that an article with no reviews must be one which would have had critical reviews, so just ignore it. No one can force you into writing a review saying in your name something you don't believe in. If you only want those with positive reviews it's okay, since those reviews still exist in your model. Nonetheless, a review is not an upvote or a downvote, it's a small summary of the paper, plus comments, plus an overall impression of what are the contributions and their qualities. So I think that signed, public, and citable reviews would actually be better than what we have today, if only because the shitty reviews we sometimes get or write would just disappear.
kriro 12y ago
This should be the norm and not something that is laughed at. For an absurd comparison, who would hire a programmer based on counting the LOC in his github repository :D Then again I suspect that quite a few companies don't read the code of applicants either and prefer job interview testing of sorts. I wouldn’t mind phone screening either. Ask the candidate to name his best/worst paper beforehand and discuss it over the phone (alternatively pick one without the candidate knowing it beforehand). Talk about methodology/his science theoretical baseline and where he thinks the field is headed.
- lazyjeff 12y ago
  The current hiring process already does this over 2 days of 1-on-1 interviews with over half the faculty. Those faculty can ask whatever questions they'd like (talking about specific papers and methods are very common). But there are a lot of smooth talkers who aren't very productive, but can market themselves well in an interview. So looking at actual published work is also important. Given the choice between pure interview, versus counting the number of peer-reviewed publications in top conferences/journals, I'd most certainly pick the latter. In the worst case, you'd get someone who can get a lot of work done and published, versus someone who can talk about "big" problems and sound confident.
impendia 12y ago
I am a mathematics professor who has served on several hiring committees. In our context this would be impossible. In particular, I was asked to help hire an algebraic geometer -- algebraic geometry not being my specialty. For me to read and understand the highly technical papers of each of 500 job applicants, outside my area of expertise, would take an astronomical amount of effort -- completely out of the question. And not only that, but distinguishing thorny unsolved problems from fairly routine technical exercises is quite a challenge -- one reason that high-end journals solicit multiple reviews. It is sometimes possible to fool many (but not all) of the experts. We rely principally on recommendation letters (generic, and submitted via a centralized system, so candidates can get one set of letters for all jobs to which they apply). But, yes, which journals candidates have published in is important.
- xerophtye 12y ago
  >high-end journals solicit multiple reviews. It is sometimes possible to fool many (but not all) of the experts. I thought this was a debated point. Don't a lot of people assert that the prestigious high end journals don't really review them for correctness etc, but instead focus on how "ground-breaking" or "news worthy" ??
  - grayclhn 12y ago
    Even that part is difficult.
  - ylem 12y ago
    As a referee you are asked about impact and correctness.
  - duaneb 12y ago
    Experts are sometimes necessary to identify and translate the papers into something that you can understand—include whether it's ground-breaking or news-worthy.
- gioele 12y ago
  > In particular, I was asked to help hire an algebraic geometer -- algebraic geometry not being my specialty. Why were you selected then? > For me to read and understand the highly technical papers of each of 500 job applicants Why did you, as a professor, wasted your time even reading the 500 names of the applicants? Wouldn't your time be better spent if somebody else did some background research, filtered 95% of the candidates and you were provided only a short list of the best candidates that you could carefully inspect?

by gioele 12y ago

p4bl0 12y ago

> The figures where something like one paper or small book read in depth for each candidate and five papers read more casually. The candidates choose the reading list. This should be the rule, not the exception. I hope we're moving towards that in the post-journal and open peer review era.

seanmcdirmid 12y ago
As a commenter already pointed put, you have to solicit reviews from experts in the field, and there is unlikely to be another one of those persons in your school. Open peer review is a nice idea, but when the people that can review your work is roughly 50 or persons whom you met already...
- p4bl0 12y ago
  > the people that can review your work is roughly 50 or persons whom you met already... I don't see the problem with this. If the reviews are signed, public, and citable, then it doesn't matter if it is your best friend who reviewed your paper. What he says in the review is public. He engages his name as much as if he was writing a paper, he can't just say "this is good" without being sure that it is and arguing why it is. Just as people would not be able to just say "this is bad" without arguing and pointing out mistakes. I'm pretty sure that such a system would indeed improve the quality of the reviews, and thus the quality of the papers.
  - gioele 12y ago
    > If the reviews are signed, public, and citable If the reviews were signed, public and citable there would be much fewer critical reviews, especially in small fields. Cue all arguments in favour of anonymous participation on the net.
    - p4bl0 12y ago
      > there would be much fewer critical reviews I'm not entirely sure about that. But even if it is the case, so what? People who know each other could continue to send everything which could have been in a critical review via private email as they do today to help improve the papers. If what you assume is true, then it would mean that an article with no reviews must be one which would have had critical reviews, so just ignore it. No one can force you into writing a review saying in your name something you don't believe in. If you only want those with positive reviews it's okay, since those reviews still exist in your model. Nonetheless, a review is not an upvote or a downvote, it's a small summary of the paper, plus comments, plus an overall impression of what are the contributions and their qualities. So I think that signed, public, and citable reviews would actually be better than what we have today, if only because the shitty reviews we sometimes get or write would just disappear.

kriro 12y ago

This should be the norm and not something that is laughed at. For an absurd comparison, who would hire a programmer based on counting the LOC in his github repository :D Then again I suspect that quite a few companies don't read the code of applicants either and prefer job interview testing of sorts. I wouldn’t mind phone screening either. Ask the candidate to name his best/worst paper beforehand and discuss it over the phone (alternatively pick one without the candidate knowing it beforehand). Talk about methodology/his science theoretical baseline and where he thinks the field is headed.

lazyjeff 12y ago
The current hiring process already does this over 2 days of 1-on-1 interviews with over half the faculty. Those faculty can ask whatever questions they'd like (talking about specific papers and methods are very common). But there are a lot of smooth talkers who aren't very productive, but can market themselves well in an interview. So looking at actual published work is also important. Given the choice between pure interview, versus counting the number of peer-reviewed publications in top conferences/journals, I'd most certainly pick the latter. In the worst case, you'd get someone who can get a lot of work done and published, versus someone who can talk about "big" problems and sound confident.

impendia 12y ago

I am a mathematics professor who has served on several hiring committees. In our context this would be impossible. In particular, I was asked to help hire an algebraic geometer -- algebraic geometry not being my specialty. For me to read and understand the highly technical papers of each of 500 job applicants, outside my area of expertise, would take an astronomical amount of effort -- completely out of the question. And not only that, but distinguishing thorny unsolved problems from fairly routine technical exercises is quite a challenge -- one reason that high-end journals solicit multiple reviews. It is sometimes possible to fool many (but not all) of the experts. We rely principally on recommendation letters (generic, and submitted via a centralized system, so candidates can get one set of letters for all jobs to which they apply). But, yes, which journals candidates have published in is important.

xerophtye 12y ago
>high-end journals solicit multiple reviews. It is sometimes possible to fool many (but not all) of the experts. I thought this was a debated point. Don't a lot of people assert that the prestigious high end journals don't really review them for correctness etc, but instead focus on how "ground-breaking" or "news worthy" ??
- grayclhn 12y ago
  Even that part is difficult.
- ylem 12y ago
  As a referee you are asked about impact and correctness.
- duaneb 12y ago
  Experts are sometimes necessary to identify and translate the papers into something that you can understand—include whether it's ground-breaking or news-worthy.
gioele 12y ago
> In particular, I was asked to help hire an algebraic geometer -- algebraic geometry not being my specialty. Why were you selected then? > For me to read and understand the highly technical papers of each of 500 job applicants Why did you, as a professor, wasted your time even reading the 500 names of the applicants? Wouldn't your time be better spent if somebody else did some background research, filtered 95% of the candidates and you were provided only a short list of the best candidates that you could carefully inspect?