40 ms·

> Does the research not stand on its own? Academic mathematician here. It does, sort of, but ..... Say a mathematics postdoc is applying for a job, and she wo

> Does the research not stand on its own? Academic mathematician here. It does, sort of, but ..... Say a mathematics postdoc is applying for a job, and she works in underwater basketweaving. The two underwater basketweavers in her dream department read her application and can see that she's brilliant. She's developed a new methodology for weaving rattan under 100 feet of ocean without even getting it wet. She has letters from the most preeminent underwater basketweavers in her field. She should be a shoo-in to get the job. One problem though. A majority of the hiring committee works in different areas of research. One thinks that underwater basketweaving is overhyped. Another thinks that great work was done in the eighties, but all the interesting questions have been solved already. And these people don't put much stock in the letter writers: they have a reputation for writing glowing letters even for mediocre candidates. And, so far, the hiring meeting is deadlocked until the basketweavers pipe in: Our candidate has published in Journal für die reine und angewandte Mathematik and Inventiones Mathematicae and Publications Mathématiques de l'IHÉS. And now, all of a sudden, that postdoc is getting the interview, because her competitors have only published in Mathematische Annalen and Indagationes Mathematicae. (These are all real names of math journals.) That was a bit long winded, and please forgive me if it's a bit silly, but the point is that we need mechanisms to compare researchers working in different subdisciplines, for researchers to "prove" to the community at large that they're doing good work. I wish we could eschew these games, and I certainly wish we could ditch the commercial publishing industry, but we compete with each other for jobs, for grant funding, and for fame, and we need to establish some kind of metric for evaluating researchers which is as fair as possible. It is not perfect; certainly it is not completely fair. But I believe it works reasonably well.

brians 10y ago
Do those sorts of differences in early publishing predict later success—say, in attracting grants or top grad students, when you adjust for the benefits of being at a top school? If not, maybe it doesn't matter which is hired, as with the undergraduates so often told that a whole admitted class could be set aside, the next N taken, and the world would go on.
- impendia 10y ago
  > Do those sorts of differences in early publishing predict later success? Absolutely yes. They are perhaps the most reliable yardstick for past success that we have available, and past success is the best predictor of future success.
  - rflrob 10y ago
    Citation? Also, it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree". Ive been thinking for a while on how you might actually measure the effect of pedigree, beyond the identity of university. I believe that the specific research group matters, but that's a lot harder to prove (smaller N, harder to get reliable data about counterfacuals).
    - JumpCrisscross 10y ago
      > it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree" This is a valid social question. As a hirer, however, you are measured by your success, not your success benchmarked to some other thing.
euyyn 10y ago
> we need mechanisms to compare researchers working in different subdisciplines, for researchers to "prove" to the community at large that they're doing good work. But the acceptance into those journals is decided by editors, with the same thought process of the hiring committee. What makes their decision any more valuable?
- mrkgnao 10y ago
  Each journal focuses on a specific subfield of math/CS/$SUBJECT, so they can make more informed evaluations on the quality of the research than an expert in some other field can.
  - lisper 10y ago
    OK, but then why don't the editorial board of these journals just get together to do reviews and jettison the ridiculously high overhead of the publishers, and the concomitant games with copyrights and paywalls needed to sustain that overhead? I mean, seriously, it's 2016. When was the last time you read a paper printed on actual paper?
    - dagw 10y ago
      When was the last time you read a paper printed on actual paper? I know it's not what you meant, but I almost always print out papers before reading them.
      - lisper 10y ago
        Yes, I should have said: when was the last time you read a paper in a bound journal printed by a publisher?
  - euyyn 10y ago
    Sorry but that's not the point GP was making: > One thinks that underwater basketweaving is overhyped. Another thinks that great work was done in the eighties, but all the interesting questions have been solved already. Then he cites three general Math journals as what the hiring committee can use to rank the candidates, not a journal specialized in underwater basket-weaving.
kriro 10y ago
I feel like one step for fixing this would be a pledge by hiring committee members. Say the university of X thinks publisher Y is a burden on the scientific community and as such will ignore everything the applicants have published there (or even weight it negatively). It's never going to happen but these issues need to be attacked from all angles that affect the careers of scientists. Employers, publishers, peers and students.
- Intermernet 10y ago
  I honestly think that it would only take one or two high class universities to boycott the large publishing corps to bring the whole house of cards down. If MIT and Stanford (for instance) made it a policy to publicly recommend and prefer arxiv etc then it would set a trend that would be emulated by the rest of the universities. It takes some high profile researchers in each field to start the ball rolling. They are the people whose work others will track down wherever it's published.
- impendia 10y ago
  I strongly disagree. You can't penalize young scientists for playing the game; they pretty much have to. I believe it is much more appropriate to put pressure on senior, tenured scientists to publish their work in "ethical" venues. It's more challenging to pick a fight with people in positions of power, and of course they can simply ignore you, but their careers aren't going to suffer nearly as much if they avoid some of the big-name journals.
CaptainZapp 10y ago
That was a bit long winded, and please forgive me if it's a bit silly It was neither. For somebody not working in science it was a fascinating and very illustrative summary and represents very much one of the reasons why I'm here. Thanks!
Al-Khwarizmi 10y ago
And that's in countries with relatively reasonable hiring practices, like the US. In Spain it's even worse: for positions at universities there are so-called "objective" scales of assessment, so that a paper scores a number of points depending on the position of the journal it's published in in the ISI JCR journal ranking. Points are added and the top scoring candidate wins. For hiring postdocs in projects, the PI typically has some more leeway to set the assessment scales, but not too much. As a PI who is hiring right now, I can use an interview or personal assessment of candidates' work to tip the scales a bit (at my place up to 15% of the grade or so, little more than a tiebreaker) but if I get a candidate with significantly more JCR-indexed papers than another, I basically have to hire them even if I think their track record is worse (which is often the case as one can get lots of "points" by submitting tons of very similar papers on an easy-to-publish area). Because doing otherwise is, you know, not "objective". So here, basically, if you publish in a discipline where JCR-indexed journals are dominated by the likes of Elsevier, you absolutely need to publish there or not get hired, period. When people in my field from more fortunate countries say things like "let's just ditch the journals and publish everything in arXiv", I would agree with them in an ideal world, but in my world they would push my whole country out of the field...
- euyyn 10y ago
  Well, you're talking of a publicly-owned university, and the perception of "mamoneo" and "amiguismo" funded by taxpayer money is, in my opinion, already high enough that I welcome anything that tries to make the hiring system more objective. Mind you: The hiring policy at Google is that managers don't get to interview the potential hires they will manage, nor have any say in the hiring process. For technical positions, five engineers interview the candidate independently, and submit their feedback and recommendation to a hiring committee that makes the decision. The process tries to be as objective as possible, and the manager has no leeway at all. That doesn't mean the Spanish system is any good, just that I don't think giving the PI more say is a great solution. The points assigned to "easy-to-publish areas" can be reduced if PIs give that as feedback. A way to evaluate a candidate without the numerical scale silliness could be devised too.
  - 10y ago

brians 10y ago

Do those sorts of differences in early publishing predict later success—say, in attracting grants or top grad students, when you adjust for the benefits of being at a top school? If not, maybe it doesn't matter which is hired, as with the undergraduates so often told that a whole admitted class could be set aside, the next N taken, and the world would go on.

impendia 10y ago
> Do those sorts of differences in early publishing predict later success? Absolutely yes. They are perhaps the most reliable yardstick for past success that we have available, and past success is the best predictor of future success.
- rflrob 10y ago
  Citation? Also, it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree". Ive been thinking for a while on how you might actually measure the effect of pedigree, beyond the identity of university. I believe that the specific research group matters, but that's a lot harder to prove (smaller N, harder to get reliable data about counterfacuals).
  - JumpCrisscross 10y ago
    > it's worth pointing out that the question was not "do they predict success", but "do they predict success controlling for pedigree" This is a valid social question. As a hirer, however, you are measured by your success, not your success benchmarked to some other thing.

euyyn 10y ago

> we need mechanisms to compare researchers working in different subdisciplines, for researchers to "prove" to the community at large that they're doing good work. But the acceptance into those journals is decided by editors, with the same thought process of the hiring committee. What makes their decision any more valuable?

mrkgnao 10y ago
Each journal focuses on a specific subfield of math/CS/$SUBJECT, so they can make more informed evaluations on the quality of the research than an expert in some other field can.
- lisper 10y ago
  OK, but then why don't the editorial board of these journals just get together to do reviews and jettison the ridiculously high overhead of the publishers, and the concomitant games with copyrights and paywalls needed to sustain that overhead? I mean, seriously, it's 2016. When was the last time you read a paper printed on actual paper?
  - dagw 10y ago
    When was the last time you read a paper printed on actual paper? I know it's not what you meant, but I almost always print out papers before reading them.
    - lisper 10y ago
      Yes, I should have said: when was the last time you read a paper in a bound journal printed by a publisher?
- euyyn 10y ago
  Sorry but that's not the point GP was making: > One thinks that underwater basketweaving is overhyped. Another thinks that great work was done in the eighties, but all the interesting questions have been solved already. Then he cites three general Math journals as what the hiring committee can use to rank the candidates, not a journal specialized in underwater basket-weaving.

kriro 10y ago

I feel like one step for fixing this would be a pledge by hiring committee members. Say the university of X thinks publisher Y is a burden on the scientific community and as such will ignore everything the applicants have published there (or even weight it negatively). It's never going to happen but these issues need to be attacked from all angles that affect the careers of scientists. Employers, publishers, peers and students.

Intermernet 10y ago
I honestly think that it would only take one or two high class universities to boycott the large publishing corps to bring the whole house of cards down. If MIT and Stanford (for instance) made it a policy to publicly recommend and prefer arxiv etc then it would set a trend that would be emulated by the rest of the universities. It takes some high profile researchers in each field to start the ball rolling. They are the people whose work others will track down wherever it's published.
impendia 10y ago
I strongly disagree. You can't penalize young scientists for playing the game; they pretty much have to. I believe it is much more appropriate to put pressure on senior, tenured scientists to publish their work in "ethical" venues. It's more challenging to pick a fight with people in positions of power, and of course they can simply ignore you, but their careers aren't going to suffer nearly as much if they avoid some of the big-name journals.

CaptainZapp 10y ago

That was a bit long winded, and please forgive me if it's a bit silly It was neither. For somebody not working in science it was a fascinating and very illustrative summary and represents very much one of the reasons why I'm here. Thanks!

Al-Khwarizmi 10y ago

And that's in countries with relatively reasonable hiring practices, like the US. In Spain it's even worse: for positions at universities there are so-called "objective" scales of assessment, so that a paper scores a number of points depending on the position of the journal it's published in in the ISI JCR journal ranking. Points are added and the top scoring candidate wins. For hiring postdocs in projects, the PI typically has some more leeway to set the assessment scales, but not too much. As a PI who is hiring right now, I can use an interview or personal assessment of candidates' work to tip the scales a bit (at my place up to 15% of the grade or so, little more than a tiebreaker) but if I get a candidate with significantly more JCR-indexed papers than another, I basically have to hire them even if I think their track record is worse (which is often the case as one can get lots of "points" by submitting tons of very similar papers on an easy-to-publish area). Because doing otherwise is, you know, not "objective". So here, basically, if you publish in a discipline where JCR-indexed journals are dominated by the likes of Elsevier, you absolutely need to publish there or not get hired, period. When people in my field from more fortunate countries say things like "let's just ditch the journals and publish everything in arXiv", I would agree with them in an ideal world, but in my world they would push my whole country out of the field...

euyyn 10y ago
Well, you're talking of a publicly-owned university, and the perception of "mamoneo" and "amiguismo" funded by taxpayer money is, in my opinion, already high enough that I welcome anything that tries to make the hiring system more objective. Mind you: The hiring policy at Google is that managers don't get to interview the potential hires they will manage, nor have any say in the hiring process. For technical positions, five engineers interview the candidate independently, and submit their feedback and recommendation to a hiring committee that makes the decision. The process tries to be as objective as possible, and the manager has no leeway at all. That doesn't mean the Spanish system is any good, just that I don't think giving the PI more say is a great solution. The points assigned to "easy-to-publish areas" can be reduced if PIs give that as feedback. A way to evaluate a candidate without the numerical scale silliness could be devised too.
- 10y ago