5 ms·

This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.

sgt101 5y ago
The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
- anonymousDan 5y ago
  REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
  - sgt101 5y ago
    >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
    - anonymousDan 5y ago
      No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are. I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
      - sgt101 5y ago
        OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...

The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate? The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics? The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.

anonymousDan 5y ago
REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
- sgt101 5y ago
  >which presumably has already passed peer review and been published in a top tier venue So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money. As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
  - anonymousDan 5y ago
    No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are. I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
    - sgt101 5y ago
      OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...