4 ms·

I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done

I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done in different fields. These might be deans, department heads, VPs of R+D labs, and so on. Ideally, all research would have clear measurable impact, but a lot of good research doesn't, at least not immediately, so there needs to be some way for non-experts to measure quality. Conference/journal publications and citation counts are highly imperfect solutions, but I'm not sure what the better candidates are.

nradov 5y ago
The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.
- sgt101 5y ago
  Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
  - BlueTemplar 5y ago
    Looks like it could work if you had to do it in a different university ? (Also, less cronyism.)
jonathanstrange 5y ago
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that. The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.
- goatinaboat 5y ago
  Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. The problem with this assertion is: non-experts are paying for all of this. Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up". Can you see why funding something with no accountability might be considered problematic? There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...
  - chalst 5y ago
    Only experts can make this judgement in the short term. As time goes by, the reception of the work accumulates and it becomes increasingly possible for non-experts to judge. It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.
  - jonathanstrange 5y ago
    If non-experts are in charge of individual funding, then you're virtually guaranteed to get nonsensical decisions. It couldn't possibly work, and even if you got it to work, the results would be abysmal. > Can you see why funding something with no accountability might be considered problematic? Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws. In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?
eek04_ 5y ago
The better candidate is spending more on the evaluation. E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores. This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.

nradov 5y ago

The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.

sgt101 5y ago
Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
- BlueTemplar 5y ago
  Looks like it could work if you had to do it in a different university ? (Also, less cronyism.)

jonathanstrange 5y ago

Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that. The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.

goatinaboat 5y ago
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense. The problem with this assertion is: non-experts are paying for all of this. Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up". Can you see why funding something with no accountability might be considered problematic? There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...
- chalst 5y ago
  Only experts can make this judgement in the short term. As time goes by, the reception of the work accumulates and it becomes increasingly possible for non-experts to judge. It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.
- jonathanstrange 5y ago
  If non-experts are in charge of individual funding, then you're virtually guaranteed to get nonsensical decisions. It couldn't possibly work, and even if you got it to work, the results would be abysmal. > Can you see why funding something with no accountability might be considered problematic? Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws. In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?

eek04_ 5y ago

The better candidate is spending more on the evaluation. E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores. This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.