3 ms·

Ex-Scientist here: While I agree the gates foundation is likely doing a better job than the nih... The danger is this, how do you decide upon a metric for prod

Ex-Scientist here: While I agree the gates foundation is likely doing a better job than the nih... The danger is this, how do you decide upon a metric for productivity? How do you then prevent 1) corruption, absent that, 2) inadvertent corruption via biases (I'm going to judge proposals in my field more favorably because they are less confusing or more comfortable, or the inverse, more harshly since I want to keep competition out or know the pitfalls better), and absent that, 3) goodhart's law?

phkahler 7y ago
Is there a psychological measure of internal bias? Hire decision makers that score low on that?
mistermann 7y ago
How about we vote on it?
pdm55 7y ago
Science is accidental, boring, repetitive, focused, finicky, wasted, political, ... A beast not easily tamed. Think of Edison's 100 trials to develop the light bulb. I continually think of my own truncated scientific career. I couldn't, early on in my projects, see the woods for the trees. About the only thing, I recognised was that I was just one cog in a vast enterprise. Looking back, I wish I had had the energy and insight to see beyond my lab work to where it might lead. Back in those days, you needed the term "business model" in your grant submission in order to attract funds. I walked away, thinking I had interests that could better generate a business.
godelski 7y ago
Current scientist: I think the answer is easy. Take more risk. Science is by nature risky. We're trying to figure out things that we don't know. That results in a lot of failure (I definitely wouldn't use the Silicon Valley model of failure being a metric of success though). One major change I would make from public money is publishing of results good or bad. There's this competition to get into top tier journals (which I don't think is bad or you can get rid of), but this is done at the cost of dumping good science just because it wasn't good enough for a top tier. Using impact as a metric is good but not great. The mark of a good scientist is their work and their passion, not where they publish. While there's a correlation, there's also Goodhart. If this is the only metric that we really concentrate on, then it will lead to slower science. We all know that we learn much more from failure than success. We all know that most of science is banging your head on the table for weeks out of frustration. But that's also not common knowledge. So I'd say that those funding science need to be more risky and those in science need to be more open about their failures and be realistic in their discussion of science.
- sah2ed 7y ago
  > One major change I would make from public money is publishing of results good or bad. There's this competition to get into top tier journals (which I don't think is bad or you can get rid of), but this is done at the cost of dumping good science just because it wasn't good enough for a top tier. Using impact as a metric is good but not great. > The mark of a good scientist is their work and their passion, not where they publish. You are overlooking an invisible but equally important mark of a good scientist: a scientist’s impression on peers aka reputation, which usually comes down to two types: prestigious publication (in a prestigious journal or with a prestigious co-author) or having a good number of citations. The first kind of reputation is bequeathed almost immediately, the second kind of reputation is deferred — it can only be gained after the passage of time (years). If a scientist is known to have co-authored a paper that peers later on characterize as a seminal contribution to the field, peers are much more likely to pay attention to future papers from that scientist. The converse is true. The number of published papers, whether in prestigious journals or otherwise, far exceeds the number of people with free time and relevant qualifications to assess them. At the end of the day, human attention is a scarce resource, which is why it is the metric those aspiring to be regarded as “good scientists” in their field optimize for. Taken together, a scientist’s publication history is used as a proxy for authority in a field, which in turn is used to filter which papers to read or send to the trash can.

phkahler 7y ago

Is there a psychological measure of internal bias? Hire decision makers that score low on that?

mistermann 7y ago

How about we vote on it?

pdm55 7y ago

Science is accidental, boring, repetitive, focused, finicky, wasted, political, ... A beast not easily tamed. Think of Edison's 100 trials to develop the light bulb. I continually think of my own truncated scientific career. I couldn't, early on in my projects, see the woods for the trees. About the only thing, I recognised was that I was just one cog in a vast enterprise. Looking back, I wish I had had the energy and insight to see beyond my lab work to where it might lead. Back in those days, you needed the term "business model" in your grant submission in order to attract funds. I walked away, thinking I had interests that could better generate a business.

godelski 7y ago

Current scientist: I think the answer is easy. Take more risk. Science is by nature risky. We're trying to figure out things that we don't know. That results in a lot of failure (I definitely wouldn't use the Silicon Valley model of failure being a metric of success though). One major change I would make from public money is publishing of results good or bad. There's this competition to get into top tier journals (which I don't think is bad or you can get rid of), but this is done at the cost of dumping good science just because it wasn't good enough for a top tier. Using impact as a metric is good but not great. The mark of a good scientist is their work and their passion, not where they publish. While there's a correlation, there's also Goodhart. If this is the only metric that we really concentrate on, then it will lead to slower science. We all know that we learn much more from failure than success. We all know that most of science is banging your head on the table for weeks out of frustration. But that's also not common knowledge. So I'd say that those funding science need to be more risky and those in science need to be more open about their failures and be realistic in their discussion of science.

sah2ed 7y ago
> One major change I would make from public money is publishing of results good or bad. There's this competition to get into top tier journals (which I don't think is bad or you can get rid of), but this is done at the cost of dumping good science just because it wasn't good enough for a top tier. Using impact as a metric is good but not great. > The mark of a good scientist is their work and their passion, not where they publish. You are overlooking an invisible but equally important mark of a good scientist: a scientist’s impression on peers aka reputation, which usually comes down to two types: prestigious publication (in a prestigious journal or with a prestigious co-author) or having a good number of citations. The first kind of reputation is bequeathed almost immediately, the second kind of reputation is deferred — it can only be gained after the passage of time (years). If a scientist is known to have co-authored a paper that peers later on characterize as a seminal contribution to the field, peers are much more likely to pay attention to future papers from that scientist. The converse is true. The number of published papers, whether in prestigious journals or otherwise, far exceeds the number of people with free time and relevant qualifications to assess them. At the end of the day, human attention is a scarce resource, which is why it is the metric those aspiring to be regarded as “good scientists” in their field optimize for. Taken together, a scientist’s publication history is used as a proxy for authority in a field, which in turn is used to filter which papers to read or send to the trash can.