2 ms·

Steadily doing good science, where good science is expected not to produce results much of the time, continues to have no support. We say this, but I haven't h

Steadily doing good science, where good science is expected not to produce results much of the time, continues to have no support. We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?

geofft 9y ago
Here's a system that's gameable, but no more gameable than other things our society seems to be fine with: have peers evaluate you for the quality of your work as opposed to your outputs. Retain people whose peers say "They do good work," even if they don't have a flashy success or novel discovery to show. This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.) As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
- sliverstorm 9y ago
  How do you allocate funding?
  - geofft 9y ago
    Give funding and recognition/status to large organizations (groups of at least 100 researchers) that produce good work, without naming names except perhaps in truly exceptional cases. Refuse meddling from outside incentives that break the organizational abstraction barrier (prizes, fellowships, grant proposals that with individuals named on them, fellowships or grant proposals that stay with individuals across organizations, etc.). Create alternative structures that recognize organizations that are producing results with fame and fortune. Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
hannob 9y ago
> We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible? Are you aware of Registered Reports? https://cos.io/rr/ https://cos.io/rr/

geofft 9y ago

Here's a system that's gameable, but no more gameable than other things our society seems to be fine with: have peers evaluate you for the quality of your work as opposed to your outputs. Retain people whose peers say "They do good work," even if they don't have a flashy success or novel discovery to show. This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.) As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.

sliverstorm 9y ago
How do you allocate funding?
- geofft 9y ago
  Give funding and recognition/status to large organizations (groups of at least 100 researchers) that produce good work, without naming names except perhaps in truly exceptional cases. Refuse meddling from outside incentives that break the organizational abstraction barrier (prizes, fellowships, grant proposals that with individuals named on them, fellowships or grant proposals that stay with individuals across organizations, etc.). Create alternative structures that recognize organizations that are producing results with fame and fortune. Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.

hannob 9y ago

> We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible? Are you aware of Registered Reports? https://cos.io/rr/ https://cos.io/rr/