7 ms·
You get what you incentivize. We incentivize papers so we get papers. Also anything used as a metric tends to cease to be a good metric.
by api 3y ago
You get what you incentivize. We incentivize papers so we get papers.
Also anything used as a metric tends to cease to be a good metric.
- caddemon 3y agoI think the main problem is not in itself incentivizing papers or using papers as a metric. The problem is that they're in many ways the only thing incentivized and the only metric given serious consideration. And the general expectations for/form of a paper are so homogenized across a given field. This makes it very feasible to game the system in a particular way that is bad for science. It's also a serious misallocation of incentives purely from a system functioning perspective. At the very least you need to incentivize peer review alongside incentivizing papers. Currently there is hardly any incentive at all to help carefully review papers, so we have no checks and balances in place to deal with the natural result of incentivizing the writing of papers. If there were greater heterogeneity in scientific roles it wouldn't be so bad for some to have heavy paper incentives -- because there'd be others with heavy incentives to do a good job highlighting the best papers and finding flaws in the bad ones.
- riku_iki 3y agoThere is a good metric in AI research: reproducible results on well known benchmarks (Big Bench for example).
- sebzim4500 3y agoThere's no good benchmark that tests for most of the issues with LLMs, for example. Maybe if we had a benchmark that told you how often a model hallucinated we would have solved the problem by now.
- riku_iki 3y agoThere are plenty of benchmarks with factual questions.