2 ms·
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accu
by pwinnski 7d ago
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper?
I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
- MikhailTal 7d agoThis is very bad logic 1) Humans also are trained on a subset of human knowledge. 2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)
- zbyforgotp 6d agoSchmidthuber has an answer: https://arxiv.org/abs/0812.4360 https://arxiv.org/abs/0812.4360 For a more practical approach you need to use proxies: https://zby.github.io/commonplace/articles/what-an-automated-reviewer-should-measure/ https://zby.github.io/commonplace/articles/what-an-automated...