4 ms·

The problem is the following: if there is no established OA journal in the field, you can't distinguish the candidate who "deliberately" publishes good quality

by a3_nm 10y ago

The problem is the following: if there is no established OA journal in the field, you can't distinguish the candidate who "deliberately" publishes good quality work in a non-selective OA journal, from the candidate who publishes low-quality work in the same non-selective OA journal. The only way to tell is to read the papers and judge for yourself, and this is the root of the problem: evaluating candidates based on the number of papers and the impact factor (or some analogous metric) of the venues is much, much easier for committees, than actually reading the papers.

brownbat 10y ago
In a very rough analogy, this is why we have film critics, because we can't trust other institutions in the industry (studios, individual theaters, directors) as a guarantee of quality. If a panel of respected tenured professors in the field ranked the best works in open access journals each month, or if there was a high profile yearly prize for best open access papers of the year, that could help disaggregate reputational effects from the publication itself. Bootstrapping this sort of thing is hard. I'm sure Nobels didn't signal as strongly in year one as they do now.
jonathanstrange 10y ago
Exactly, this nonsense of mostly just counting publications must stop. If at all, there is only a very light correlation (or perhaps even a negative one, who knows?) between the number of publication and the quality of a scientist's research. And what's so crazy about it is that everybody knows this. No one has ever been called a great scientist, because he published many articles. Such scientists may be called prolific, but that doesn't necessarily reflect on the quality of the content. The same for counting quotations, by the way. This may be different in STEM fields, but I'm in the humanities. In the humanities bad papers are sometimes quoted as often as brilliant papers, because everybody wants to correct the authors. I've seen more than one colleague literally make his career by writing a polemic and obviously wrong book (books=often improper peer review) and then sneaking a few apologetic articles in which they half-retracted their claims past the reviewers. What's worse, I know postdocs who are actively looking for such untenable topics that will surely stir up a reaction. We're talking about theses, conjectures or claims that nobody defended previously, because they are so obviously wrong or flawed. I've also met people at the postdoc level who literally replied "Who cares, it's one more publication" when I raised doubts about their main claim in an article and mentioned that it might not 'work' for purely formal reasons (but adding that I'd have to check, to be polite). I'm sure it's a bit less drastic in CS and the STEM fields, but in the humanities publication counting almost enforces mediocrity and the publication of uninteresting, pointless papers with vastly overlapping topics - and it's completely obvious that it does so. The situation is particularly frustrating, because the system makes it harder to publish good science. Good science needs larger studies and/or better formal underpinnings, both of which take more time and more effort than duplicating fake papers about fake problems. Well, enough ranting for today, you get the point.
- inimino 10y ago
  It's almost as if academia were set up to advance the careers of those within academia rather than to advance human knowledge.
  - marcosdumay 10y ago
    People will game the metric you impose on them. The onus is on the administrations to get a metric that is gamed by advancing human knowledge, instead of throwing logs at flame wars. That's why they get the big salaries.

by a3_nm 10y ago

brownbat 10y ago

In a very rough analogy, this is why we have film critics, because we can't trust other institutions in the industry (studios, individual theaters, directors) as a guarantee of quality. If a panel of respected tenured professors in the field ranked the best works in open access journals each month, or if there was a high profile yearly prize for best open access papers of the year, that could help disaggregate reputational effects from the publication itself. Bootstrapping this sort of thing is hard. I'm sure Nobels didn't signal as strongly in year one as they do now.

jonathanstrange 10y ago

Exactly, this nonsense of mostly just counting publications must stop. If at all, there is only a very light correlation (or perhaps even a negative one, who knows?) between the number of publication and the quality of a scientist's research. And what's so crazy about it is that everybody knows this. No one has ever been called a great scientist, because he published many articles. Such scientists may be called prolific, but that doesn't necessarily reflect on the quality of the content. The same for counting quotations, by the way. This may be different in STEM fields, but I'm in the humanities. In the humanities bad papers are sometimes quoted as often as brilliant papers, because everybody wants to correct the authors. I've seen more than one colleague literally make his career by writing a polemic and obviously wrong book (books=often improper peer review) and then sneaking a few apologetic articles in which they half-retracted their claims past the reviewers. What's worse, I know postdocs who are actively looking for such untenable topics that will surely stir up a reaction. We're talking about theses, conjectures or claims that nobody defended previously, because they are so obviously wrong or flawed. I've also met people at the postdoc level who literally replied "Who cares, it's one more publication" when I raised doubts about their main claim in an article and mentioned that it might not 'work' for purely formal reasons (but adding that I'd have to check, to be polite). I'm sure it's a bit less drastic in CS and the STEM fields, but in the humanities publication counting almost enforces mediocrity and the publication of uninteresting, pointless papers with vastly overlapping topics - and it's completely obvious that it does so. The situation is particularly frustrating, because the system makes it harder to publish good science. Good science needs larger studies and/or better formal underpinnings, both of which take more time and more effort than duplicating fake papers about fake problems. Well, enough ranting for today, you get the point.

inimino 10y ago
It's almost as if academia were set up to advance the careers of those within academia rather than to advance human knowledge.
- marcosdumay 10y ago
  People will game the metric you impose on them. The onus is on the administrations to get a metric that is gamed by advancing human knowledge, instead of throwing logs at flame wars. That's why they get the big salaries.