6 ms·

> Many hiring decisions in academia still rely strongly on the reputation of journals where applicants published. This is very weird to me. It seems to me like

by Aaargh20318 9y ago

> Many hiring decisions in academia still rely strongly on the reputation of journals where applicants published. This is very weird to me. It seems to me like a variant of the 'argument from authority' fallacy. Publications should be judged on their content, not on who owns the printing press.

pfortuny 9y ago
Oh, yes. The problem is not rasearch, the problem is the Administration and its permanent quest for "objective" (whatever that means) rating criteria...
ptero 9y ago
If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare). Self-published, unknown journal gives me none of that. Thus when making hiring decisions it does help to overweight established journals. And most tenure tracks in academia make mistakes costly. I do not like the current system, but established, well curated journals do provide some benefits in academia hiring. They do damage, too, but to a different group. My 2c.
- lucasverra 9y ago
  Would it a good use case for a blockchain validation system ? Decentralized (scientific) content reviewer ? Pay scarce resource (attention odf the well educated/trained scientist) with some new coin "SCI-coin" ?
- bobcostas55 9y ago
  >BS can still get through, but it is rare The data[0] suggests the exact opposite: the more prestigious the journal, the more bad science it attracts. [0] http://bjoern.brembs.net/2016/01/even-without-retractions-top-journals-publish-the-least-reliable-science/ http://bjoern.brembs.net/2016/01/even-without-retractions-to...
  - Ar-Curunir 9y ago
    That's possible in a few exceptional cases, but at least in CS, you can see a very visible quality drop as you go from the top conferences to the second-tier conferences.
  - jtmcmc 9y ago
    That was looking at only a very small part of the curve of journals. There is a long tail of really terrible journals publishing terrible science.
- madez 9y ago
  > Self-published, unknown journal gives me none of that. How can you know that? Edit: I don't know why I get the downvotes. My question might be naive, but its serious. I think it is not self-evident why an unkown self-published journal couldn't give you that. Elsevier was also once unknown. For me it sounds like a logical fallacy, but I wanted to know your take on it. It's pretty cheap to just downvote me for asking.
  - rocqua 9y ago
    It's the opposite. You can't be sure that a self-published journal has editors and reviewers do a first check. This means you need to confirm and audit their practices. Compare this to a publication in nature. There, by virtue of the name, you know the article is worth more. (Note that in practice, Nature apparently has quite a few sham publications)
    - madez 9y ago
      If we were talking about self-published articles, I'd agree, but we are not. We are talking about self-published journals. Under self-published journals I understand a journal brought out by the researchers or research organisatons. I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature.
      - HarryHirsch 9y ago
        I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature. You mean a university press or a society journal? That model worked extremely well for > 100 years, until in the 1990s the societies sold their journals to Wiley, Springer and Elsevier, and somewhere around that time Springer took a wrong turn. You'd really like to ask why the selloff happened when it did.
        9y ago
- amelius 9y ago
  Perhaps we should use the equivalent of "page-rank" for scientific papers and their citations as a quality measure (?)
  - ptero 9y ago
    Maybe, but it will take some work. IMO the main challenge is to get wide buy in. Many researchers do not love the current system, but are mostly OK with it. They are not looking to topple it; if everyone is using a different option, they will switch, too, but they are not your passionaries to drive the change.
  - stult 9y ago
    Well, that exists in the form of citations. The more a paper is cited the more valued are its authors. Just as the more often a site is linked to, the higher its page rank. And since you're not dealing with massive numbers of sites with lots of SEO experts, straight citation count is good enough without any of the corrections included in the page rank algorithm. Hiring committees look at both quantity of publications and citations to those publications. The problem is that quantity doesn't always indicate quality. Which is why journal prestige acts as a proxy measure.
    - amelius 9y ago
      > Well, that exists in the form of citations. Page-rank is "transitive" in the sense that it not only counts citations, but also citations-of-citations, etc. I wonder though if such a system can be easily gamed (see SEO), without damaging the reputation of offending authors. Also, we should perhaps have the concept of "negative citations". For example, when the text contains "In this paper we show that the approach of [..] is incorrect."
    - rocqua 9y ago
      I always though this was the impact factor of a journal. However, according to [1] that is not the case. [2] Mentions the 'eigenfactor' which seems to be closer to emulating page-rank. I haven't read much about it though. [1] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [2] http://researchguides.uic.edu/if/impact http://researchguides.uic.edu/if/impact
- nonbel 9y ago
  >"If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare)" You think stuff published in nature is better? I found the opposite, that Nature publishes papers with "sexy" results and poorly described methodology. Nature is one of the worst journals, I cringe when info I want is in a Nature article (although with supplements it is getting better).
tomw2005 9y ago
> Publications should be judged on their content They are but it takes time to build that reputation. Also publishing houses with big pockets can hire good reviewers etc and get established quicker. If Nature churned out rubbish papers all the time people would stop trusting it but their hit rate is pretty high and retractions are issued for poor papers. There are some open source websites that are trusted (e.g. https://arxiv.org/ https://arxiv.org/ and https://www.ncbi.nlm.nih.gov/pmc/ https://www.ncbi.nlm.nih.gov/pmc/) but they are archiving content from already trusted journals. Then there is the 1 Open Source journal I would trust* : http://journals.plos.org/plosone/ http://journals.plos.org/plosone/ I hope that we can accelerate this open sourcing in the years to come. * I'm coming from a Medical Physics background. There may be others I'm not aware of that are also worthwhile.
- roel_v 9y ago
  "There are some open source websites that are trusted (e.g. https://arxiv.org" https://arxiv.org" Lol. Citing arxiv is even worse than citing wikipedia, and some reviewers will reject arxiv citations all together. It's full of junk 'science' by crackpots. Not just 'hey look I ran this regression on a public dataset' bad, but all-out 'I was abducted by aliens, and I made up an equation to show that they took me to Pluto' bad. Maybe it differs by field, I don't know - but suggesting arxiv is 'trusted' the same way Nature is 'trusted' is inane.
  - lorenzhs 9y ago
    Yeah, nobody would consider an arXiv-only article by an unknown author. There are plenty of P=NP "proofs" published on the arXiv. But that isn't what it is for. It's a preprint server where you have to do your own quality control. A lot of work in the theoretical CS community is published in conferences and on the arXiv, often in extended form (additional proofs, plots, etc that didn't fit into the conference publication's page limit) From https://arxiv.org/help/general https://arxiv.org/help/general: "Disclaimer: Papers will be entered in the listings in order of receipt on an impartial basis and appearance of a paper is not intended in any way to convey tacit approval of its assumptions, methods, or conclusions by any agent (electronic, mechanical, or other)."
nnq 9y ago
> should be judged on their content This works in compsci and related because evaluating the quality of content is easy. For biomedical research you generally need one of the very few people with verified expertise in a very narrow sub-sub-field to distinguish solid research vs. "mostly bullshit" vs. "useless intellectual masturbation"... you can't just replicate in order to verify a study where you might need a >1M budget to replicate. And these people's time is extremely expensive when they are simply not "unavailable". Also, even people's knowledge is hard to verify in there areas. Someone might "sound highly informed and experienced" but be "100% full of bullshit" while holding a tenured position... again, especially in the medical science area. So people simply go for some type or 'argument from authority' simply because finding an alternative solution is too hard (and would also piss off too many people and/or corporate interests in the process...) to be practical. A "human chain of trust" is the only economical alternative, unless you want to re-make the entire academic system from scratch along better principles in these areas... So at least you try to ensure that a for-profit entity like Elsevier doesn't "own the human chain of trust".
Vinnl 9y ago
That sounds to be a better idea, but people say (and as an outsider, it does sound believable, I guess) that that's pretty much impossible - see for example [1]: > I regret to say that in reviewing perhaps a hundred grants or job applications and trying to find the ten grants to fund or one person to employ, I do not read every paper in the bibliography and assess the research on the basis of my limited understanding. I just don’t have the time or expertise to read and judge all the papers. [1] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea...

pfortuny 9y ago

Oh, yes. The problem is not rasearch, the problem is the Administration and its permanent quest for "objective" (whatever that means) rating criteria...

ptero 9y ago

If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare). Self-published, unknown journal gives me none of that. Thus when making hiring decisions it does help to overweight established journals. And most tenure tracks in academia make mistakes costly. I do not like the current system, but established, well curated journals do provide some benefits in academia hiring. They do damage, too, but to a different group. My 2c.

lucasverra 9y ago
Would it a good use case for a blockchain validation system ? Decentralized (scientific) content reviewer ? Pay scarce resource (attention odf the well educated/trained scientist) with some new coin "SCI-coin" ?
bobcostas55 9y ago
>BS can still get through, but it is rare The data[0] suggests the exact opposite: the more prestigious the journal, the more bad science it attracts. [0] http://bjoern.brembs.net/2016/01/even-without-retractions-top-journals-publish-the-least-reliable-science/ http://bjoern.brembs.net/2016/01/even-without-retractions-to...
- Ar-Curunir 9y ago
  That's possible in a few exceptional cases, but at least in CS, you can see a very visible quality drop as you go from the top conferences to the second-tier conferences.
- jtmcmc 9y ago
  That was looking at only a very small part of the curve of journals. There is a long tail of really terrible journals publishing terrible science.
madez 9y ago
> Self-published, unknown journal gives me none of that. How can you know that? Edit: I don't know why I get the downvotes. My question might be naive, but its serious. I think it is not self-evident why an unkown self-published journal couldn't give you that. Elsevier was also once unknown. For me it sounds like a logical fallacy, but I wanted to know your take on it. It's pretty cheap to just downvote me for asking.
- rocqua 9y ago
  It's the opposite. You can't be sure that a self-published journal has editors and reviewers do a first check. This means you need to confirm and audit their practices. Compare this to a publication in nature. There, by virtue of the name, you know the article is worth more. (Note that in practice, Nature apparently has quite a few sham publications)
  - madez 9y ago
    If we were talking about self-published articles, I'd agree, but we are not. We are talking about self-published journals. Under self-published journals I understand a journal brought out by the researchers or research organisatons. I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature.
    - HarryHirsch 9y ago
      I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature. You mean a university press or a society journal? That model worked extremely well for > 100 years, until in the 1990s the societies sold their journals to Wiley, Springer and Elsevier, and somewhere around that time Springer took a wrong turn. You'd really like to ask why the selloff happened when it did.
      - 9y ago
amelius 9y ago
Perhaps we should use the equivalent of "page-rank" for scientific papers and their citations as a quality measure (?)
- ptero 9y ago
  Maybe, but it will take some work. IMO the main challenge is to get wide buy in. Many researchers do not love the current system, but are mostly OK with it. They are not looking to topple it; if everyone is using a different option, they will switch, too, but they are not your passionaries to drive the change.
- stult 9y ago
  Well, that exists in the form of citations. The more a paper is cited the more valued are its authors. Just as the more often a site is linked to, the higher its page rank. And since you're not dealing with massive numbers of sites with lots of SEO experts, straight citation count is good enough without any of the corrections included in the page rank algorithm. Hiring committees look at both quantity of publications and citations to those publications. The problem is that quantity doesn't always indicate quality. Which is why journal prestige acts as a proxy measure.
  - amelius 9y ago
    > Well, that exists in the form of citations. Page-rank is "transitive" in the sense that it not only counts citations, but also citations-of-citations, etc. I wonder though if such a system can be easily gamed (see SEO), without damaging the reputation of offending authors. Also, we should perhaps have the concept of "negative citations". For example, when the text contains "In this paper we show that the approach of [..] is incorrect."
  - rocqua 9y ago
    I always though this was the impact factor of a journal. However, according to [1] that is not the case. [2] Mentions the 'eigenfactor' which seems to be closer to emulating page-rank. I haven't read much about it though. [1] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [2] http://researchguides.uic.edu/if/impact http://researchguides.uic.edu/if/impact
nonbel 9y ago
>"If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare)" You think stuff published in nature is better? I found the opposite, that Nature publishes papers with "sexy" results and poorly described methodology. Nature is one of the worst journals, I cringe when info I want is in a Nature article (although with supplements it is getting better).

tomw2005 9y ago

> Publications should be judged on their content They are but it takes time to build that reputation. Also publishing houses with big pockets can hire good reviewers etc and get established quicker. If Nature churned out rubbish papers all the time people would stop trusting it but their hit rate is pretty high and retractions are issued for poor papers. There are some open source websites that are trusted (e.g. https://arxiv.org/ https://arxiv.org/ and https://www.ncbi.nlm.nih.gov/pmc/ https://www.ncbi.nlm.nih.gov/pmc/) but they are archiving content from already trusted journals. Then there is the 1 Open Source journal I would trust* : http://journals.plos.org/plosone/ http://journals.plos.org/plosone/ I hope that we can accelerate this open sourcing in the years to come. * I'm coming from a Medical Physics background. There may be others I'm not aware of that are also worthwhile.

roel_v 9y ago
"There are some open source websites that are trusted (e.g. https://arxiv.org" https://arxiv.org" Lol. Citing arxiv is even worse than citing wikipedia, and some reviewers will reject arxiv citations all together. It's full of junk 'science' by crackpots. Not just 'hey look I ran this regression on a public dataset' bad, but all-out 'I was abducted by aliens, and I made up an equation to show that they took me to Pluto' bad. Maybe it differs by field, I don't know - but suggesting arxiv is 'trusted' the same way Nature is 'trusted' is inane.
- lorenzhs 9y ago
  Yeah, nobody would consider an arXiv-only article by an unknown author. There are plenty of P=NP "proofs" published on the arXiv. But that isn't what it is for. It's a preprint server where you have to do your own quality control. A lot of work in the theoretical CS community is published in conferences and on the arXiv, often in extended form (additional proofs, plots, etc that didn't fit into the conference publication's page limit) From https://arxiv.org/help/general https://arxiv.org/help/general: "Disclaimer: Papers will be entered in the listings in order of receipt on an impartial basis and appearance of a paper is not intended in any way to convey tacit approval of its assumptions, methods, or conclusions by any agent (electronic, mechanical, or other)."

nnq 9y ago

> should be judged on their content This works in compsci and related because evaluating the quality of content is easy. For biomedical research you generally need one of the very few people with verified expertise in a very narrow sub-sub-field to distinguish solid research vs. "mostly bullshit" vs. "useless intellectual masturbation"... you can't just replicate in order to verify a study where you might need a >1M budget to replicate. And these people's time is extremely expensive when they are simply not "unavailable". Also, even people's knowledge is hard to verify in there areas. Someone might "sound highly informed and experienced" but be "100% full of bullshit" while holding a tenured position... again, especially in the medical science area. So people simply go for some type or 'argument from authority' simply because finding an alternative solution is too hard (and would also piss off too many people and/or corporate interests in the process...) to be practical. A "human chain of trust" is the only economical alternative, unless you want to re-make the entire academic system from scratch along better principles in these areas... So at least you try to ensure that a for-profit entity like Elsevier doesn't "own the human chain of trust".

Vinnl 9y ago

That sounds to be a better idea, but people say (and as an outsider, it does sound believable, I guess) that that's pretty much impossible - see for example [1]: > I regret to say that in reviewing perhaps a hundred grants or job applications and trying to find the ten grants to fund or one person to employ, I do not read every paper in the bibliography and assess the research on the basis of my limited understanding. I just don’t have the time or expertise to read and judge all the papers. [1] https://theconversation.com/why-i-disagree-with-nobel-laureates-when-it-comes-to-career-advice-for-scientists-80079 https://theconversation.com/why-i-disagree-with-nobel-laurea...