5 ms·

If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare). Self-publis

by ptero 9y ago

If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare). Self-published, unknown journal gives me none of that. Thus when making hiring decisions it does help to overweight established journals. And most tenure tracks in academia make mistakes costly. I do not like the current system, but established, well curated journals do provide some benefits in academia hiring. They do damage, too, but to a different group. My 2c.

lucasverra 9y ago
Would it a good use case for a blockchain validation system ? Decentralized (scientific) content reviewer ? Pay scarce resource (attention odf the well educated/trained scientist) with some new coin "SCI-coin" ?
bobcostas55 9y ago
>BS can still get through, but it is rare The data[0] suggests the exact opposite: the more prestigious the journal, the more bad science it attracts. [0] http://bjoern.brembs.net/2016/01/even-without-retractions-top-journals-publish-the-least-reliable-science/ http://bjoern.brembs.net/2016/01/even-without-retractions-to...
- Ar-Curunir 9y ago
  That's possible in a few exceptional cases, but at least in CS, you can see a very visible quality drop as you go from the top conferences to the second-tier conferences.
- jtmcmc 9y ago
  That was looking at only a very small part of the curve of journals. There is a long tail of really terrible journals publishing terrible science.
madez 9y ago
> Self-published, unknown journal gives me none of that. How can you know that? Edit: I don't know why I get the downvotes. My question might be naive, but its serious. I think it is not self-evident why an unkown self-published journal couldn't give you that. Elsevier was also once unknown. For me it sounds like a logical fallacy, but I wanted to know your take on it. It's pretty cheap to just downvote me for asking.
- rocqua 9y ago
  It's the opposite. You can't be sure that a self-published journal has editors and reviewers do a first check. This means you need to confirm and audit their practices. Compare this to a publication in nature. There, by virtue of the name, you know the article is worth more. (Note that in practice, Nature apparently has quite a few sham publications)
  - madez 9y ago
    If we were talking about self-published articles, I'd agree, but we are not. We are talking about self-published journals. Under self-published journals I understand a journal brought out by the researchers or research organisatons. I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature.
    - HarryHirsch 9y ago
      I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature. You mean a university press or a society journal? That model worked extremely well for > 100 years, until in the 1990s the societies sold their journals to Wiley, Springer and Elsevier, and somewhere around that time Springer took a wrong turn. You'd really like to ask why the selloff happened when it did.
      - madez 9y ago
        But then why again did I get downvoted?
        upvotinglurker 9y ago
        I believe when they said "self-published, unknown" journals, the key word was unknown rather than self-published. They're not talking about journals self-published by established scholarly societies and so forth, who are "known." Nature (or other known journals) come with an established reputation (deserved or not) for publishing high quality work; an unknown journal (i.e. one with no established reputation one way or the other) by definition cannot provide this. In recent years, journals that claim to have a peer review process but which actually offer dubious or no review or quality control have proliferated (some are even published by companies like Elsevier). Having no established reputation doesn't prove that a journal is bad, but having an established reputation is a heuristic shortcut for evaluating whether it is good (at least in the "no one was ever fired for buying IBM" sense). Hiring committees use this heuristic to save time when filtering candidates.
amelius 9y ago
Perhaps we should use the equivalent of "page-rank" for scientific papers and their citations as a quality measure (?)
- ptero 9y ago
  Maybe, but it will take some work. IMO the main challenge is to get wide buy in. Many researchers do not love the current system, but are mostly OK with it. They are not looking to topple it; if everyone is using a different option, they will switch, too, but they are not your passionaries to drive the change.
- stult 9y ago
  Well, that exists in the form of citations. The more a paper is cited the more valued are its authors. Just as the more often a site is linked to, the higher its page rank. And since you're not dealing with massive numbers of sites with lots of SEO experts, straight citation count is good enough without any of the corrections included in the page rank algorithm. Hiring committees look at both quantity of publications and citations to those publications. The problem is that quantity doesn't always indicate quality. Which is why journal prestige acts as a proxy measure.
  - amelius 9y ago
    > Well, that exists in the form of citations. Page-rank is "transitive" in the sense that it not only counts citations, but also citations-of-citations, etc. I wonder though if such a system can be easily gamed (see SEO), without damaging the reputation of offending authors. Also, we should perhaps have the concept of "negative citations". For example, when the text contains "In this paper we show that the approach of [..] is incorrect."
  - rocqua 9y ago
    I always though this was the impact factor of a journal. However, according to [1] that is not the case. [2] Mentions the 'eigenfactor' which seems to be closer to emulating page-rank. I haven't read much about it though. [1] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [2] http://researchguides.uic.edu/if/impact http://researchguides.uic.edu/if/impact
nonbel 9y ago
>"If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare)" You think stuff published in nature is better? I found the opposite, that Nature publishes papers with "sexy" results and poorly described methodology. Nature is one of the worst journals, I cringe when info I want is in a Nature article (although with supplements it is getting better).

by ptero 9y ago

lucasverra 9y ago

Would it a good use case for a blockchain validation system ? Decentralized (scientific) content reviewer ? Pay scarce resource (attention odf the well educated/trained scientist) with some new coin "SCI-coin" ?

bobcostas55 9y ago

>BS can still get through, but it is rare The data[0] suggests the exact opposite: the more prestigious the journal, the more bad science it attracts. [0] http://bjoern.brembs.net/2016/01/even-without-retractions-top-journals-publish-the-least-reliable-science/ http://bjoern.brembs.net/2016/01/even-without-retractions-to...

Ar-Curunir 9y ago
That's possible in a few exceptional cases, but at least in CS, you can see a very visible quality drop as you go from the top conferences to the second-tier conferences.
jtmcmc 9y ago
That was looking at only a very small part of the curve of journals. There is a long tail of really terrible journals publishing terrible science.

madez 9y ago

> Self-published, unknown journal gives me none of that. How can you know that? Edit: I don't know why I get the downvotes. My question might be naive, but its serious. I think it is not self-evident why an unkown self-published journal couldn't give you that. Elsevier was also once unknown. For me it sounds like a logical fallacy, but I wanted to know your take on it. It's pretty cheap to just downvote me for asking.

rocqua 9y ago
It's the opposite. You can't be sure that a self-published journal has editors and reviewers do a first check. This means you need to confirm and audit their practices. Compare this to a publication in nature. There, by virtue of the name, you know the article is worth more. (Note that in practice, Nature apparently has quite a few sham publications)
- madez 9y ago
  If we were talking about self-published articles, I'd agree, but we are not. We are talking about self-published journals. Under self-published journals I understand a journal brought out by the researchers or research organisatons. I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature.
  - HarryHirsch 9y ago
    I don't see why Germany's or even Europe's universities and research institutions couldn't publish a peer-reviewed and checked journal in collaboration, without a third entity, that can provide the quality of, say, nature. You mean a university press or a society journal? That model worked extremely well for > 100 years, until in the 1990s the societies sold their journals to Wiley, Springer and Elsevier, and somewhere around that time Springer took a wrong turn. You'd really like to ask why the selloff happened when it did.
    - madez 9y ago
      But then why again did I get downvoted?
      - upvotinglurker 9y ago
        I believe when they said "self-published, unknown" journals, the key word was unknown rather than self-published. They're not talking about journals self-published by established scholarly societies and so forth, who are "known." Nature (or other known journals) come with an established reputation (deserved or not) for publishing high quality work; an unknown journal (i.e. one with no established reputation one way or the other) by definition cannot provide this. In recent years, journals that claim to have a peer review process but which actually offer dubious or no review or quality control have proliferated (some are even published by companies like Elsevier). Having no established reputation doesn't prove that a journal is bad, but having an established reputation is a heuristic shortcut for evaluating whether it is good (at least in the "no one was ever fired for buying IBM" sense). Hiring committees use this heuristic to save time when filtering candidates.

amelius 9y ago

Perhaps we should use the equivalent of "page-rank" for scientific papers and their citations as a quality measure (?)

ptero 9y ago
Maybe, but it will take some work. IMO the main challenge is to get wide buy in. Many researchers do not love the current system, but are mostly OK with it. They are not looking to topple it; if everyone is using a different option, they will switch, too, but they are not your passionaries to drive the change.
stult 9y ago
Well, that exists in the form of citations. The more a paper is cited the more valued are its authors. Just as the more often a site is linked to, the higher its page rank. And since you're not dealing with massive numbers of sites with lots of SEO experts, straight citation count is good enough without any of the corrections included in the page rank algorithm. Hiring committees look at both quantity of publications and citations to those publications. The problem is that quantity doesn't always indicate quality. Which is why journal prestige acts as a proxy measure.
- amelius 9y ago
  > Well, that exists in the form of citations. Page-rank is "transitive" in the sense that it not only counts citations, but also citations-of-citations, etc. I wonder though if such a system can be easily gamed (see SEO), without damaging the reputation of offending authors. Also, we should perhaps have the concept of "negative citations". For example, when the text contains "In this paper we show that the approach of [..] is incorrect."
- rocqua 9y ago
  I always though this was the impact factor of a journal. However, according to [1] that is not the case. [2] Mentions the 'eigenfactor' which seems to be closer to emulating page-rank. I haven't read much about it though. [1] https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [2] http://researchguides.uic.edu/if/impact http://researchguides.uic.edu/if/impact

nonbel 9y ago

>"If I'm reading a paper in Nature I can assume that the editors and reviewers did first checks on content (BS can still get through, but it is rare)" You think stuff published in nature is better? I found the opposite, that Nature publishes papers with "sexy" results and poorly described methodology. Nature is one of the worst journals, I cringe when info I want is in a Nature article (although with supplements it is getting better).