11 ms·

Journals are used as a proxy for quality. A publication in "Science" or "Nature" is generally regarded as being better than a publication in, say, the "Proceedi

by dalke 11y ago

Journals are used as a proxy for quality. A publication in "Science" or "Nature" is generally regarded as being better than a publication in, say, the "Proceedings of the National Academy of Science". Bibliographic metrics, like the number of publications and where they are published, are use as a proxy for the quality of someone's research. It can affect grant funding and career advancement. Researchers can publish their own papers. But researchers also tend to want that work to be disseminated, both to spread knowledge and get recognition for the work. Journals help simplify that process. A field typically has one or two main journals, which most people in the field track. It's more likely that a publication will be noticed if it's published in one of those papers.

chris_wot 11y ago
If we were to work out what makes a journal so prestigious, what are the objective measures that should be considered?
- kowdermeister 11y ago
  100 years of track record publishing the least amount of junk.
  - chris_wot 11y ago
    Worst metric evar.
- parenthephobia 11y ago
  Mainly it's a chicken and egg situation. A journal is prestigious because the papers it publishes have an allusion of quality, because they are published by a prestigious journal. A prestigious journal publishes high-quality papers because it can attract high-quality authors and reviewers, because it is a prestigious journal. A journal is prestigious because researchers choose to (try to) be published in it, because their articles will be widely read by other researchers, because it is a prestigious journal. If you were trying to bootstrap a new journal, you'd have a tough time; unless you didn't care about its quality. The bare minimum would be a staff who have the time and skill to sift through obviously junk papers and then know which reviewers to use for the legitimate papers. Your best bet would probably be to assemble a staff of well-regarded researchers in whatever field(s) you wanted the journal to cover.
  - tomp 11y ago
    > If you were trying to bootstrap a new journal, you'd have a tough time; unless you didn't care about its quality. You could (should?) start by going through freely published (open access) papers, collecting the most quality ones, then publishing them in your journal. That's most of the value of journals, as well - informing the subscribers of noteworthy research.
    - parenthephobia 11y ago
      But how do I do that? I can interpret a paper and apply it to my problems, but I'm not qualified to tell if it's a good paper[1]. In my field -- computing -- research papers are often written in a language that only just makes sense to me. I'm sure it'd be easier if I was a computer scientist that regularly published, but that's why I suggest you'd need a team of actual scientists who are used to reading, writing, and reviewing papers. As Kernighan almost wrote: Reviewing the paper is twice as hard as reading the paper in the first place. Therefore, if you have to be as clever as possible to read the paper, you are, by definition, not smart enough to review it. [1] Although I can tell whether the code works, the quality of a paper is also about how well-written it is, whether the arguments are valid and supported by the data, and whether it provides something future research can be built on. A terrible paper that happens to contain code that works is useful for me, not so much for science in general.
      - dalke 11y ago
        I believe tomp's suggestion is that you can bootstrap as an overlay journal (see https://en.wikipedia.org/wiki/Overlay_journal https://en.wikipedia.org/wiki/Overlay_journal ) to build up that expertise. You don't need to have experienced people from the start, because they can learn while doing. At the start, no one is going to pay attention to the journal. Then as it gets known for being a good curator, people will start requesting to be reviewed for inclusion. Poof you've got a journal. You might even start with only yourself. And in 5 years if you want to distance yourself from your earlier, naïve efforts, then simply announce a name change. Here's the ugly truth - it's hard to tell if most papers are "good", and there are many types of "good".
Fuxy 11y ago
Haven't we solved the quality issue in other situations by giving users the ability to rate a scientific paper. What's wrong with giving other researchers the ability to rate and comment on a paper. It's a not a perfect filter but I would argue it works just as well a this journal reputation system and it prevents issues like people having to publish with them to get recognized not to mention it encourages discussion between academics which could lead to some breakthroughs as well. You want peer recognition so why not go to your peers directly instead of a proxy like these publications. This system is a product of a bygone era and as all other systems like it it is resisting change by relying on copyright. I see copyright as the root of most of our current societies issues it is just not compatible with how we work not to mention most of these papers were made with public funding so there shouldn’t be any copyright on it to begin with the copyright belongs to the public domain they payed for it. A publication just publishing it on their website and slapping a price on it is pure theft and they should be treated like the pirates they are pursuing.
- NotOscarWilde 11y ago
  I will comment on it from my point of view (theoretical CS PhD student): > What's wrong with giving other researchers the ability to rate and comment on a paper. I do not see nothing inherently bad with this (provided very good moderation tools are in place). However... * > I would argue it works just as well a this journal reputation system * As a journal reputation system, possibly; although I find that experts have often very different taste compared to say general Hacker News public -- for instance, machine learning has a very high "coolness factor" these days compared to other areas of computer science, but it would be unwise to make a system where a weaker paper on machine learning gets vastly more credit than a better paper on say approximation algorithms. However, an important part of the journal submission process is the fact that the editors ask the members of the academic community for spending several days on the paper, noting major and minor details and checking the validity of the proof. Since the reviewers do their work for free, there is no reason why there cannot be an open system that replicates the same process; still, just "adding a comment system to arXiv" will probably not be enough, as people spend rarely that much time on comments (15 minutes tops, not entire days).
nailer 11y ago
> A publication in "Science" or "Nature" What excludes the academic from also publishing elsewhere?
- deleted 11y ago
  [deleted]
- lrem 11y ago
  There is an expectation that each publication is a new piece of research and having the same work published in more than one journal and one conference is considered to be somewhat fraudulent. Just think of it: if researchers started to double publish, what would happen with the administrative "objective" process of giving the money to the guy with most bullet points on his list?
- deleted 11y ago
  [deleted]
- parenthephobia 11y ago
  The terms of the agreement with the journal. Nature, for example, permits private redistribution of accepted papers - I could send you a copy if you asked me for one - but requires exclusive publishing rights - I couldn't just put it on my website for anyone to take.
- deleted 11y ago
  [deleted]
- return0 11y ago
  Even if they could (and they can publish their preprints), scientists rarely bother. That's because they want to communicate their science to people who matter to them (e.g. grant giving bodies), not to the general public.
  - DarkContinent 11y ago
    Which raises the question of why we the taxpayers should be funding this research if it's not easily available for the public good.
raverbashing 11y ago
Pretty much this: Researcher: "I have to go to this club so people think I'm fancy" Club: "The club charges a lot so only fancy people go there" Other people: "This person is very fancy because they go to that club" End result is that people buy their "fanciness" by being on that club
comicjk 11y ago
I'd like to point out that a journal is not purely a proxy or even an aggregator - I've gotten more detailed peer review with the bigger journals, because the reviewers are more critical and the editors more selective. I didn't start out as a fan of peer review, but now I say the more the better; it has markedly increased the quality of all of my papers which have undergone it.
- DaveWalk 11y ago
  I really like your counter argument, focusing on the process instead of the output. However I have to ask: isn't the end goal of submitting a manuscript to get recognition for it? Do you genuinely hope every time that your submission will be mercilessly torn apart and possibly rejected instead of getting the recognition for the work you've done? Truly I have seen article reviews ruin a budding research scientist's career. Of course one could argue that other problems contribute to finding themselves in that position.
  - comicjk 11y ago
    If there is something worth tearing apart, I would much rather it be torn apart than published. Recognition for my work is not what scientific articles are for, right? When I read a paper that has unaddressed problems, it's a waste of time - possibly years of my time if I base my own work on that weak foundation. I don't want to inflict that on anyone. I would think that if peer review ruins a scientist's career, it would only be through a lack of support from the advisor. If the review is accurate the advisor should support the revisions, and if the review is totally mistaken the advisor should support resubmitting elsewhere.
- drewm1980 11y ago
  I think reviews tend to be overly critical primarily because reviewers are forced to find excuses to reject 9/10 of the papers that landed on their desk, most of which they weren't even interested in reading. I strongly believe that review could be almost exclusively through positive feedback, with crowd-sourced minimal annotations in the margins, i.e. "the idea in this paragraph came from <earlier paper link>", and "this idea is novel; wish I thought of it", and "there is a strong analogy between this paragraph and <some other tangentially related idea>"
Chathamization 11y ago
I'm confused by this. The researchers don't get paid for their paper, and it seems like the peer reviewers don't get paid for doing the peer review: http://www.nature.com/news/open-access-the-true-cost-of-science-publishing-1.12676 http://www.nature.com/news/open-access-the-true-cost-of-scie... So why exactly are publications locking these papers behind massively expensive paywalls when writing them and reviewing them are being done for free?
- bradleyjg 11y ago
  Who gets paid more: a tenure track professor doing research or an Associate Dean for Research? This is the same phenomenon.
- gautamdivgi 11y ago
  You confuse two things. The main career goal of a researcher in academia is generally moving up to the tenure track. The method to measure this progress is typically a)publication in journals with a high impact factor and b) Being on the technical review committee of said journals. The reason researchers publish in these journals is it gives them street cred. There are several open access journals - but scanty participation since their impact factor is unknown and does little to promote the researchers career. Speaking as someone who has published myself, the paywalls are highway robbery. But, imho - it is a problem only the researchers can solve by concertedly supporting open access journals and getting involved in them (maybe a yc research idea???). The established professors would need to take the lead on this - not the underlings. Personally - I've seen the "light" and stayed back in industry as opposed to moving into academia as I had hoped. The rat race for funding & publications & street cred seemed a lot worse there.
  - mmierz 11y ago
    Another important reason for slow adoption of open access is that most open access journals charge significant publication fees ($1000 or more.) Most academic labs run on shoestring budgets, so that's a lot of money. The principal investigator has to choose between standing on principle and publishing open access, or paying for equipment and reagents or sending students to conferences. That's not an easy choice to make, especially pre-tenure.

by dalke 11y ago

chris_wot 11y ago

If we were to work out what makes a journal so prestigious, what are the objective measures that should be considered?

kowdermeister 11y ago
100 years of track record publishing the least amount of junk.
- chris_wot 11y ago
  Worst metric evar.
parenthephobia 11y ago
Mainly it's a chicken and egg situation. A journal is prestigious because the papers it publishes have an allusion of quality, because they are published by a prestigious journal. A prestigious journal publishes high-quality papers because it can attract high-quality authors and reviewers, because it is a prestigious journal. A journal is prestigious because researchers choose to (try to) be published in it, because their articles will be widely read by other researchers, because it is a prestigious journal. If you were trying to bootstrap a new journal, you'd have a tough time; unless you didn't care about its quality. The bare minimum would be a staff who have the time and skill to sift through obviously junk papers and then know which reviewers to use for the legitimate papers. Your best bet would probably be to assemble a staff of well-regarded researchers in whatever field(s) you wanted the journal to cover.
- tomp 11y ago
  > If you were trying to bootstrap a new journal, you'd have a tough time; unless you didn't care about its quality. You could (should?) start by going through freely published (open access) papers, collecting the most quality ones, then publishing them in your journal. That's most of the value of journals, as well - informing the subscribers of noteworthy research.
  - parenthephobia 11y ago
    But how do I do that? I can interpret a paper and apply it to my problems, but I'm not qualified to tell if it's a good paper[1]. In my field -- computing -- research papers are often written in a language that only just makes sense to me. I'm sure it'd be easier if I was a computer scientist that regularly published, but that's why I suggest you'd need a team of actual scientists who are used to reading, writing, and reviewing papers. As Kernighan almost wrote: Reviewing the paper is twice as hard as reading the paper in the first place. Therefore, if you have to be as clever as possible to read the paper, you are, by definition, not smart enough to review it. [1] Although I can tell whether the code works, the quality of a paper is also about how well-written it is, whether the arguments are valid and supported by the data, and whether it provides something future research can be built on. A terrible paper that happens to contain code that works is useful for me, not so much for science in general.
    - dalke 11y ago
      I believe tomp's suggestion is that you can bootstrap as an overlay journal (see https://en.wikipedia.org/wiki/Overlay_journal https://en.wikipedia.org/wiki/Overlay_journal ) to build up that expertise. You don't need to have experienced people from the start, because they can learn while doing. At the start, no one is going to pay attention to the journal. Then as it gets known for being a good curator, people will start requesting to be reviewed for inclusion. Poof you've got a journal. You might even start with only yourself. And in 5 years if you want to distance yourself from your earlier, naïve efforts, then simply announce a name change. Here's the ugly truth - it's hard to tell if most papers are "good", and there are many types of "good".

Fuxy 11y ago

Haven't we solved the quality issue in other situations by giving users the ability to rate a scientific paper. What's wrong with giving other researchers the ability to rate and comment on a paper. It's a not a perfect filter but I would argue it works just as well a this journal reputation system and it prevents issues like people having to publish with them to get recognized not to mention it encourages discussion between academics which could lead to some breakthroughs as well. You want peer recognition so why not go to your peers directly instead of a proxy like these publications. This system is a product of a bygone era and as all other systems like it it is resisting change by relying on copyright. I see copyright as the root of most of our current societies issues it is just not compatible with how we work not to mention most of these papers were made with public funding so there shouldn’t be any copyright on it to begin with the copyright belongs to the public domain they payed for it. A publication just publishing it on their website and slapping a price on it is pure theft and they should be treated like the pirates they are pursuing.

NotOscarWilde 11y ago
I will comment on it from my point of view (theoretical CS PhD student): > What's wrong with giving other researchers the ability to rate and comment on a paper. I do not see nothing inherently bad with this (provided very good moderation tools are in place). However... * > I would argue it works just as well a this journal reputation system * As a journal reputation system, possibly; although I find that experts have often very different taste compared to say general Hacker News public -- for instance, machine learning has a very high "coolness factor" these days compared to other areas of computer science, but it would be unwise to make a system where a weaker paper on machine learning gets vastly more credit than a better paper on say approximation algorithms. However, an important part of the journal submission process is the fact that the editors ask the members of the academic community for spending several days on the paper, noting major and minor details and checking the validity of the proof. Since the reviewers do their work for free, there is no reason why there cannot be an open system that replicates the same process; still, just "adding a comment system to arXiv" will probably not be enough, as people spend rarely that much time on comments (15 minutes tops, not entire days).

nailer 11y ago

> A publication in "Science" or "Nature" What excludes the academic from also publishing elsewhere?

deleted 11y ago
[deleted]
lrem 11y ago
There is an expectation that each publication is a new piece of research and having the same work published in more than one journal and one conference is considered to be somewhat fraudulent. Just think of it: if researchers started to double publish, what would happen with the administrative "objective" process of giving the money to the guy with most bullet points on his list?
deleted 11y ago
[deleted]
parenthephobia 11y ago
The terms of the agreement with the journal. Nature, for example, permits private redistribution of accepted papers - I could send you a copy if you asked me for one - but requires exclusive publishing rights - I couldn't just put it on my website for anyone to take.
deleted 11y ago
[deleted]
return0 11y ago
Even if they could (and they can publish their preprints), scientists rarely bother. That's because they want to communicate their science to people who matter to them (e.g. grant giving bodies), not to the general public.
- DarkContinent 11y ago
  Which raises the question of why we the taxpayers should be funding this research if it's not easily available for the public good.

raverbashing 11y ago

Pretty much this: Researcher: "I have to go to this club so people think I'm fancy" Club: "The club charges a lot so only fancy people go there" Other people: "This person is very fancy because they go to that club" End result is that people buy their "fanciness" by being on that club

comicjk 11y ago

I'd like to point out that a journal is not purely a proxy or even an aggregator - I've gotten more detailed peer review with the bigger journals, because the reviewers are more critical and the editors more selective. I didn't start out as a fan of peer review, but now I say the more the better; it has markedly increased the quality of all of my papers which have undergone it.

DaveWalk 11y ago
I really like your counter argument, focusing on the process instead of the output. However I have to ask: isn't the end goal of submitting a manuscript to get recognition for it? Do you genuinely hope every time that your submission will be mercilessly torn apart and possibly rejected instead of getting the recognition for the work you've done? Truly I have seen article reviews ruin a budding research scientist's career. Of course one could argue that other problems contribute to finding themselves in that position.
- comicjk 11y ago
  If there is something worth tearing apart, I would much rather it be torn apart than published. Recognition for my work is not what scientific articles are for, right? When I read a paper that has unaddressed problems, it's a waste of time - possibly years of my time if I base my own work on that weak foundation. I don't want to inflict that on anyone. I would think that if peer review ruins a scientist's career, it would only be through a lack of support from the advisor. If the review is accurate the advisor should support the revisions, and if the review is totally mistaken the advisor should support resubmitting elsewhere.
drewm1980 11y ago
I think reviews tend to be overly critical primarily because reviewers are forced to find excuses to reject 9/10 of the papers that landed on their desk, most of which they weren't even interested in reading. I strongly believe that review could be almost exclusively through positive feedback, with crowd-sourced minimal annotations in the margins, i.e. "the idea in this paragraph came from <earlier paper link>", and "this idea is novel; wish I thought of it", and "there is a strong analogy between this paragraph and <some other tangentially related idea>"

Chathamization 11y ago

I'm confused by this. The researchers don't get paid for their paper, and it seems like the peer reviewers don't get paid for doing the peer review: http://www.nature.com/news/open-access-the-true-cost-of-science-publishing-1.12676 http://www.nature.com/news/open-access-the-true-cost-of-scie... So why exactly are publications locking these papers behind massively expensive paywalls when writing them and reviewing them are being done for free?

bradleyjg 11y ago
Who gets paid more: a tenure track professor doing research or an Associate Dean for Research? This is the same phenomenon.
gautamdivgi 11y ago
You confuse two things. The main career goal of a researcher in academia is generally moving up to the tenure track. The method to measure this progress is typically a)publication in journals with a high impact factor and b) Being on the technical review committee of said journals. The reason researchers publish in these journals is it gives them street cred. There are several open access journals - but scanty participation since their impact factor is unknown and does little to promote the researchers career. Speaking as someone who has published myself, the paywalls are highway robbery. But, imho - it is a problem only the researchers can solve by concertedly supporting open access journals and getting involved in them (maybe a yc research idea???). The established professors would need to take the lead on this - not the underlings. Personally - I've seen the "light" and stayed back in industry as opposed to moving into academia as I had hoped. The rat race for funding & publications & street cred seemed a lot worse there.
- mmierz 11y ago
  Another important reason for slow adoption of open access is that most open access journals charge significant publication fees ($1000 or more.) Most academic labs run on shoestring budgets, so that's a lot of money. The principal investigator has to choose between standing on principle and publishing open access, or paying for equipment and reagents or sending students to conferences. That's not an easy choice to make, especially pre-tenure.