5 ms·

Haven't we solved the quality issue in other situations by giving users the ability to rate a scientific paper. What's wrong with giving other researchers the

by Fuxy 10y ago

Haven't we solved the quality issue in other situations by giving users the ability to rate a scientific paper. What's wrong with giving other researchers the ability to rate and comment on a paper. It's a not a perfect filter but I would argue it works just as well a this journal reputation system and it prevents issues like people having to publish with them to get recognized not to mention it encourages discussion between academics which could lead to some breakthroughs as well. You want peer recognition so why not go to your peers directly instead of a proxy like these publications. This system is a product of a bygone era and as all other systems like it it is resisting change by relying on copyright. I see copyright as the root of most of our current societies issues it is just not compatible with how we work not to mention most of these papers were made with public funding so there shouldn’t be any copyright on it to begin with the copyright belongs to the public domain they payed for it. A publication just publishing it on their website and slapping a price on it is pure theft and they should be treated like the pirates they are pursuing.

NotOscarWilde 10y ago
I will comment on it from my point of view (theoretical CS PhD student): > What's wrong with giving other researchers the ability to rate and comment on a paper. I do not see nothing inherently bad with this (provided very good moderation tools are in place). However... * > I would argue it works just as well a this journal reputation system * As a journal reputation system, possibly; although I find that experts have often very different taste compared to say general Hacker News public -- for instance, machine learning has a very high "coolness factor" these days compared to other areas of computer science, but it would be unwise to make a system where a weaker paper on machine learning gets vastly more credit than a better paper on say approximation algorithms. However, an important part of the journal submission process is the fact that the editors ask the members of the academic community for spending several days on the paper, noting major and minor details and checking the validity of the proof. Since the reviewers do their work for free, there is no reason why there cannot be an open system that replicates the same process; still, just "adding a comment system to arXiv" will probably not be enough, as people spend rarely that much time on comments (15 minutes tops, not entire days).
- codedokode 10y ago
  Why do reviewers work for free? And why scientists agree to give away their work for free?
  - richard_mcp 10y ago
    Reviewers work for free for a few reasons reasons. First, it's prestigious to be a reviewer for a top journal and very beneficial for getting tenure. Second, it allows you to create connections with other top researchers. Third, it's giving back to the community by doing a service that needs to be done.
    - dalke 10y ago
      How do people know that I have (or haven't) been a reviewer for a top journal? Do they have a public list of reviewers? Of the people I meet at a conference, how do I figure out who has been a reviewer for Nature, and how do I verify that claim?
      - mbrundle 10y ago
        My thoughts exactly. I've never known a journal to do anything to promote the fact that someone has reviewed for them (eg produce a letter on request that states which articles someone has reviewed for them). It'd be a nice and free way of creating value to reviewers who generally work for free, but I don't know any journal who's ever done this.
  - chimprich 10y ago
    Reviewers don't really work for free; it's generally accepted that reviewing is part of their salaried job.
    - matwood 10y ago
      For free in the sense that the journals pay them nothing. So lets recap; the public/university pays for the researcher and the research, the public/university pays for the reviewing and submission process, and finally the public university pays to access the work they already paid to create and review. And people wonder why everyone but the journals think the journal process is broken.
notahacker 10y ago
A rating system would generally become an indicator of other academics' level of agreement with a paper, rather than an indication that it passed a quality and significance threshold. Citation ranking schemes already exist, but the citation counts are heavily influenced by academics being unable to NOT mention a dissenting view because it was recently published in a prominent journal. And I'm not sure examples like Amazon book ratings, HN karma or professionally-optimized PageRank can really be claimed with a straight face to be superior for curating quality content than good old fashioned peer review. I do think open commenting systems would be a boon to academic discourse though.
- Chinjut 10y ago
  Does pre-publication "peer review" not already function to some extent similarly as an indicator of level of agreement rather than a disinterested, objective quality and significance threshold? Everything I know of human processes suggests that it would.
  - slavik81 10y ago
    I worry that a rating system would reduce the amount of thought each reviewer puts into their decision, but it's certainly true that peer review isn't perfect. Famously, the Fourier transform was not published for several years due to a disagreement with Lagrange. > When Fourier submitted his paper in 1807, the committee (which included Lagrange, Laplace, Malus and Legendre, among others) concluded: ...the manner in which the author arrives at these equations is not exempt of difficulties and [...] his analysis to integrate them still leaves something to be desired on the score of generality and even rigour. https://en.wikipedia.org/wiki/List_of_important_publications_in_mathematics#M.C3.A9moire_sur_la_propagation_de_la_chaleur_dans_les_corps_solides https://en.wikipedia.org/wiki/List_of_important_publications...
dalke 10y ago
"by giving users the ability to rate a scientific paper." LOL! No. Ratings are nearly useless. By which I mean I don't know how I would incorporate ratings into my own research. How do you keep a ratings system from being a proxy for popularity, rather than a measure of novelty or utility? How do you keep it from being easily gamed? Does a "5.2" for a 1995 paper have the same meaning as a "5.2" for a 2010 paper? "You want peer recognition so why not go to your peers directly instead of a proxy like these publications" Do tell me how. Post on my blog and hope people come across it? Journals outsource marketing. Get rid of journals and something must replace that service, or I have to become a marketer. "This system is a product of a bygone era .. relying on copyright" You are mistaken. The system does not rely on copyright. The scientific publication system dates from the 1600s, dated from Oldenburg's 'Philosophical Transactions of the Royal Society'. Copyright dates from the 1700s, dated from the Statute of Anne. The scientific publication system depends on reliable selection and curation. Its current form is dominated by large publishing companies which use copyright law for revenue generation, but that's only been true for the last few decades. Scientific publication also existed in the USSR even without copyright protection. (article 103(2) according to https://en.wikipedia.org/wiki/Copyright_law_of_the_Soviet_Union https://en.wikipedia.org/wiki/Copyright_law_of_the_Soviet_Un... gave "permission to reproduce published scientific, artistic, or literary works as excerpts (or even entirely) in scientific, critical, or educational publications"). "most of these papers were made with public funding so there shouldn’t be any copyright on it to begin with the copyright belongs to the public domain they payed for it." I understand and sympathize with your argument. But in practice things aren't so clear cut. Rarely does the money come 100% from public sources. Is your argument that even 1 cent of public funding means the entire work product must be in the public domain? Are you making the moral argument that copyright should to the funding source? Or is it limited only to public funding sources? For example, if 100% of the money for a project comes from the Bill & Melinda Gates Foundation, then do you want the foundation to control the copyright the work product? What if it's 80% Gates and 20% public funded? Is it public domain then? If 3 years of research was publicly funded, and resulted in paper, and 2 more years was privately funded by BigCorp., to analyze the data further, then must that additional two years also be in the public domain? And on and on. Unless "1 cent => public domain always", then there's no clear line. Is that your argument? (BTW, there is another clear line - work by a US government employee is in the public domain. A small percentage of the scientific papers are in the public domain because of that.)
matt4077 10y ago
It's hard to explain how important the journals are for a scientists. It's not even the effect that a paper in Nature is (in some disciplines) the fastest way to tenure. It's got an even bigger psychological factor: If you get into Science, you frame it and hang it on the most prominent wall in your living room or office. With a Nobel too rare to practically hope for, this system is the only reward all those mice-conditioning mice are conditioned to.

by Fuxy 10y ago

NotOscarWilde 10y ago

I will comment on it from my point of view (theoretical CS PhD student): > What's wrong with giving other researchers the ability to rate and comment on a paper. I do not see nothing inherently bad with this (provided very good moderation tools are in place). However... * > I would argue it works just as well a this journal reputation system * As a journal reputation system, possibly; although I find that experts have often very different taste compared to say general Hacker News public -- for instance, machine learning has a very high "coolness factor" these days compared to other areas of computer science, but it would be unwise to make a system where a weaker paper on machine learning gets vastly more credit than a better paper on say approximation algorithms. However, an important part of the journal submission process is the fact that the editors ask the members of the academic community for spending several days on the paper, noting major and minor details and checking the validity of the proof. Since the reviewers do their work for free, there is no reason why there cannot be an open system that replicates the same process; still, just "adding a comment system to arXiv" will probably not be enough, as people spend rarely that much time on comments (15 minutes tops, not entire days).

codedokode 10y ago
Why do reviewers work for free? And why scientists agree to give away their work for free?
- richard_mcp 10y ago
  Reviewers work for free for a few reasons reasons. First, it's prestigious to be a reviewer for a top journal and very beneficial for getting tenure. Second, it allows you to create connections with other top researchers. Third, it's giving back to the community by doing a service that needs to be done.
  - dalke 10y ago
    How do people know that I have (or haven't) been a reviewer for a top journal? Do they have a public list of reviewers? Of the people I meet at a conference, how do I figure out who has been a reviewer for Nature, and how do I verify that claim?
    - mbrundle 10y ago
      My thoughts exactly. I've never known a journal to do anything to promote the fact that someone has reviewed for them (eg produce a letter on request that states which articles someone has reviewed for them). It'd be a nice and free way of creating value to reviewers who generally work for free, but I don't know any journal who's ever done this.
- chimprich 10y ago
  Reviewers don't really work for free; it's generally accepted that reviewing is part of their salaried job.
  - matwood 10y ago
    For free in the sense that the journals pay them nothing. So lets recap; the public/university pays for the researcher and the research, the public/university pays for the reviewing and submission process, and finally the public university pays to access the work they already paid to create and review. And people wonder why everyone but the journals think the journal process is broken.

notahacker 10y ago

A rating system would generally become an indicator of other academics' level of agreement with a paper, rather than an indication that it passed a quality and significance threshold. Citation ranking schemes already exist, but the citation counts are heavily influenced by academics being unable to NOT mention a dissenting view because it was recently published in a prominent journal. And I'm not sure examples like Amazon book ratings, HN karma or professionally-optimized PageRank can really be claimed with a straight face to be superior for curating quality content than good old fashioned peer review. I do think open commenting systems would be a boon to academic discourse though.

Chinjut 10y ago
Does pre-publication "peer review" not already function to some extent similarly as an indicator of level of agreement rather than a disinterested, objective quality and significance threshold? Everything I know of human processes suggests that it would.
- slavik81 10y ago
  I worry that a rating system would reduce the amount of thought each reviewer puts into their decision, but it's certainly true that peer review isn't perfect. Famously, the Fourier transform was not published for several years due to a disagreement with Lagrange. > When Fourier submitted his paper in 1807, the committee (which included Lagrange, Laplace, Malus and Legendre, among others) concluded: ...the manner in which the author arrives at these equations is not exempt of difficulties and [...] his analysis to integrate them still leaves something to be desired on the score of generality and even rigour. https://en.wikipedia.org/wiki/List_of_important_publications_in_mathematics#M.C3.A9moire_sur_la_propagation_de_la_chaleur_dans_les_corps_solides https://en.wikipedia.org/wiki/List_of_important_publications...

dalke 10y ago

"by giving users the ability to rate a scientific paper." LOL! No. Ratings are nearly useless. By which I mean I don't know how I would incorporate ratings into my own research. How do you keep a ratings system from being a proxy for popularity, rather than a measure of novelty or utility? How do you keep it from being easily gamed? Does a "5.2" for a 1995 paper have the same meaning as a "5.2" for a 2010 paper? "You want peer recognition so why not go to your peers directly instead of a proxy like these publications" Do tell me how. Post on my blog and hope people come across it? Journals outsource marketing. Get rid of journals and something must replace that service, or I have to become a marketer. "This system is a product of a bygone era .. relying on copyright" You are mistaken. The system does not rely on copyright. The scientific publication system dates from the 1600s, dated from Oldenburg's 'Philosophical Transactions of the Royal Society'. Copyright dates from the 1700s, dated from the Statute of Anne. The scientific publication system depends on reliable selection and curation. Its current form is dominated by large publishing companies which use copyright law for revenue generation, but that's only been true for the last few decades. Scientific publication also existed in the USSR even without copyright protection. (article 103(2) according to https://en.wikipedia.org/wiki/Copyright_law_of_the_Soviet_Union https://en.wikipedia.org/wiki/Copyright_law_of_the_Soviet_Un... gave "permission to reproduce published scientific, artistic, or literary works as excerpts (or even entirely) in scientific, critical, or educational publications"). "most of these papers were made with public funding so there shouldn’t be any copyright on it to begin with the copyright belongs to the public domain they payed for it." I understand and sympathize with your argument. But in practice things aren't so clear cut. Rarely does the money come 100% from public sources. Is your argument that even 1 cent of public funding means the entire work product must be in the public domain? Are you making the moral argument that copyright should to the funding source? Or is it limited only to public funding sources? For example, if 100% of the money for a project comes from the Bill & Melinda Gates Foundation, then do you want the foundation to control the copyright the work product? What if it's 80% Gates and 20% public funded? Is it public domain then? If 3 years of research was publicly funded, and resulted in paper, and 2 more years was privately funded by BigCorp., to analyze the data further, then must that additional two years also be in the public domain? And on and on. Unless "1 cent => public domain always", then there's no clear line. Is that your argument? (BTW, there is another clear line - work by a US government employee is in the public domain. A small percentage of the scientific papers are in the public domain because of that.)

matt4077 10y ago

It's hard to explain how important the journals are for a scientists. It's not even the effect that a paper in Nature is (in some disciplines) the fastest way to tenure. It's got an even bigger psychological factor: If you get into Science, you frame it and hang it on the most prominent wall in your living room or office. With a Nobel too rare to practically hope for, this system is the only reward all those mice-conditioning mice are conditioned to.