4 ms·

While I'm in favor of opening up journals and possibly putting out some nonprofit alternatives, and while I recognize that the peer review process is definitely

While I'm in favor of opening up journals and possibly putting out some nonprofit alternatives, and while I recognize that the peer review process is definitely not without its problems, be careful not to discount it completely. I cringe when I hear phrases like "value to a community" and "collective opinion" used to refer to scientific articles. The average reader is not qualified to judge the merits of an article, hence the peer review process. Some great examples: the age of the earth, evolution, global climate change; three areas where public opinion is divided but experts overwhelmingly agree. The value of science is based on its accuracy (which is best determined by experts) not its appeal to the masses.

hotpockets 15y ago
I was assuming that the article was suggesting a system that is similar to reddit, but with important differences. For instance, requiring real names and public voting records. Or, you could reduce a person's vote importance if they are voting on an article outside their area of expertise. There are probably other things you could do to.
- radarsat1 15y ago
  > Or, you could reduce a person's vote importance if they are voting on an article outside their area of expertise. I see an immediate problem there. There are magnitudes more people _outside_ any given area of expertise than inside. Even small votes add up.
  - juiceandjuice 15y ago
    Ladies and gentleman, I present to you the logarithmic scale.
- rlpb 15y ago
  Apart from the lack of anonymity, I think that the ideals of such a system are very similar to the ideas of any social voting system (such as Stack Overflow, Slashdot, reddit, HN). The goal is to simply have a rating system that works, right? The trouble is that we still haven't managed to do this well in the technology world. And some of us believe that we need to go back to some level of curation. I'd love to see a system as described that actually works. But until we (the tech world) manage to figure out a system that works, I'm not convinced that scientific publishing will fare any differently.
  - Joakal 15y ago
    The technology world already has it; We have Wikipedia (Edit history), Advogato (Paths), StackOverflow (Rep), etc that encourage personal profiles. You are more valued as a loyal member than one that just started. Unlike Reddit, Slashdot and HN that do not encourage personal profiles. Do you even remember my name? What about other HNers besides pg? Why do you read their posts anyway? They work because it's a form of almost pure democracy being applied and have the same problems of it eg tyranny of the majority. What the scientific model can do is a multitude of above; institution API confirming credentials (Referral), history (paths, did this person do other reviews?), etc. Cheap and still effective peer-review. I'm not sure which kind of government type applies to my suggestion above, but there are many forms of government they can apply. More: https://secure.wikimedia.org/wikipedia/en/wiki/List_of_forms_of_government https://secure.wikimedia.org/wikipedia/en/wiki/List_of_forms...
- rflrob 15y ago
  Even cooler would be to have each person (explicitly or implicitly) set other people's vote importance. I can imagine a complicated system wherein voting cliques are detected, and used to feed back into the rankings of what papers show up on your page. Ideally, the voting could be more than just "+1/-1" as well, potentially with a couple different axes, so that I could, for instance, say that I think a paper is important to the field, but likely wrong. Just because a paper is controversial doesn't mean I don't want it to be discussed (for instance, if it introduces a new computational technique, but the input data is flawed somehow).
  - wickedchicken 15y ago
    You're matching an engineering solution to a social problem. Trying to map a paper's social significance to an abstract number is really hard, and in the process you lose a lot of ability to present papers based on taste. Showing me a paper because "these numbers are high and you tend to like these kinds of numbers" is fundamentally different than "you should see this because it is groundbreaking." On the other hand, if we can create a metric for elegance and another for "brings the reader closer to enlightenment," then I'm all for it.
  - chalst 15y ago
    The property you describe is attack resistance of trust metrics. http://usenix.com/publications/library/proceedings/sec98/full_papers/levien/levien.pdf http://usenix.com/publications/library/proceedings/sec98/ful...
- Steko 15y ago
  All these systems are incredibly vulnerable to capture. Despite their many drawbacks, the best thing peer reviewed journals have going for them is their relative independence.
econgeeker 15y ago
You cringe when you hear phrases like "collective opinion" regarding science, because science is supposed to be objective, correct? Further, I suppose you recognize that science is not subject to the will of the majority, but to the scientific process, right? Yet, when it comes to scientific papers, you want to see them subject to the filtering of "experts" to assure they are politically correct? During my time working in a national lab I had several papers published. Around me, there were over 100 Phds, all of whom were actively publishing papers, and we worked together on them, at least to review them. We were investigating an area of physics regarding the electrical properties of some rather exotic materials. One thing about the peer review process is that it is very highly political. Often, perfectly legitimate papers would be refused for publication because they disagreed with a scientific view of one of the peers. Sometimes we'd have to submit the paper to another journal. But at least this level of "politics" was scientific politics. I don't like that some Professor Emeritus is holding to an older view, as that forces us to make our argument a bit stronger, or to do a bit more experimentation to resolve his concerns. But, sometimes, there was just no way we were getting published in that journal. This is completely subjective. Sometimes our case was really not strong enough, but often, it was simply our position was too modern. And that was the state of affairs (at that time, in that area) without any undue government influence. I believe the earth is very old and evolution is an observable phenomena. But I don't see how any scientist who also holds this view is threatened by articles proposing the alternative. Some creationist makes a publication claiming that evolution is wrong-- should be amusing! On "global climate change" there is no scientific consensus. The idea that there is a "consensus" is a political talking point that lets people who don't understand the science short circuit debate by claiming that the scientific community has reached a conclusion. Worse, given the political nature of that position, government is starting to influence the actual publication of papers to make sure they comply with the AGW political correctness filter. I shudder to think the road this country is going down if political correctness remains in the review system. However, I recognize the right of journals to choose their publication standards as they wish. This is also why I oppose any sort of government "blessing" or requirements for journals. Having made "peer reviewed" a phrase that is code for "legitimate", I don't want to see the politically correct crowd start dictating which panels of peers are "legitimate" and which ones are not. If it is time for a new journal to cover a new branch, then let a new journal form. This is how many of them got started in the past-- people unable to find publication elsewhere, for reasons such as topic, length, nature of the research, formed new journals. PS- FWIW, AGW is really easily disproven if you are familiar with the basic science. The IR absorption of CO2 is below that of water vapor, the CO2 levels in the past have been higher than the "tipping point", etc. AGW, is really a political, not a scientific, movement. This is what is so insidious about the idea that "All scientists agree"- it makes proponents of this theory feel like they don't have to defend their position, and that anyone who disagrees is "rejecting science", when it is the reverse. "collective opinion" is not how science works. Yet, I've never met an AGW proponent who actually understood the AGW theory, and the science behind it, well enough to debate the issue, but I've gotten a lot of hate for failing to regurgitate the party line. I half expect to get banned from this forum for taking this minority view, in fact.

hotpockets 15y ago

I was assuming that the article was suggesting a system that is similar to reddit, but with important differences. For instance, requiring real names and public voting records. Or, you could reduce a person's vote importance if they are voting on an article outside their area of expertise. There are probably other things you could do to.

radarsat1 15y ago
> Or, you could reduce a person's vote importance if they are voting on an article outside their area of expertise. I see an immediate problem there. There are magnitudes more people _outside_ any given area of expertise than inside. Even small votes add up.
- juiceandjuice 15y ago
  Ladies and gentleman, I present to you the logarithmic scale.
rlpb 15y ago
Apart from the lack of anonymity, I think that the ideals of such a system are very similar to the ideas of any social voting system (such as Stack Overflow, Slashdot, reddit, HN). The goal is to simply have a rating system that works, right? The trouble is that we still haven't managed to do this well in the technology world. And some of us believe that we need to go back to some level of curation. I'd love to see a system as described that actually works. But until we (the tech world) manage to figure out a system that works, I'm not convinced that scientific publishing will fare any differently.
- Joakal 15y ago
  The technology world already has it; We have Wikipedia (Edit history), Advogato (Paths), StackOverflow (Rep), etc that encourage personal profiles. You are more valued as a loyal member than one that just started. Unlike Reddit, Slashdot and HN that do not encourage personal profiles. Do you even remember my name? What about other HNers besides pg? Why do you read their posts anyway? They work because it's a form of almost pure democracy being applied and have the same problems of it eg tyranny of the majority. What the scientific model can do is a multitude of above; institution API confirming credentials (Referral), history (paths, did this person do other reviews?), etc. Cheap and still effective peer-review. I'm not sure which kind of government type applies to my suggestion above, but there are many forms of government they can apply. More: https://secure.wikimedia.org/wikipedia/en/wiki/List_of_forms_of_government https://secure.wikimedia.org/wikipedia/en/wiki/List_of_forms...
rflrob 15y ago
Even cooler would be to have each person (explicitly or implicitly) set other people's vote importance. I can imagine a complicated system wherein voting cliques are detected, and used to feed back into the rankings of what papers show up on your page. Ideally, the voting could be more than just "+1/-1" as well, potentially with a couple different axes, so that I could, for instance, say that I think a paper is important to the field, but likely wrong. Just because a paper is controversial doesn't mean I don't want it to be discussed (for instance, if it introduces a new computational technique, but the input data is flawed somehow).
- wickedchicken 15y ago
  You're matching an engineering solution to a social problem. Trying to map a paper's social significance to an abstract number is really hard, and in the process you lose a lot of ability to present papers based on taste. Showing me a paper because "these numbers are high and you tend to like these kinds of numbers" is fundamentally different than "you should see this because it is groundbreaking." On the other hand, if we can create a metric for elegance and another for "brings the reader closer to enlightenment," then I'm all for it.
- chalst 15y ago
  The property you describe is attack resistance of trust metrics. http://usenix.com/publications/library/proceedings/sec98/full_papers/levien/levien.pdf http://usenix.com/publications/library/proceedings/sec98/ful...
Steko 15y ago
All these systems are incredibly vulnerable to capture. Despite their many drawbacks, the best thing peer reviewed journals have going for them is their relative independence.

econgeeker 15y ago

You cringe when you hear phrases like "collective opinion" regarding science, because science is supposed to be objective, correct? Further, I suppose you recognize that science is not subject to the will of the majority, but to the scientific process, right? Yet, when it comes to scientific papers, you want to see them subject to the filtering of "experts" to assure they are politically correct? During my time working in a national lab I had several papers published. Around me, there were over 100 Phds, all of whom were actively publishing papers, and we worked together on them, at least to review them. We were investigating an area of physics regarding the electrical properties of some rather exotic materials. One thing about the peer review process is that it is very highly political. Often, perfectly legitimate papers would be refused for publication because they disagreed with a scientific view of one of the peers. Sometimes we'd have to submit the paper to another journal. But at least this level of "politics" was scientific politics. I don't like that some Professor Emeritus is holding to an older view, as that forces us to make our argument a bit stronger, or to do a bit more experimentation to resolve his concerns. But, sometimes, there was just no way we were getting published in that journal. This is completely subjective. Sometimes our case was really not strong enough, but often, it was simply our position was too modern. And that was the state of affairs (at that time, in that area) without any undue government influence. I believe the earth is very old and evolution is an observable phenomena. But I don't see how any scientist who also holds this view is threatened by articles proposing the alternative. Some creationist makes a publication claiming that evolution is wrong-- should be amusing! On "global climate change" there is no scientific consensus. The idea that there is a "consensus" is a political talking point that lets people who don't understand the science short circuit debate by claiming that the scientific community has reached a conclusion. Worse, given the political nature of that position, government is starting to influence the actual publication of papers to make sure they comply with the AGW political correctness filter. I shudder to think the road this country is going down if political correctness remains in the review system. However, I recognize the right of journals to choose their publication standards as they wish. This is also why I oppose any sort of government "blessing" or requirements for journals. Having made "peer reviewed" a phrase that is code for "legitimate", I don't want to see the politically correct crowd start dictating which panels of peers are "legitimate" and which ones are not. If it is time for a new journal to cover a new branch, then let a new journal form. This is how many of them got started in the past-- people unable to find publication elsewhere, for reasons such as topic, length, nature of the research, formed new journals. PS- FWIW, AGW is really easily disproven if you are familiar with the basic science. The IR absorption of CO2 is below that of water vapor, the CO2 levels in the past have been higher than the "tipping point", etc. AGW, is really a political, not a scientific, movement. This is what is so insidious about the idea that "All scientists agree"- it makes proponents of this theory feel like they don't have to defend their position, and that anyone who disagrees is "rejecting science", when it is the reverse. "collective opinion" is not how science works. Yet, I've never met an AGW proponent who actually understood the AGW theory, and the science behind it, well enough to debate the issue, but I've gotten a lot of hate for failing to regurgitate the party line. I half expect to get banned from this forum for taking this minority view, in fact.