5 ms·

Okay, I just read http://breast-cancer-research.com/content/12/S4/S13 http://breast-cancer-research.com/content/12/S4/S13 Yup, again, his ideas are wacko and

Okay, I just read http://breast-cancer-research.com/content/12/S4/S13 http://breast-cancer-research.com/content/12/S4/S13 Yup, again, his ideas are wacko and won't work. Again, the main problem with just putting the papers on-line and letting people 'review' them then is that mostly the papers won't get read nearly as carefully as with the present peer-review process; or, even if some paper does get read carefully, other readers mostly won't have any very good way to know this. Uh, publish several well regarded papers in specialty S, and then maybe get invited to review some papers in specialty S. Do well reviewing and impress some journals in specialty S, and maybe get invited to be an editor of a journal in specialty S. Do well as an editor, and when Elsevier, etc., starts a new journal in specialty S, maybe get invited to the the editor in chief. That takes ballpark 20 years. Net, the present paper review system has a severe 'promotion' mechanism that is an enormous aid to quality the journal readers can value. That is, the readers have a good way to know about the quality of the reviewing. With the proposal of this thread, even if a paper got a careful review, the readers would not know this. That is, the proposal throws out the baby with the bathwater. As I described, reading research papers and judging if they are new, correct, and significant is TOUGH work. The present process makes a good effort at this tough work, and his proposal omits any very serious effort at such work. With irony, eventually his proposal would lead to a system of 'upvotes' from 'respected reviewers'! Or, "I don't understand the paper, but it was upvoted by Professor Issac Iatrogenic at Sawbones Medical School."! He still wants voting, but he has no well designed 'voting system', and the present peer-review process does. If he wants a better system, then okay, but his proposal is for essentially no system, and that would be much worse. For his many arguments, mostly they omit a big, huge point: He's considering the problems with the papers that were accepted but is not considering the no doubt on average much more severe problems with the huge number of papers that were rejected. His proposal to 'publish' just by posting on the Internet would result in all the rejected papers also being posted and, then, of course, soon, still more junk papers. Why buy Tide laundry detergent? Why drink Coke? Are we afraid that both Tide and Coke might give us something toxic? No. Of course not. Why? Because Tide and Coke have been used by many millions of people and are from companies with a huge financial fortune to lose if their quality falls. So, we 'trust' Tide and Coke. So, Tide and Coke are for us valuable 'brands', "valuable" because we can trust them because we know how much evidence there is that the products are at least okay. So, similarly, the peer-reviewed journals provide researchers with such valuable 'brand names' they can trust better than just some paper posted on the Internet. Net, the author of http://breast-cancer-research.com/content/12/S4/S13 http://breast-cancer-research.com/content/12/S4/S13 just doesn't 'get it'. He's looking for something he wants, doesn't get as much as he wants, and proposes a system that is still worse. Let's take his point about publication delays. That doesn't matter very much! Once the paper is submitted, it's no longer a secret, and mostly relevant researchers don't actually have to wait for the printed version two years later to make use of the paper. Let's take his point about the 'cost' of reviewing: Well the reviewers LIKE to review because it helps them keep up in their fields and, thus, is good for their careers! The effort to review is not a 'cost of the peer-review system' but just routine work by the reviewers for their own careers. In a hot field, reviewing the papers can be an advantage in time; the reviewer gets the papers maybe two years before someone just looking at published papers. Let's take his point about reviews of papers after they are published: Sure, over time, including after publication, the reputation of a paper can change. He wants to use this effect to replace the initial peer-review process. But he ignores that he is talking about papers that have been published and, thus, have passed peer-review, that is, are already 'certified' as being on the inside of the fence. If just publish papers on the Internet, then that first step of being 'certified' will be missing and the paper might just languish much longer before anything like the after publication review process he likes now. Or, he's missing the big point about how to filter water: First, just before the pick up of the pump, have a screen that filters out the rocks down to the size of sand. Second, have a fiber filter that filters out all the solids down to about 10 microns. Maybe next deal with the usual suspects of compounds of calcium, iron, and manganese. Next deal with bacteria. Next maybe use reverse osmosis or distillation to deal with everything else. Likely now have some good water to drink. Well, paper reviewing also has 'stages', and one of the early ones, but not the last one, but one that helps the last ones, is peer-review. E.g., we don't ask the reverse osmosis to filter out the gravel! His point about novel, original, or innovative work being rejected is poorly considered: We're not talking popularity of fashion frocks here. Instead, novel, etc., or not, a paper still has to be new, correct, and significant. Believe me, a solid proof that P = NP will not be rejected because it is too 'novel'! It will be accepted because the work is, again, new, correct, and significant. It is 'novel' mostly because it is especially significant. Again, to get published, pass new, correct, and significant. If after that the paper is also wildly 'novel', so be it; it still got published! He missed the big, huge point about focus on research specialties: As I described, if paper from author A goes to journal X in specialty S, then likely already the paper is in the right set or an appropriate set of editors, reviewers, and researchers. If the paper is well regarded, then the people in specialty S learn about it fairly quickly. So journal X serves as an 'accumulation point' for specialty S. If just publish papers on the Internet, then lose the value of such 'sets' and 'accumulation points'. So, indeed, if his proposal were to become accepted, an early step would be to have, for each specialty S, a special place to post papers in specialty S. Then to keep down the noise level, that special place for specialty S would have a 'moderator'. A good 'moderator' would be the editor in chief of journal X. Soon he would get too busy and would want some editors, say, from journal X. The editors would get too busy and soon would want some reviewers, say, from journal X. Then all concerned would want some quality control for their special place for specialty S. So, the moderator would call himself the editor in chief again and insist that his editors insist that their reviewers actually do good peer-reviews. Maybe the whole Internet site would be sponsored by Elsevier. Then they would charge for the papers that passed the reviews. Now except for tree cutting, we would be back where we are now. If he wants to improve the quality in publishing, then I have a suggestion for him: Do some brilliant research work, write it up as a paper, and submit it!

chalst 15y ago
Richard Smith does not propose a voting system, but the complete elimination of gatekeepers. It's worth emphasising that he is talking about biomed exclusively: other disciplines have different scholarly cultures. Most crucially, the commercial impact of biomed papers is higher than in other fields. When you write >reading research papers and judging if they are new, correct, and significant is TOUGH work. I agree, but when you write: >The present process makes a good effort at this tough work I am surprised you say this. Smith points to studies that show that referees' opinions have very little correlation, which suggests the minimum level of filtering actively done by referees is very low. It does not follow from this, though, that the institution of journals considered as a whole does not perform a useful filtering function, which is to say that I agree with you about your point about successive filters. >Publication delays Many journals follow anonymous reviewing and forbid distribution of preprints. The other point, which he does not make, is that referees have very little incentive, besides the opinion of the editorial board, to do their job properly. >His point about novel, original, or innovative work being rejected is poorly considered. ... If after that the paper is also wildly 'novel', so be it; it still got published! There's no doubt in that case that the journal has been an impediment to science. I don't advocate abandoning peer review. I do think we should be more open-eyed about the problems with it. It is a costly system, and one that does not work the way it is intended. Science needs gatekeepers, as you argue, although they need not resemble the current journal system.
- NY_Entrepreneur 15y ago
  On Smith's article and a voting system, uh, I read Smith's article ASAP! I was assuming, without careful checking, that Smith's article was much like that of this thread. Looks like you got me in some too fast reading! Yes, it looks like these articles are for biomedical work. I'm sure it's different in many small respects and maybe some larger ones. Uh, I said that the present system makes a good "effort", and you countered with some of the problems with the "results"! You may be right! The "correlation" is not very impressive: The review process is not quite that simple. Here's a dark, dirty little secret: Often at some point in the toughest part of the paper, some reviewer thinks, "I tried to look up the background for that, and I don't have the prerequisites even for the background. I can't take out a year to study to be ready to read this paper. The parts of the paper I can read look rock solid. The new parts I can read look nice. His references are high quality. His writing is carefully done. He doesn't make wild claims. It looks like a solid piece of work. I can't find anything wrong with it. Let him publish it: If it has a big error, then maybe that will come out and be on him. I'll give him a pass and go to dinner." or some such. So in this case correlation doesn't mean much! "Many journals follow anonymous reviewing and forbid distribution of preprints." Sure. In most simple respects, the paper is still all locked up until it appears in print maybe two years after submission. Still work goes on! So, the author of the paper can present parts of the work in lectures and seminars. Guys down the hall know and can start working on extensions. The author can work on extensions and give talks and reference the paper as "in submission". Generally the word gets around! Net, the publication delay doesn't much slow the actual research work. That is, the actual exploitation of the work in the paper doesn't really start just with the publication and people reading that, Instead, 'the word gets around'. "The other point, which he does not make, is that referees have very little incentive, besides the opinion of the editorial board, to do their job properly." The referees have a LOT of "incentive ... to do their work properly" if they want progress in their academic careers! Good referees get invited to be editors; good editors get invited to be good editors in chief. A referee, editor, or editor in chief is a 'gatekeeper' and, thus, a somewhat powerful guy. If the journal is good, the names are up in lights. So, professional reputation is enhanced. Professional connections are made. In terms of HN, there are connections in a professional social graph. A referee, editor, etc. gets to know who the up and comers are in their research field. They get early notice of new research results and, then, can start working on extensions. If they do a bad job, then their editor will soon not send them anymore papers to review! "There's no doubt in that case that the journal has been an impediment to science." I don't see that. Einstein published general relativity. People are still trying to understand it, test it, and understand the implications. So, at the time of publication, just how novel it was was not at all clear. But it got published and passed the first step in the filter. Slowly people understood that the paper was very important and worked hard on it. The publication process didn't end the work on the paper but did its job. I can't believe that many of the reviewers actually understood the Riemannian geometry. Still the paper got published. If the paper was actually nonsense, then that would be on Uncle Al. I believe that the current system is okay. The people objecting just want the system to be much more than it is; they may be asking for too much. One way to improve is to quit chopping down trees, but that change is likely on the way. But replacing the current system by putting papers on the Internet and letting nearly anyone 'vote' would be much worse.

chalst 15y ago

Richard Smith does not propose a voting system, but the complete elimination of gatekeepers. It's worth emphasising that he is talking about biomed exclusively: other disciplines have different scholarly cultures. Most crucially, the commercial impact of biomed papers is higher than in other fields. When you write >reading research papers and judging if they are new, correct, and significant is TOUGH work. I agree, but when you write: >The present process makes a good effort at this tough work I am surprised you say this. Smith points to studies that show that referees' opinions have very little correlation, which suggests the minimum level of filtering actively done by referees is very low. It does not follow from this, though, that the institution of journals considered as a whole does not perform a useful filtering function, which is to say that I agree with you about your point about successive filters. >Publication delays Many journals follow anonymous reviewing and forbid distribution of preprints. The other point, which he does not make, is that referees have very little incentive, besides the opinion of the editorial board, to do their job properly. >His point about novel, original, or innovative work being rejected is poorly considered. ... If after that the paper is also wildly 'novel', so be it; it still got published! There's no doubt in that case that the journal has been an impediment to science. I don't advocate abandoning peer review. I do think we should be more open-eyed about the problems with it. It is a costly system, and one that does not work the way it is intended. Science needs gatekeepers, as you argue, although they need not resemble the current journal system.

NY_Entrepreneur 15y ago
On Smith's article and a voting system, uh, I read Smith's article ASAP! I was assuming, without careful checking, that Smith's article was much like that of this thread. Looks like you got me in some too fast reading! Yes, it looks like these articles are for biomedical work. I'm sure it's different in many small respects and maybe some larger ones. Uh, I said that the present system makes a good "effort", and you countered with some of the problems with the "results"! You may be right! The "correlation" is not very impressive: The review process is not quite that simple. Here's a dark, dirty little secret: Often at some point in the toughest part of the paper, some reviewer thinks, "I tried to look up the background for that, and I don't have the prerequisites even for the background. I can't take out a year to study to be ready to read this paper. The parts of the paper I can read look rock solid. The new parts I can read look nice. His references are high quality. His writing is carefully done. He doesn't make wild claims. It looks like a solid piece of work. I can't find anything wrong with it. Let him publish it: If it has a big error, then maybe that will come out and be on him. I'll give him a pass and go to dinner." or some such. So in this case correlation doesn't mean much! "Many journals follow anonymous reviewing and forbid distribution of preprints." Sure. In most simple respects, the paper is still all locked up until it appears in print maybe two years after submission. Still work goes on! So, the author of the paper can present parts of the work in lectures and seminars. Guys down the hall know and can start working on extensions. The author can work on extensions and give talks and reference the paper as "in submission". Generally the word gets around! Net, the publication delay doesn't much slow the actual research work. That is, the actual exploitation of the work in the paper doesn't really start just with the publication and people reading that, Instead, 'the word gets around'. "The other point, which he does not make, is that referees have very little incentive, besides the opinion of the editorial board, to do their job properly." The referees have a LOT of "incentive ... to do their work properly" if they want progress in their academic careers! Good referees get invited to be editors; good editors get invited to be good editors in chief. A referee, editor, or editor in chief is a 'gatekeeper' and, thus, a somewhat powerful guy. If the journal is good, the names are up in lights. So, professional reputation is enhanced. Professional connections are made. In terms of HN, there are connections in a professional social graph. A referee, editor, etc. gets to know who the up and comers are in their research field. They get early notice of new research results and, then, can start working on extensions. If they do a bad job, then their editor will soon not send them anymore papers to review! "There's no doubt in that case that the journal has been an impediment to science." I don't see that. Einstein published general relativity. People are still trying to understand it, test it, and understand the implications. So, at the time of publication, just how novel it was was not at all clear. But it got published and passed the first step in the filter. Slowly people understood that the paper was very important and worked hard on it. The publication process didn't end the work on the paper but did its job. I can't believe that many of the reviewers actually understood the Riemannian geometry. Still the paper got published. If the paper was actually nonsense, then that would be on Uncle Al. I believe that the current system is okay. The people objecting just want the system to be much more than it is; they may be asking for too much. One way to improve is to quit chopping down trees, but that change is likely on the way. But replacing the current system by putting papers on the Internet and letting nearly anyone 'vote' would be much worse.