5 ms·

Real peer review has never been tried

131 points by bilsbie 4y ago

Blahah 4y ago
This article seems like it would be very interesting to me but after dismissing three intrusive popups in my mobile browser I stopped trying to read it. Does anyone know of a mobile browser friendly way to read this kind of site without being constantly bombarded? It's exhausting. It seems like the sort of problem that would have been solved, but my attempts to discover the solution have failed.
- SCNP 4y ago
  The 'Kill Sticky' plugin for Firefox works pretty well for me for most stuff. Most popups darken the screen, though, and it doesn't handle that. It's very useful to me nonetheless. If you don't use Firefox, you can go their site and copy/paste it to your bookmarks.
- roninghost 4y ago
  On android: fennec(firefox fork) + uBlockOrogin, NoScript, PrivacyBadger and Decentraleyes. I only had de reject all cookies prompt. With firefox you can aldo use the reade mode, that extracts the main taxt and displays it plainly.
  - truffdog 4y ago
    Firefox+uBlock is enough to make it readable on Android
- tommy5dollar 4y ago
  There's just a cookie prompt and then a newsletter sign-up a little later? I don't think there's any constant bombardment, if there is for you I'm interested to know more.
  - Blahah 4y ago
    I dismissed 3 overlays. 1. cookies 2. email newsletter 3. cookies again The constant bombardment was more in reference to the state of the web generally. Popups and overlays that aren't a result of an interaction I've chosen to make are exhausting for me (and people with ADHD generally, and many others). I don't go back to sites that have them, but would be happy to read (and often pay for) the content if that wasn't the context.
Barrera 4y ago
I was expecting the author to define "real peer review," but didn't see that. The best approximation is probably gleaned from the conclusion: - integration of preprint servers and alt metrics - tweaking incentives to review - making comments on papers public - use of software to detect fraud - directing resources specifically to improving peer review The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers. The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well. So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?
- lacker 4y ago
  Nowadays, journals and peer review solve the problem of, people need to make hiring decisions and funding decisions. But these decisionmakers don't have enough technical expertise or time to evaluate all the papers from all the applicants. The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.
  - cryptonector 4y ago
    It sounds funny, but it's true.
- bryanrasmussen 4y ago
  >Wide distribution of scientific information no longer requires scientific publishers. Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)
lacker 4y ago
The problem is that looking at journal quality is the best quick way to evaluate how good a paper is, when you aren't an expert on the topic. And a lot of employment and funding decisions are made by people who aren't experts. Nowadays a journal provides essentially no distribution, but there's no good alternative to journals as a "stamp of quality".
- elashri 4y ago
  > And a lot of employment and funding decisions are made by people who aren't experts. This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. At least you are not expert in the same field but have knowledge and it is easier for you to communicate and discuss the proposals. But getting someone who never did a real scientific research and the last time he wrote a scientific essay was when he was in college to determine which research should be funded is wrong.
  - throwawaylinux 4y ago
    > This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. Because funding requires taking from something else -- either raising more revenue or reducing funding for any of the infinite possibilities that it could be spent on. And nobody is an expert in everything. Where a particular area of expertise does cover the choices to direct funding, it generally does. With a lot of inefficiency and corruption and incompetence thrown in of course, but politicians and bureaucrats sure didn't decide how every dollar on the JWST would be spent.
  - impendia 4y ago
    I am an academic mathematician, who (like most) is an expert in a sub-subfield in my discipline. Outside my specialty I certainly have some knowledge, but I usually can't distinguish a major breakthrough from incremental progress. Meanwhile, I've been asked to evaluate hundreds of other mathematicians, on hiring committees and grant panels. Most of the applications are in specialities bordering mine, but not squarely in my area of expertise. Usually I can get a sense of what they've accomplished, but in most cases I can't reliably evaluate its depth or importance. Unfortunately, external markers of quality are needed.
Fomite 4y ago
"And reviews also vary widely in their conclusions – how they rate papers and whether they recommend editors to accept or reject the paper." This is assumed to be a bug, and not a feature. A lot of this comes from different perspectives, weighting, etc. Widening the number of reviewers isn't going to reduce the variance on this. You see it in grant review panels as well, which are much more "discuss and engage with your fellow reviewers". "We live in a very different world today than we did when journals were circulated through print: One with the Internet, where data and research don’t need to be circulated in limited and expensive publications. Research can be shared almost immediately with massive networks of scientists through blogs, preprints [1] , data storage platforms and online forums, at little to no cost." One of the assumptions here is that this access is uniform. It isn't. During COVID-19, some senior scientists essentially went with "Publication via press release", when more junior scientists can't do that. The popularity of figures on my field's Twitter hashtag is astonishingly non-uniform, cuts out whole swathes of people (govt. scientists), and is only lightly correlated IMO with the impact of their work. I will say that, when I was a junior scientist, getting some of my work in good journals carried it much further than internet distribution would have. "Finally, other types of centralized platforms could be hugely important too. Journals currently have exclusivity agreements (meaning researchers can only submit to one journal at a time) but they also have different formatting requirements, which take a long time to fulfill. By one survey estimate, articles are submitted an average of two times before they are published, and formatting takes up a total of 14 hours per article." The number of journals that only require formatting post-acceptance is growing, and is awesome. When it comes down to it, I think peer review's biggest strength is actually something the author takes as a negative. It's not "This paper has only been reviewed by two people", it's that every paper in this journal has been reviewed by at least two people, plus the public. For a lot of papers, I'm actually skeptical that that number is lower than if we went with "Well, the community will review it in the fullness of time".
reagle 4y ago
Frontiers has a great review/publication model https://www.frontiersin.org/about/peer-review/ https://www.frontiersin.org/about/peer-review/
tomcam 4y ago
> Before the adoption of peer review, journals focused on quickly disseminating letters and communications between scientists, with little to no editing or external reviewing It seems to me that a return to this standard is inevitable: * Scientific papers are in enormously difficult to peer review properly because fields have become so specialized * The incentive structure is highly unbalanced, and the truth is, the people you’d like to peer review your papers are already too busy * Peer review is very expensive (in terms of time) for a conscientious reviewer * Many of these papers are low quality, serving to buttress the reputation of the author more than moving the field forward Since it’s an open secret that so many journals are worthless, it would probably be better just to return to the publish fast model instead of trying to fix a sick endeavor.
pabs3 4y ago
I wonder if many scientists publish their day-to-day in-progress work/thoughts/journal/data etc.
Heleana 4y ago
I wonder if many scientists publish their daily work, thoughts, journals, data, etc. that are still in progress.

131 points by bilsbie 4y ago

This article seems like it would be very interesting to me but after dismissing three intrusive popups in my mobile browser I stopped trying to read it. Does anyone know of a mobile browser friendly way to read this kind of site without being constantly bombarded? It's exhausting. It seems like the sort of problem that would have been solved, but my attempts to discover the solution have failed.

SCNP 4y ago
The 'Kill Sticky' plugin for Firefox works pretty well for me for most stuff. Most popups darken the screen, though, and it doesn't handle that. It's very useful to me nonetheless. If you don't use Firefox, you can go their site and copy/paste it to your bookmarks.
roninghost 4y ago
On android: fennec(firefox fork) + uBlockOrogin, NoScript, PrivacyBadger and Decentraleyes. I only had de reject all cookies prompt. With firefox you can aldo use the reade mode, that extracts the main taxt and displays it plainly.
- truffdog 4y ago
  Firefox+uBlock is enough to make it readable on Android
tommy5dollar 4y ago
There's just a cookie prompt and then a newsletter sign-up a little later? I don't think there's any constant bombardment, if there is for you I'm interested to know more.
- Blahah 4y ago
  I dismissed 3 overlays. 1. cookies 2. email newsletter 3. cookies again The constant bombardment was more in reference to the state of the web generally. Popups and overlays that aren't a result of an interaction I've chosen to make are exhausting for me (and people with ADHD generally, and many others). I don't go back to sites that have them, but would be happy to read (and often pay for) the content if that wasn't the context.

Barrera 4y ago

I was expecting the author to define "real peer review," but didn't see that. The best approximation is probably gleaned from the conclusion: - integration of preprint servers and alt metrics - tweaking incentives to review - making comments on papers public - use of software to detect fraud - directing resources specifically to improving peer review The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers. The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well. So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?

lacker 4y ago
Nowadays, journals and peer review solve the problem of, people need to make hiring decisions and funding decisions. But these decisionmakers don't have enough technical expertise or time to evaluate all the papers from all the applicants. The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.
- cryptonector 4y ago
  It sounds funny, but it's true.
bryanrasmussen 4y ago
>Wide distribution of scientific information no longer requires scientific publishers. Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)

lacker 4y ago

The problem is that looking at journal quality is the best quick way to evaluate how good a paper is, when you aren't an expert on the topic. And a lot of employment and funding decisions are made by people who aren't experts. Nowadays a journal provides essentially no distribution, but there's no good alternative to journals as a "stamp of quality".

elashri 4y ago
> And a lot of employment and funding decisions are made by people who aren't experts. This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. At least you are not expert in the same field but have knowledge and it is easier for you to communicate and discuss the proposals. But getting someone who never did a real scientific research and the last time he wrote a scientific essay was when he was in college to determine which research should be funded is wrong.
- throwawaylinux 4y ago
  > This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. Because funding requires taking from something else -- either raising more revenue or reducing funding for any of the infinite possibilities that it could be spent on. And nobody is an expert in everything. Where a particular area of expertise does cover the choices to direct funding, it generally does. With a lot of inefficiency and corruption and incompetence thrown in of course, but politicians and bureaucrats sure didn't decide how every dollar on the JWST would be spent.
- impendia 4y ago
  I am an academic mathematician, who (like most) is an expert in a sub-subfield in my discipline. Outside my specialty I certainly have some knowledge, but I usually can't distinguish a major breakthrough from incremental progress. Meanwhile, I've been asked to evaluate hundreds of other mathematicians, on hiring committees and grant panels. Most of the applications are in specialities bordering mine, but not squarely in my area of expertise. Usually I can get a sense of what they've accomplished, but in most cases I can't reliably evaluate its depth or importance. Unfortunately, external markers of quality are needed.

Fomite 4y ago

"And reviews also vary widely in their conclusions – how they rate papers and whether they recommend editors to accept or reject the paper." This is assumed to be a bug, and not a feature. A lot of this comes from different perspectives, weighting, etc. Widening the number of reviewers isn't going to reduce the variance on this. You see it in grant review panels as well, which are much more "discuss and engage with your fellow reviewers". "We live in a very different world today than we did when journals were circulated through print: One with the Internet, where data and research don’t need to be circulated in limited and expensive publications. Research can be shared almost immediately with massive networks of scientists through blogs, preprints [1] , data storage platforms and online forums, at little to no cost." One of the assumptions here is that this access is uniform. It isn't. During COVID-19, some senior scientists essentially went with "Publication via press release", when more junior scientists can't do that. The popularity of figures on my field's Twitter hashtag is astonishingly non-uniform, cuts out whole swathes of people (govt. scientists), and is only lightly correlated IMO with the impact of their work. I will say that, when I was a junior scientist, getting some of my work in good journals carried it much further than internet distribution would have. "Finally, other types of centralized platforms could be hugely important too. Journals currently have exclusivity agreements (meaning researchers can only submit to one journal at a time) but they also have different formatting requirements, which take a long time to fulfill. By one survey estimate, articles are submitted an average of two times before they are published, and formatting takes up a total of 14 hours per article." The number of journals that only require formatting post-acceptance is growing, and is awesome. When it comes down to it, I think peer review's biggest strength is actually something the author takes as a negative. It's not "This paper has only been reviewed by two people", it's that every paper in this journal has been reviewed by at least two people, plus the public. For a lot of papers, I'm actually skeptical that that number is lower than if we went with "Well, the community will review it in the fullness of time".

reagle 4y ago

Frontiers has a great review/publication model https://www.frontiersin.org/about/peer-review/ https://www.frontiersin.org/about/peer-review/

tomcam 4y ago

> Before the adoption of peer review, journals focused on quickly disseminating letters and communications between scientists, with little to no editing or external reviewing It seems to me that a return to this standard is inevitable: * Scientific papers are in enormously difficult to peer review properly because fields have become so specialized * The incentive structure is highly unbalanced, and the truth is, the people you’d like to peer review your papers are already too busy * Peer review is very expensive (in terms of time) for a conscientious reviewer * Many of these papers are low quality, serving to buttress the reputation of the author more than moving the field forward Since it’s an open secret that so many journals are worthless, it would probably be better just to return to the publish fast model instead of trying to fix a sick endeavor.

pabs3 4y ago

I wonder if many scientists publish their day-to-day in-progress work/thoughts/journal/data etc.

Heleana 4y ago

I wonder if many scientists publish their daily work, thoughts, journals, data, etc. that are still in progress.