7 ms·

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible

by chmike 10y ago

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

ManlyBread 10y ago
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
- randall 10y ago
  But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.
  - nurettin 10y ago
    The problem is not about proving your identity, it is about having the facilities, funds and connections for publishing papers which the gatekeepers seem to have. Cryptographic signing solves a different problem. And it is the least concerning one.
  - yayday 10y ago
    There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and especially the PI. If the PI is a heavy hitter in my field then there's my trustability factor right there. If I don't know the PI but I know the laboratory or university it's affiliated with then there's my third line of a trustability metric.
    - apathy 10y ago
      You don't consider the journal? You're a better man (or woman) than I am, if so. Otherwise agreed, reluctantly. This implicit trust is frequently manipulated to relax standards by experienced PIs, and in biomedical work that is a source of numerous problems. In an ideal world, work from an unknown author would receive the same scrutiny as that from an established lab, and vice versa. Obviously this isn't the case, but the further we drift from it, the more likely we become to see sloppy results wasting everyone's time. It can't just be about the narrative. That simply isn't science. That's storytelling, and confusing the two has caused a great deal of harm to science in the public eye.
      - yayday 10y ago
        I'd be lying if I said the journal didn't matter strictly speaking, but in general as long as it's a "real" journal (as opposed to one of those obviously fake journals I keep getting spammed by) then I don't really care. Don't get me wrong though, just because the paper has a heavy hitter as the PI or even as first author, doesn't mean I treat the content with less scrutiny. It's more that I know my time is extremely limited so I'm more willing to spend more time dissecting a paper from an established first author or PI than an unknown person (unknown also meaning that it's not a paper someone recommended to me or a paper someone cited in another work). It's more like, I'll _make_ time to read papers with certain authors on it because I know that they do good work. I'll also _make_ time to read something if it's published in Nature et al for example. But a random paper from a random author in a random group? It's not that I don't trust this person's work, it's more that I'm unfamiliar with that person's work and there's other things I could spend my time doing as well. I'll download the paper, throw it on my iPad, etc, and I'll get around to it eventually.
- elementalest 10y ago
  Yeah, it also means minorities can be silenced via the majority. While this is good for protecting against incorrect/poorly researched work, it can also prevent novel ideas from gaining traction. Though, this problem exists in the system we have now.
- chmike 10y ago
  A fully open system can have a ranking on articles and reviewers (voters). Wikipedia has no ranking on articles and no rankings of contributors/reviewers (at least initially). So these are different systems. I'm convinced that random tourists vote can't have the same weight as recognized experts in the field who know the subject well and whose judgments are appreciated in their form and content. I'm convinced of this because it is the fundamental principle of the referee's review.
splatcollision 10y ago
Had the same kind of idea - apparently "Facebook for Scientists" is apparently a thing. http://www.sciencemag.org/careers/2014/08/researchgate-facebook-science http://www.sciencemag.org/careers/2014/08/researchgate-faceb... https://www.researchgate.net/ https://www.researchgate.net/
- apathy 10y ago
  researchgate is a fucking cancer. Dark patterns rule there
ap22213 10y ago
Isn't 'the scientific paper' a kind of dated concept? I'd love to see the whole scientific process from start to finish in a 'notebook' form. Why summarize findings in a paper at the end of it all, when all steps (including conception, data gathering, successes, missteps, comments, and corrections) could all be viewed as a whole? It would seem to be more in line with the intent of science. I'm not a scientist, obviously. And, I realize that practical science has as much to do with reputation, fame, and the climbing of academic hierarchies as it does with research. But, it would be so cool to see science more as an evolving, collaborative process.
- dflock 10y ago
  Because a summary is very useful. Imagine you are reading about 5 years of day-in-day-out work by a team of 1500 physicists (eg. CERN, LHC, etc...). Would you rather read the entire as-it-happened stream of research, going back 7500 person-years - or would you prefer to read a summary, in the form of a paper? I agree that ideally the stream should be publically available as well, for those that want to consult it - but having regular summaries, written at intervals is very useful.
jessriedel 10y ago
Physicists have access to this if they combine the arXiv with Scirate. https://scirate.com/ https://scirate.com/ Very few choose to, and very little discussion actually happens. There is something of a chicken-or-egg problem, but the site is sufficiently established in one community (quantum info) that if folks wanted to discuss papers they could. So I take it as weak evidence that they just don't have the incentives to do so. If a central discussion forum were anointed as the One True Forum, I weakly predict physicists would spend time defending themselves from "drive-by criticism" (i.e., criticism that is vague and takes little thought, but consumes large resources to defend against) but little constructive would get done.
- chmike 10y ago
  arxiv is not open. At least not as open as it should be in my opinion. But I fully agree on the chicken-or-egg problem. The main problem is bootsrapping such web service. I don't think that "anoiting" a forum/web service would be a good strategy. We need to keep open the possibility of a new and better system to take over. My feeling is that two fundamental requirements must be satisfied. 1. Anybody should be able to submit a paper. 2. Anybody must have an opportunity to become a recognized expert in a field. By anybody I mean that academic or career credentials must not come into play. Obviously this begs for a rating system on the articles and on the "experts" too. Obviously recognized "experts" should have a bigger weight in the rating than random members. This makes it different from reddit and hacker news. But the idea is similar. Members should also be able to filter papers by rating and be notified only when a paper reaches a desired ranking. This is another difference from hacker news and reddit. I would be very interested in participating in a project of this kind.
  - jessriedel 10y ago
    http://vixra.org/ http://vixra.org/
  - jsprogrammer 10y ago
    One could start with a model similar to that in which most people using systems which this new system is meant to replace already find themselves. You should be able to select peers (though, maybe you should be given some as well?) and "experts" to follow. You would only see their activity when you want to, but you could also participate in the larger network. As a practical matter of implementation, I think allowing groups to run segregated networks (ie. their own "node") which can later be joined to each other makes sense for getting actual adoption and for reducing development, infrastructure, and support costs. I think there are likely many problems with such a crude model, but that is where we find ourselves it seems. Waiting to think up something better might be prudent, but that might be difficult without trying something out.

by chmike 10y ago

ManlyBread 10y ago

I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.

randall 10y ago
But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.
- nurettin 10y ago
  The problem is not about proving your identity, it is about having the facilities, funds and connections for publishing papers which the gatekeepers seem to have. Cryptographic signing solves a different problem. And it is the least concerning one.
- yayday 10y ago
  There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and especially the PI. If the PI is a heavy hitter in my field then there's my trustability factor right there. If I don't know the PI but I know the laboratory or university it's affiliated with then there's my third line of a trustability metric.
  - apathy 10y ago
    You don't consider the journal? You're a better man (or woman) than I am, if so. Otherwise agreed, reluctantly. This implicit trust is frequently manipulated to relax standards by experienced PIs, and in biomedical work that is a source of numerous problems. In an ideal world, work from an unknown author would receive the same scrutiny as that from an established lab, and vice versa. Obviously this isn't the case, but the further we drift from it, the more likely we become to see sloppy results wasting everyone's time. It can't just be about the narrative. That simply isn't science. That's storytelling, and confusing the two has caused a great deal of harm to science in the public eye.
    - yayday 10y ago
      I'd be lying if I said the journal didn't matter strictly speaking, but in general as long as it's a "real" journal (as opposed to one of those obviously fake journals I keep getting spammed by) then I don't really care. Don't get me wrong though, just because the paper has a heavy hitter as the PI or even as first author, doesn't mean I treat the content with less scrutiny. It's more that I know my time is extremely limited so I'm more willing to spend more time dissecting a paper from an established first author or PI than an unknown person (unknown also meaning that it's not a paper someone recommended to me or a paper someone cited in another work). It's more like, I'll _make_ time to read papers with certain authors on it because I know that they do good work. I'll also _make_ time to read something if it's published in Nature et al for example. But a random paper from a random author in a random group? It's not that I don't trust this person's work, it's more that I'm unfamiliar with that person's work and there's other things I could spend my time doing as well. I'll download the paper, throw it on my iPad, etc, and I'll get around to it eventually.
elementalest 10y ago
Yeah, it also means minorities can be silenced via the majority. While this is good for protecting against incorrect/poorly researched work, it can also prevent novel ideas from gaining traction. Though, this problem exists in the system we have now.
chmike 10y ago
A fully open system can have a ranking on articles and reviewers (voters). Wikipedia has no ranking on articles and no rankings of contributors/reviewers (at least initially). So these are different systems. I'm convinced that random tourists vote can't have the same weight as recognized experts in the field who know the subject well and whose judgments are appreciated in their form and content. I'm convinced of this because it is the fundamental principle of the referee's review.

splatcollision 10y ago

Had the same kind of idea - apparently "Facebook for Scientists" is apparently a thing. http://www.sciencemag.org/careers/2014/08/researchgate-facebook-science http://www.sciencemag.org/careers/2014/08/researchgate-faceb... https://www.researchgate.net/ https://www.researchgate.net/

apathy 10y ago
researchgate is a fucking cancer. Dark patterns rule there

ap22213 10y ago

Isn't 'the scientific paper' a kind of dated concept? I'd love to see the whole scientific process from start to finish in a 'notebook' form. Why summarize findings in a paper at the end of it all, when all steps (including conception, data gathering, successes, missteps, comments, and corrections) could all be viewed as a whole? It would seem to be more in line with the intent of science. I'm not a scientist, obviously. And, I realize that practical science has as much to do with reputation, fame, and the climbing of academic hierarchies as it does with research. But, it would be so cool to see science more as an evolving, collaborative process.

dflock 10y ago
Because a summary is very useful. Imagine you are reading about 5 years of day-in-day-out work by a team of 1500 physicists (eg. CERN, LHC, etc...). Would you rather read the entire as-it-happened stream of research, going back 7500 person-years - or would you prefer to read a summary, in the form of a paper? I agree that ideally the stream should be publically available as well, for those that want to consult it - but having regular summaries, written at intervals is very useful.

jessriedel 10y ago

Physicists have access to this if they combine the arXiv with Scirate. https://scirate.com/ https://scirate.com/ Very few choose to, and very little discussion actually happens. There is something of a chicken-or-egg problem, but the site is sufficiently established in one community (quantum info) that if folks wanted to discuss papers they could. So I take it as weak evidence that they just don't have the incentives to do so. If a central discussion forum were anointed as the One True Forum, I weakly predict physicists would spend time defending themselves from "drive-by criticism" (i.e., criticism that is vague and takes little thought, but consumes large resources to defend against) but little constructive would get done.

chmike 10y ago
arxiv is not open. At least not as open as it should be in my opinion. But I fully agree on the chicken-or-egg problem. The main problem is bootsrapping such web service. I don't think that "anoiting" a forum/web service would be a good strategy. We need to keep open the possibility of a new and better system to take over. My feeling is that two fundamental requirements must be satisfied. 1. Anybody should be able to submit a paper. 2. Anybody must have an opportunity to become a recognized expert in a field. By anybody I mean that academic or career credentials must not come into play. Obviously this begs for a rating system on the articles and on the "experts" too. Obviously recognized "experts" should have a bigger weight in the rating than random members. This makes it different from reddit and hacker news. But the idea is similar. Members should also be able to filter papers by rating and be notified only when a paper reaches a desired ranking. This is another difference from hacker news and reddit. I would be very interested in participating in a project of this kind.
- jessriedel 10y ago
  http://vixra.org/ http://vixra.org/
- jsprogrammer 10y ago
  One could start with a model similar to that in which most people using systems which this new system is meant to replace already find themselves. You should be able to select peers (though, maybe you should be given some as well?) and "experts" to follow. You would only see their activity when you want to, but you could also participate in the larger network. As a practical matter of implementation, I think allowing groups to run segregated networks (ie. their own "node") which can later be joined to each other makes sense for getting actual adoption and for reducing development, infrastructure, and support costs. I think there are likely many problems with such a crude model, but that is where we find ourselves it seems. Waiting to think up something better might be prudent, but that might be difficult without trying something out.