15 ms·

Let’s make peer review scientific

279 points by return0 10y ago

danieltillett 10y ago
This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to just sit there thinking if this sort of analysis was in a paper they were reviewing they would have ripped the authors to shreds. Peer review fails on all levels. It does not functions as a source of quality control (everything gets published eventually) and even worse it rarely improves the quality of the paper being reviewed. I have published dozens of papers over the years and on only one occasion has the review process improved the paper - in most cases the reviewers demands made the papers worse by forcing me to remove important information or include irrelevant details (citing the reviewers publications mostly).
- chunky1994 10y ago
  I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.
  - danieltillett 10y ago
    You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.
- quantumhobbit 10y ago
  Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.
  - danieltillett 10y ago
    Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
    - dangerlibrary 10y ago
      At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.
      - danieltillett 10y ago
        I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.
        jsprogrammer 10y ago
        Why is science viewed as a competition there?
- xioxox 10y ago
  > and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.
  - danieltillett 10y ago
    I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published. The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.
    - xioxox 10y ago
      The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory. One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.
      - danieltillett 10y ago
        I wish that was the way the biological sciences worked. My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)
lordnacho 10y ago
Perhaps it's a question of incentives. What exactly do you get out of reading a quite complex piece of work and saying your opinion? The closest I can come, as a non-academic, is perhaps reading other people's code. It's bloody hard. You need to concentrate, and it's not like reading a newspaper article at all. Even small errors are not easy to spot, and it's even hard to know whether the code is structured in the way the comments say. It somewhat makes sense to do the exercise if I have to be using the codebase. If I'm just commenting and giving a thumbs up/down, it's quite easy to reduce effort, come up with some generic comments, and see how the other reviewers do. Which is a recipe for letting errors through.
- jsprogrammer 10y ago
  Reviews shouldn't be opinion, but facts. Those facts can then be disseminated to larger audiences. The incentive is to ensure that only factual information is claimed in publication so that everyone else can be as informed as possible, thereby (hopefully) allowing us to all make better decisions (and, perhaps, this could even accelerate the above described process).
  - apathy 10y ago
    Every time you use the word "should" you beg the question. Why, exactly, are incentives aligned to make it so? Or is the problem that the incentives are not in fact set up to encourage the desired outcome? Anything else is wishful thinking. People seldom do what they know to be right; they do what is convenient, then repent. It follows that one must make the right thing the convenient or rewarding thing if one wishes to see humans doing it. By and large, people are somewhat predictable. This can be used for good as well as the more familiar evil uses.
    - jsprogrammer 10y ago
      Reviews should be facts by definition of a review (as I use it), so the question is not begged. The incentives are 'aligned' that way because it is the only way to benefit. Publishing into an echo chamber (or void) is not science, by definition.
altoz 10y ago
Hypothesis: peer-reviewed articles aren't rigorous Prediction: a flawed article will pass peer-review Testing: https://svpow.com/2013/10/03/john-bohannons-peer-review-sting-against-science/ https://svpow.com/2013/10/03/john-bohannons-peer-review-stin... Analysis: Peer review is not rigorous
- wapapaloobop 10y ago
  Isn't there are standard just below rigour, though, that is still worth checking for, something like 'no obvious errors within the current paradigm'?
chmike 10y ago
I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
- ManlyBread 10y ago
  I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
  - randall 10y ago
    But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.
    - nurettin 10y ago
      The problem is not about proving your identity, it is about having the facilities, funds and connections for publishing papers which the gatekeepers seem to have. Cryptographic signing solves a different problem. And it is the least concerning one.
    - yayday 10y ago
      There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and especially the PI. If the PI is a heavy hitter in my field then there's my trustability factor right there. If I don't know the PI but I know the laboratory or university it's affiliated with then there's my third line of a trustability metric.
Toenex 10y ago
We could allow everything to be published and then use a continual monitoring approach. After all the limiting resource these days is not paper but reviewers time, so lets distribute that. What is wrong with model like HN uses for open review and comment on work? Even a voting system to help identify the most 'important' works. Edit: Haha great minds chmike
jkot 10y ago
I personally prefer reproducibility over peer review.
- peatmoss 10y ago
  I'd be loathe to choose. That said, reproducibility is the part that's obviously broken in many cases. Peer review is broken in more obfuscated ways.
- Fomite 10y ago
  I personally prefer not choosing between the two.
mariusz79 10y ago
I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.
- Jabbles 10y ago
  That's not going to work for Astronomy, or results of multi-year studies.
  - mariusz79 10y ago
    it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.
    - Declanomous 10y ago
      You can fabricate data nearly as easily as you can fabricate results. It is easier to detect fabricated results from raw data than it is from the final results, but fraud adapts to the methods used to detect it. If you create a new way to detect fraud, fraud will change to avoid that method. Detecting fraud is an arms race.
      - apathy 10y ago
        At some point it becomes easier to just run the damned experiment. Similar to buying a safe based on making its contents not worth the trouble to crack it. This would be a good first step. "Photoshop for biologists" (as a workshop) is very much a thing.
    - yayday 10y ago
      You're already moving your goalposts. Your first post said replicates the finding. If I simply post my results and someone else looks at the data and agrees with me then are they really replicating my finding or did they, oh what's the word… peer review my work and agreed?
joelg 10y ago
Shameless plug: I'm working at the MIT Media Lab on the PubPub project (http://www.pubpub.org http://www.pubpub.org), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentive of publication itself as the ultimate goal. Instead, we need to think of documents as evolving, growing bodies of knowledge and compilations of ongoing research. Every step of the scientific process is important, yet most of it is flattened and compressed and lost, like most negative results, which are ditched in search of sexy click-bait headliner results. Another is the role of publishers as gatekeepers and arbiters of truth. We need a medium in which anyone can curate a journal, and in which submission, review, and acceptance procedures are consistent and transparent. Another is the nature of the medium itself. It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced! Insert your favorite Bret Victor/Ted Nelson rant here: we need modern, digitally-native documents that are as rich as the information they contain. Another is reproduciblity. We should be able to see the code that transformed the raw dataset, tweak it, and publish our own fork, while automatically keeping the thread of attribution. The list goes on and on...
- p4bl0 10y ago
  Hi! I really think what you are doing here is great and I hope you will succeed! Is there a way to test the platform without creating to much garbage? Like a sandbox version maybe?
  - joelg 10y ago
    Everything is open-sourced, and we've tried to document as much as possible to make it easy for people to host their own instances! Alternatively, you can just create arbitrarily many drafts without publishing any of them. https://github.com/pubpub/pubpub https://github.com/pubpub/pubpub
    - p4bl0 10y ago
      Running my own instance is a bit heavy just to try the platform :). But thanks for the tips, I can create draft and delete them without trashing your instance. The fact that it is open source is a major positive point. I suppose you have heard of SJS [1] which in a way is very close to what PubPub is, except I do not believe in SJS because of it being closed source: it is way too unsafe for a scientific community to take the risk of adopting such a media for a new journal or a conference as the platform is not open and there is a risk of Elsevier buying it to shut it down or closing it even more, for instance. [1] http://sjscience.org/ http://sjscience.org/
zyxzevn 10y ago
The peer-review is mixed with a reductionistic structure. Each scientific area is specialized. The articles are reviewed by people in the specialized fields. Often they find something that is not entirely in their field of knowledge. That is because they are trying to "science" something new. So they can produce theories that seem a solution from their field of knowledge, but are not valid in another field. The peers will not see this problem. Another problem is that engineers that try to apply this science find practical problems. But because they are seen as "of lesser knowledge" their practical criticism is often rejected. The best example I know is the "theory" that magnetic fields can bump into each other, producing energy. In this model the fields are made of flux lines, which can bumb into other flux lines. The flux lines will then break and reorganize, producing energy. Yet, as I write this, you may already think that this is bullshit, because flux-lines are imaginary lines to describe a magnetic field. Now, with this in mind, look at magnetic reconnection. This is a theory made scientists specialized in astronomy, and not in the area of electromagnetism. I believe that this problem is in every (specialized) area of science.
teslaberry 10y ago
Science (Doing and reviewing it)is an art, not a science.
Pinatubo 10y ago
There's a peer review scandal currently underway involving a top economics journal. https://gborjas.org/2016/06/30/a-rant-on-peer-review/ https://gborjas.org/2016/06/30/a-rant-on-peer-review/
aaxe 10y ago
www.Examine.com does a great job analyzing nutrition/supplements studies.
apathy 10y ago
Buried in the middle of this (wonderful) article is the heart of the problem -- academia has foolishly placed its metrics into the hands of editors and publishers, who have corrupted the living hell out of it. Ctrl-F "Cochrane" and witness the exchange between an editor, who benefited from the status quo, and a scholar, who did not and does not. Academia != scholarship and has not for some time. There is no longer a good reason for traditional for-profit journals to exist. (Before someone says it: SpringerNature likes to pretend that editorial independence is possible, but they'll have no choice save to fire their "news" guys if their board asks for this). Please recall that the entire point of the World Wide Web was to share physics papers. The arXiv exists because of physicists (who quickly noticed that HTML wasn't a good substitute for LaTeX when writing math-heavy papers). The problem is not technological. It's social. And until the incentives are fixed (the Cochrane Collaboration in Britain has gone a long, long way to address this, and now the Wellcome Trust is going even further), nothing of any real import will change. In the USA, NIH could make a lot of positive changes (and in exchanges with mid- to senior executive level directors, I honestly believe they're trying to do so). But it will take time. Academia moves with glacial speed, when it moves at all.
arca_vorago 10y ago
I'll preface by saying I'm only a sysadmin with a love of science, but I learned the most about peer review and scientific publishing while working in biotech at a genetics company. At some point I realized I actually had to have some understanding of the science to properly admin it, and began reading lots of papers, and I slowly started to realize just how bad many papers are. There are a few key issues: 1. Scientists who "collaborate" with other scientists but do a small fraction of actual work get their names on papers as number fodder. Anytime I hear "I've been published in over 1000 journal articles" now I generally become more skeptical. 2. Lack of reproducibility. Not only in the methods and the documentation of the methods, but also the fact that most things just simply aren't even tested by a third party. 3. Publishing. They have far too long locked up information the public deserves to know, which is bad enough, but then they do a bad job of it and allow bad science in. The largest part of the problem imho because they created the situation that semi forces the scientists into questionable paper writing tactics.

279 points by return0 10y ago

This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to just sit there thinking if this sort of analysis was in a paper they were reviewing they would have ripped the authors to shreds. Peer review fails on all levels. It does not functions as a source of quality control (everything gets published eventually) and even worse it rarely improves the quality of the paper being reviewed. I have published dozens of papers over the years and on only one occasion has the review process improved the paper - in most cases the reviewers demands made the papers worse by forcing me to remove important information or include irrelevant details (citing the reviewers publications mostly).

chunky1994 10y ago
I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.
- danieltillett 10y ago
  You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.
quantumhobbit 10y ago
Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.
- danieltillett 10y ago
  Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
  - dangerlibrary 10y ago
    At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.
    - danieltillett 10y ago
      I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.
      - jsprogrammer 10y ago
        Why is science viewed as a competition there?
xioxox 10y ago
> and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.
- danieltillett 10y ago
  I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published. The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.
  - xioxox 10y ago
    The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory. One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.
    - danieltillett 10y ago
      I wish that was the way the biological sciences worked. My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)

lordnacho 10y ago

Perhaps it's a question of incentives. What exactly do you get out of reading a quite complex piece of work and saying your opinion? The closest I can come, as a non-academic, is perhaps reading other people's code. It's bloody hard. You need to concentrate, and it's not like reading a newspaper article at all. Even small errors are not easy to spot, and it's even hard to know whether the code is structured in the way the comments say. It somewhat makes sense to do the exercise if I have to be using the codebase. If I'm just commenting and giving a thumbs up/down, it's quite easy to reduce effort, come up with some generic comments, and see how the other reviewers do. Which is a recipe for letting errors through.

jsprogrammer 10y ago
Reviews shouldn't be opinion, but facts. Those facts can then be disseminated to larger audiences. The incentive is to ensure that only factual information is claimed in publication so that everyone else can be as informed as possible, thereby (hopefully) allowing us to all make better decisions (and, perhaps, this could even accelerate the above described process).
- apathy 10y ago
  Every time you use the word "should" you beg the question. Why, exactly, are incentives aligned to make it so? Or is the problem that the incentives are not in fact set up to encourage the desired outcome? Anything else is wishful thinking. People seldom do what they know to be right; they do what is convenient, then repent. It follows that one must make the right thing the convenient or rewarding thing if one wishes to see humans doing it. By and large, people are somewhat predictable. This can be used for good as well as the more familiar evil uses.
  - jsprogrammer 10y ago
    Reviews should be facts by definition of a review (as I use it), so the question is not begged. The incentives are 'aligned' that way because it is the only way to benefit. Publishing into an echo chamber (or void) is not science, by definition.

altoz 10y ago

Hypothesis: peer-reviewed articles aren't rigorous Prediction: a flawed article will pass peer-review Testing: https://svpow.com/2013/10/03/john-bohannons-peer-review-sting-against-science/ https://svpow.com/2013/10/03/john-bohannons-peer-review-stin... Analysis: Peer review is not rigorous

wapapaloobop 10y ago
Isn't there are standard just below rigour, though, that is still worth checking for, something like 'no obvious errors within the current paradigm'?

chmike 10y ago

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

ManlyBread 10y ago
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
- randall 10y ago
  But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.
  - nurettin 10y ago
    The problem is not about proving your identity, it is about having the facilities, funds and connections for publishing papers which the gatekeepers seem to have. Cryptographic signing solves a different problem. And it is the least concerning one.
  - yayday 10y ago
    There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and especially the PI. If the PI is a heavy hitter in my field then there's my trustability factor right there. If I don't know the PI but I know the laboratory or university it's affiliated with then there's my third line of a trustability metric.

Toenex 10y ago

We could allow everything to be published and then use a continual monitoring approach. After all the limiting resource these days is not paper but reviewers time, so lets distribute that. What is wrong with model like HN uses for open review and comment on work? Even a voting system to help identify the most 'important' works. Edit: Haha great minds chmike

jkot 10y ago

I personally prefer reproducibility over peer review.

peatmoss 10y ago
I'd be loathe to choose. That said, reproducibility is the part that's obviously broken in many cases. Peer review is broken in more obfuscated ways.
Fomite 10y ago
I personally prefer not choosing between the two.

mariusz79 10y ago

I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.

Jabbles 10y ago
That's not going to work for Astronomy, or results of multi-year studies.
- mariusz79 10y ago
  it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.
  - Declanomous 10y ago
    You can fabricate data nearly as easily as you can fabricate results. It is easier to detect fabricated results from raw data than it is from the final results, but fraud adapts to the methods used to detect it. If you create a new way to detect fraud, fraud will change to avoid that method. Detecting fraud is an arms race.
    - apathy 10y ago
      At some point it becomes easier to just run the damned experiment. Similar to buying a safe based on making its contents not worth the trouble to crack it. This would be a good first step. "Photoshop for biologists" (as a workshop) is very much a thing.
  - yayday 10y ago
    You're already moving your goalposts. Your first post said replicates the finding. If I simply post my results and someone else looks at the data and agrees with me then are they really replicating my finding or did they, oh what's the word… peer review my work and agreed?

joelg 10y ago

Shameless plug: I'm working at the MIT Media Lab on the PubPub project (http://www.pubpub.org http://www.pubpub.org), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentive of publication itself as the ultimate goal. Instead, we need to think of documents as evolving, growing bodies of knowledge and compilations of ongoing research. Every step of the scientific process is important, yet most of it is flattened and compressed and lost, like most negative results, which are ditched in search of sexy click-bait headliner results. Another is the role of publishers as gatekeepers and arbiters of truth. We need a medium in which anyone can curate a journal, and in which submission, review, and acceptance procedures are consistent and transparent. Another is the nature of the medium itself. It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced! Insert your favorite Bret Victor/Ted Nelson rant here: we need modern, digitally-native documents that are as rich as the information they contain. Another is reproduciblity. We should be able to see the code that transformed the raw dataset, tweak it, and publish our own fork, while automatically keeping the thread of attribution. The list goes on and on...

p4bl0 10y ago
Hi! I really think what you are doing here is great and I hope you will succeed! Is there a way to test the platform without creating to much garbage? Like a sandbox version maybe?
- joelg 10y ago
  Everything is open-sourced, and we've tried to document as much as possible to make it easy for people to host their own instances! Alternatively, you can just create arbitrarily many drafts without publishing any of them. https://github.com/pubpub/pubpub https://github.com/pubpub/pubpub
  - p4bl0 10y ago
    Running my own instance is a bit heavy just to try the platform :). But thanks for the tips, I can create draft and delete them without trashing your instance. The fact that it is open source is a major positive point. I suppose you have heard of SJS [1] which in a way is very close to what PubPub is, except I do not believe in SJS because of it being closed source: it is way too unsafe for a scientific community to take the risk of adopting such a media for a new journal or a conference as the platform is not open and there is a risk of Elsevier buying it to shut it down or closing it even more, for instance. [1] http://sjscience.org/ http://sjscience.org/

zyxzevn 10y ago

The peer-review is mixed with a reductionistic structure. Each scientific area is specialized. The articles are reviewed by people in the specialized fields. Often they find something that is not entirely in their field of knowledge. That is because they are trying to "science" something new. So they can produce theories that seem a solution from their field of knowledge, but are not valid in another field. The peers will not see this problem. Another problem is that engineers that try to apply this science find practical problems. But because they are seen as "of lesser knowledge" their practical criticism is often rejected. The best example I know is the "theory" that magnetic fields can bump into each other, producing energy. In this model the fields are made of flux lines, which can bumb into other flux lines. The flux lines will then break and reorganize, producing energy. Yet, as I write this, you may already think that this is bullshit, because flux-lines are imaginary lines to describe a magnetic field. Now, with this in mind, look at magnetic reconnection. This is a theory made scientists specialized in astronomy, and not in the area of electromagnetism. I believe that this problem is in every (specialized) area of science.

teslaberry 10y ago

Science (Doing and reviewing it)is an art, not a science.

Pinatubo 10y ago

There's a peer review scandal currently underway involving a top economics journal. https://gborjas.org/2016/06/30/a-rant-on-peer-review/ https://gborjas.org/2016/06/30/a-rant-on-peer-review/

aaxe 10y ago

www.Examine.com does a great job analyzing nutrition/supplements studies.

apathy 10y ago

Buried in the middle of this (wonderful) article is the heart of the problem -- academia has foolishly placed its metrics into the hands of editors and publishers, who have corrupted the living hell out of it. Ctrl-F "Cochrane" and witness the exchange between an editor, who benefited from the status quo, and a scholar, who did not and does not. Academia != scholarship and has not for some time. There is no longer a good reason for traditional for-profit journals to exist. (Before someone says it: SpringerNature likes to pretend that editorial independence is possible, but they'll have no choice save to fire their "news" guys if their board asks for this). Please recall that the entire point of the World Wide Web was to share physics papers. The arXiv exists because of physicists (who quickly noticed that HTML wasn't a good substitute for LaTeX when writing math-heavy papers). The problem is not technological. It's social. And until the incentives are fixed (the Cochrane Collaboration in Britain has gone a long, long way to address this, and now the Wellcome Trust is going even further), nothing of any real import will change. In the USA, NIH could make a lot of positive changes (and in exchanges with mid- to senior executive level directors, I honestly believe they're trying to do so). But it will take time. Academia moves with glacial speed, when it moves at all.

arca_vorago 10y ago

I'll preface by saying I'm only a sysadmin with a love of science, but I learned the most about peer review and scientific publishing while working in biotech at a genetics company. At some point I realized I actually had to have some understanding of the science to properly admin it, and began reading lots of papers, and I slowly started to realize just how bad many papers are. There are a few key issues: 1. Scientists who "collaborate" with other scientists but do a small fraction of actual work get their names on papers as number fodder. Anytime I hear "I've been published in over 1000 journal articles" now I generally become more skeptical. 2. Lack of reproducibility. Not only in the methods and the documentation of the methods, but also the fact that most things just simply aren't even tested by a third party. 3. Publishing. They have far too long locked up information the public deserves to know, which is bad enough, but then they do a bad job of it and allow bad science in. The largest part of the problem imho because they created the situation that semi forces the scientists into questionable paper writing tactics.