6 ms·

Everyone is being told to put faith into "science" as somehow incorruptible, only to have articles like this written. We are supposed to be able to rely on the

Everyone is being told to put faith into "science" as somehow incorruptible, only to have articles like this written. We are supposed to be able to rely on the people writing papers being ethical, when it seems the reality is a race to the bottom like everywhere else in society.

Thriptic 7y ago
Well, the problem in this case isn't really in the scientific or peer review process itself, it's in using citation count as a metric for paper quality or researcher expertise. I've had people play this game on several of my papers under review. Reviews will say something like "high quality submission but the authors should consider including work X for completeness" etc where it's obvious that work X was written by the reviewer. As an author, I just found some place to plausibly cite the work and moved on. The quality or validity of my manuscript was not impacted in any way. In my admittedly limited experience, reviewers will rarely green light obviously inferior work just to get an additional citation as it impacts their reputation and it's an inefficient way to stat pad (self-citation or trading citations with your collaborators is by far the most efficient way to pump citation count, especially in the context of review papers). Additionally, obviously inferior works don't even make it to the peer review process; they are rejected by the journal editors in a first pass screen. There are more insidious games people play that DO impact publication quality, such as requesting friendly reviewers (this mechanism is normally legitimately used to help editors route manuscripts for review to domain experts), submitting work to journals where the PI is an editor / has relationships with editors, attaching well known PIs to manuscripts to add the appearance of legitimacy even though they did no work and were not involved at all, etc. These are problems that won't go away because there are far too many researchers and not enough money / positions to go around, so people are heavily incentivized to play games.
kerkeslager 7y ago
Science is incorruptible. Scientists are not. Don't conflate the imperfections of the humans implementing science as being problems with science. Whatever you're proposing as the alternative to science has the same problem: it's implemented by imperfect humans. What's your alternative?
- xhkkffbf 7y ago
  Ah, I wouldn't be so sure. John Ioannidis's work does a great job of showing how science as we've defined it can generate results that are wrong. There are plenty of phenomena that can't be easily sussed out using the scientific method of controlled experimentation. The simplest example is if you need two parts A AND B to generate an effect. If you add A, you get nothing. If you add B, you get nothing. The scientific method suggests that both A and B do nothing. It takes a bit more clever work to recognize that both are required, but the scientific method insists on simplifying the model. The point is that even science isn't well-defined and we need to be super flexible.
  - kerkeslager 7y ago
    > The scientific method suggests that both A and B do nothing. It takes a bit more clever work to recognize that both are required, but the scientific method insists on simplifying the model. Huh? No it doesn't. This is complete nonsense. The scientific method would deal with this by hypothesizing that A combined with B might do something, then testing the hypothesis, and discovering that it does. Generating that hypothesis might be difficult--nobody said science was easy. But there's nothing about the scientific method that can't handle the situation you propose. There are hundreds of examples of this. Try, for example, hydrochloric acid and baking soda as your A and B--I've never mixed these two and don't know what they produce, but based on my knowledge of acids and baking soda, I hypothesize they will result in CO2, even though neither of these decomposes into that gas by itself.
- einpoklum 7y ago
  > Science is incorruptible You're taking that a tad too far. Remember, for long periods of time, certain disciplines were considered to be "science" which weren't quite that. It's not just individual personal failings.
  - kerkeslager 7y ago
    Okay, but that's a semantic argument. The word "science" doesn't mean that any more.
jefft255 7y ago
Scientists are human and guidelines dictating acceptable behavior must exists and be enforced; just like any other field of work. A more open peer-review system would help this issue and expose abusers like this, but it's not clear (to me) how such a system could work... At least this issue does not directly affect the quality of research, but rather corrupt the already noisy quality metric of citation count.
- truncate 7y ago
  I think open peer-review system would be great. It would also keep reviewers much more careful and accountable for their work. It is a bit different than design docs/code review in the sense that, there are N different authors competing to grab a place in conference or journal -- so they may try to game the system, while in case of design docs/codereview there is no such competition.
cannonedhamster 7y ago
The difference is with science the truth will eventually out. No one says science is incorruptible, but the process of science means that it generally keeps moving in the correct direction, despite normal human tendencies. The other part is that when science is found to be wrong, scientists change their assumptions based on new information. There's numerous examples of science being used for ill. Light bulbs that burn out faster because of a cabal of bulb makers, functional vaccines for diseases locked up for years unused, and mistakes like radiation poisoning of workers and people because we just didn't understand the ramifications at the time. All of these were eventually corrected by other scientists. Now specific to writing papers, paper writing is required for some educational job progress. The journals are supposed to be ethical, however this problem has been known about and pointed out numerous times by ethical scientists.
bonoboTP 7y ago
I agree that the general public has basically no idea how the academic world operates and surely their trust in it would decrease if they knew. There are many sketchy practices and a lot is based on an "honor" system, which often devolves to back scratching. Lots of inflated claims for sexy publicity, mass produced PhDs etc. People imagine science as this serious rigorous thing where everything is objective and unbiased careful "monks" investigate all the details with scrutiny. When actually it's often like a bazaar. You have to sway the reviewer with overinflated claims and "sell yourself" in a way that fully contradicts the scientific virtue of self-skepticism (finding reasons why you're wrong, not right). There's also a lot of who knows/likes who and subjective human factors. These are all obvious for people after a few months or 1-2 years as a researcher, but science has a very different image for laypeople who mostly know about it from school, mostly results that were produced in a different era. The number of scientists has exploded recently, there are tons and tons of papers written and they are all supposed to be some novel discovery and this great leap forward, which is clearly unrealistic at this scale. At the same time thorough analysis and skeptical papers are "boo"-ed down, seen as confrontational and are undervalued compared to sexy new claims. Many institutions have become paper mills. Raw metrics like paper counts and citation counts have replaced careful consideration in many evaluations, it's basically an industrial scale mechanized process. Feels like a house of cards that can not be kept up indefinitely. The reputation of science will be hurt for sure as the public starts to find out more.
- conjectures 7y ago
  On the other hand, to list some random newish things: * Genome sequencing. * New malaria drugs. * Graphene. * Better batteries. * Deep learning. * fMRI. * CRISPR. From the outside, the public doesn't care about the trials and tribulations of PhDs. Or irrelevant puffed up papers. Or the organisational politics of research.
  - bonoboTP 7y ago
    That's true, there is a lot of good work out there of course. And I'd say the system is still of concern for the public. Few readers of pop science articles recognize when they are essentially reading a pr marketing piece, and not something like a consensus textbook-like description of consolidated knowledge. The authors have special interests and biases beyond uncovering the truth, such as pressure for publication counts, tenure possibilities, filling your CV, all sorts of political considerations, flag planting, etc. Another thing is how these discoveries are made. I think people imagine it as if researchers research and ponder about things, then when they find something, they write it up to share it with the world and contribute their new found interesting knowledge. I know someone who did that in his PhD, he quit after several years without any paper submissions. When asked, he said he hasn't found anything yet worth publishing. In most places, people decide in advance that they're going to have a paper this year for this journal/conference and then they plan a solid way there. You decide what result would be sexy (keeping some secondary plans in place) and then do the whole thing. Several times things will not go as you expect, but the paper must be written. Then you rewrite history and come up with ridiculous "storylines" to make it seem like all your hypotheses naturally flow from prior work and everything fits in this nice story. Then you torture the data until it sings because there must be a paper. You look for the smallest indication of something interesting, inflate it to sound like a breakthrough, speculate about far reaching conclusions etc. And then if your university has a good pr department, journalists will eat it all up and you'll be out in the media, impacting normal people's understanding as well. (I realize I may be a bit too cynical here, and I exaggerate somewhat to get the message through.)
conjectures 7y ago
Massive misundertanding here: Science doesn't work by noble individuals scrupulously following rules that they could ignore if they chose to. Where science works, it works by incentivising one scientist to tear apart the next scientist's position. There's no better way to avoid having your argument torn apart than for it to be genuinely solid. This is aptly demonstrated by Nature tearing into Elsevier... The time to get really worried is when you stop hearing anything like this and get a message only of, 'Everything is fine, nothing to see here'.
- refurb 7y ago
  Where science works, it works by incentivising one scientist to tear apart the next scientist's position. This is very true. In my graduate education that came out loud and clear - your job is to try and punch holes in other people’s work. If you find a hole, it’s their job to fill it. That’s what good science looks like.

Thriptic 7y ago

Well, the problem in this case isn't really in the scientific or peer review process itself, it's in using citation count as a metric for paper quality or researcher expertise. I've had people play this game on several of my papers under review. Reviews will say something like "high quality submission but the authors should consider including work X for completeness" etc where it's obvious that work X was written by the reviewer. As an author, I just found some place to plausibly cite the work and moved on. The quality or validity of my manuscript was not impacted in any way. In my admittedly limited experience, reviewers will rarely green light obviously inferior work just to get an additional citation as it impacts their reputation and it's an inefficient way to stat pad (self-citation or trading citations with your collaborators is by far the most efficient way to pump citation count, especially in the context of review papers). Additionally, obviously inferior works don't even make it to the peer review process; they are rejected by the journal editors in a first pass screen. There are more insidious games people play that DO impact publication quality, such as requesting friendly reviewers (this mechanism is normally legitimately used to help editors route manuscripts for review to domain experts), submitting work to journals where the PI is an editor / has relationships with editors, attaching well known PIs to manuscripts to add the appearance of legitimacy even though they did no work and were not involved at all, etc. These are problems that won't go away because there are far too many researchers and not enough money / positions to go around, so people are heavily incentivized to play games.

kerkeslager 7y ago

Science is incorruptible. Scientists are not. Don't conflate the imperfections of the humans implementing science as being problems with science. Whatever you're proposing as the alternative to science has the same problem: it's implemented by imperfect humans. What's your alternative?

xhkkffbf 7y ago
Ah, I wouldn't be so sure. John Ioannidis's work does a great job of showing how science as we've defined it can generate results that are wrong. There are plenty of phenomena that can't be easily sussed out using the scientific method of controlled experimentation. The simplest example is if you need two parts A AND B to generate an effect. If you add A, you get nothing. If you add B, you get nothing. The scientific method suggests that both A and B do nothing. It takes a bit more clever work to recognize that both are required, but the scientific method insists on simplifying the model. The point is that even science isn't well-defined and we need to be super flexible.
- kerkeslager 7y ago
  > The scientific method suggests that both A and B do nothing. It takes a bit more clever work to recognize that both are required, but the scientific method insists on simplifying the model. Huh? No it doesn't. This is complete nonsense. The scientific method would deal with this by hypothesizing that A combined with B might do something, then testing the hypothesis, and discovering that it does. Generating that hypothesis might be difficult--nobody said science was easy. But there's nothing about the scientific method that can't handle the situation you propose. There are hundreds of examples of this. Try, for example, hydrochloric acid and baking soda as your A and B--I've never mixed these two and don't know what they produce, but based on my knowledge of acids and baking soda, I hypothesize they will result in CO2, even though neither of these decomposes into that gas by itself.
einpoklum 7y ago
> Science is incorruptible You're taking that a tad too far. Remember, for long periods of time, certain disciplines were considered to be "science" which weren't quite that. It's not just individual personal failings.
- kerkeslager 7y ago
  Okay, but that's a semantic argument. The word "science" doesn't mean that any more.

jefft255 7y ago

Scientists are human and guidelines dictating acceptable behavior must exists and be enforced; just like any other field of work. A more open peer-review system would help this issue and expose abusers like this, but it's not clear (to me) how such a system could work... At least this issue does not directly affect the quality of research, but rather corrupt the already noisy quality metric of citation count.

truncate 7y ago
I think open peer-review system would be great. It would also keep reviewers much more careful and accountable for their work. It is a bit different than design docs/code review in the sense that, there are N different authors competing to grab a place in conference or journal -- so they may try to game the system, while in case of design docs/codereview there is no such competition.

cannonedhamster 7y ago

The difference is with science the truth will eventually out. No one says science is incorruptible, but the process of science means that it generally keeps moving in the correct direction, despite normal human tendencies. The other part is that when science is found to be wrong, scientists change their assumptions based on new information. There's numerous examples of science being used for ill. Light bulbs that burn out faster because of a cabal of bulb makers, functional vaccines for diseases locked up for years unused, and mistakes like radiation poisoning of workers and people because we just didn't understand the ramifications at the time. All of these were eventually corrected by other scientists. Now specific to writing papers, paper writing is required for some educational job progress. The journals are supposed to be ethical, however this problem has been known about and pointed out numerous times by ethical scientists.

bonoboTP 7y ago

I agree that the general public has basically no idea how the academic world operates and surely their trust in it would decrease if they knew. There are many sketchy practices and a lot is based on an "honor" system, which often devolves to back scratching. Lots of inflated claims for sexy publicity, mass produced PhDs etc. People imagine science as this serious rigorous thing where everything is objective and unbiased careful "monks" investigate all the details with scrutiny. When actually it's often like a bazaar. You have to sway the reviewer with overinflated claims and "sell yourself" in a way that fully contradicts the scientific virtue of self-skepticism (finding reasons why you're wrong, not right). There's also a lot of who knows/likes who and subjective human factors. These are all obvious for people after a few months or 1-2 years as a researcher, but science has a very different image for laypeople who mostly know about it from school, mostly results that were produced in a different era. The number of scientists has exploded recently, there are tons and tons of papers written and they are all supposed to be some novel discovery and this great leap forward, which is clearly unrealistic at this scale. At the same time thorough analysis and skeptical papers are "boo"-ed down, seen as confrontational and are undervalued compared to sexy new claims. Many institutions have become paper mills. Raw metrics like paper counts and citation counts have replaced careful consideration in many evaluations, it's basically an industrial scale mechanized process. Feels like a house of cards that can not be kept up indefinitely. The reputation of science will be hurt for sure as the public starts to find out more.

conjectures 7y ago
On the other hand, to list some random newish things: * Genome sequencing. * New malaria drugs. * Graphene. * Better batteries. * Deep learning. * fMRI. * CRISPR. From the outside, the public doesn't care about the trials and tribulations of PhDs. Or irrelevant puffed up papers. Or the organisational politics of research.
- bonoboTP 7y ago
  That's true, there is a lot of good work out there of course. And I'd say the system is still of concern for the public. Few readers of pop science articles recognize when they are essentially reading a pr marketing piece, and not something like a consensus textbook-like description of consolidated knowledge. The authors have special interests and biases beyond uncovering the truth, such as pressure for publication counts, tenure possibilities, filling your CV, all sorts of political considerations, flag planting, etc. Another thing is how these discoveries are made. I think people imagine it as if researchers research and ponder about things, then when they find something, they write it up to share it with the world and contribute their new found interesting knowledge. I know someone who did that in his PhD, he quit after several years without any paper submissions. When asked, he said he hasn't found anything yet worth publishing. In most places, people decide in advance that they're going to have a paper this year for this journal/conference and then they plan a solid way there. You decide what result would be sexy (keeping some secondary plans in place) and then do the whole thing. Several times things will not go as you expect, but the paper must be written. Then you rewrite history and come up with ridiculous "storylines" to make it seem like all your hypotheses naturally flow from prior work and everything fits in this nice story. Then you torture the data until it sings because there must be a paper. You look for the smallest indication of something interesting, inflate it to sound like a breakthrough, speculate about far reaching conclusions etc. And then if your university has a good pr department, journalists will eat it all up and you'll be out in the media, impacting normal people's understanding as well. (I realize I may be a bit too cynical here, and I exaggerate somewhat to get the message through.)

conjectures 7y ago

Massive misundertanding here: Science doesn't work by noble individuals scrupulously following rules that they could ignore if they chose to. Where science works, it works by incentivising one scientist to tear apart the next scientist's position. There's no better way to avoid having your argument torn apart than for it to be genuinely solid. This is aptly demonstrated by Nature tearing into Elsevier... The time to get really worried is when you stop hearing anything like this and get a message only of, 'Everything is fine, nothing to see here'.

refurb 7y ago
Where science works, it works by incentivising one scientist to tear apart the next scientist's position. This is very true. In my graduate education that came out loud and clear - your job is to try and punch holes in other people’s work. If you find a hole, it’s their job to fill it. That’s what good science looks like.