6 ms·

> Many researchers are privately terrified of being falsely accused by the "data cops," one scientist said. But no one wants to criticize them because no one wa

> Many researchers are privately terrified of being falsely accused by the "data cops," one scientist said. But no one wants to criticize them because no one wants a target on their own back — including this individual, who was granted anonymity for this reason. "Batman is a vigilante," the scientist said. "So is the Joker." > For collaborators, it's a stressful and infuriating time. When a paper is retracted, the research is erased, and all related citations are lost. > Others who attempted to build on Gino's studies are grappling with having wasted time, money, and energy. It feels, to me, that they're making Data Colada (the data vigilante group) the bad guys here. It's like blaming the Police for stopping a criminal because they took away the criminal's source of income... I'm kinda uneasy with this line of editing :/

tgv 3y ago
Indeed. That's nasty. They throw in a detracting bit of racist accusations as well. > all related citations are lost What an extreme BS argument. It's really grasping at straws.
- prepend 3y ago
  The citations aren’t lost though, are they. They still exist in the paper, they just point to a retraction. You can still find the papers that cited the retracted paper. And I suspect you’ll still get new citations since it’s not unheard of that people cite a paper based on seeing it in some other paper, rather than reading the paper directly. I don’t think people should do that, but they do.
  - tokai 3y ago
    The citations won't be counted by the big databases like scopus and web of science, which are often a prerequisite when being measured for grants etc.
    - casey2 3y ago
      Next to nothing related to scientific research is tracked in scopus and web of science anyway, so your point is moot.
      - tokai 3y ago
        I don't follow you. They track scientific publications and their citation networks. I feel like you are reading something in my post I didn't claim. My point was that the citations can be lost in the sense than they wont be counted by the bibliographic databases that are usually used by administrators to get citation numbers for evaluating researchers and institutions. Of course the citations still exist between the papers. But for a researcher their bibliometric indicators will come out lower after a redaction, in most cases when they are evaluated for grants or career progression.
        prepend 3y ago
        But that measurement isn’t done through Scopus or Web of Science. It’s done by evaluating biosketches of grant applicants in their application. It’s not like the grant reviewer checks Scopus for the applicants citation count or something. I recently reviewed some grant applications and I reviewed the applicants bio sketch including their experience and publications. I didnt count the citations for each paper. But if I did, I certainly wouldn’t count citations for any papers that were retracted.
    - prepend 3y ago
      Of course they shouldn’t be measured. They are recalled. Of course, I don’t think people with recalled papers due to their fraud need to ever concern themselves with grants. (Kind of like felons can never be an officer of a publicly traded company)
- nelgaard 3y ago
  Of course the citations should be lost (not counted). But the paragraph is about collaboraters being angry and frustrated. And they seem to be angry at those who faked the data. Which is understandable. You would hope that this would also mean that collaborators getting their name on a paper would put a little more effort into scutinizing the data provided by co-authors. Maybe it is just a little to easy to work on paper and just accept the underlying data without asking any questions.
bakuninsbart 3y ago
That narrative is insane. It is one thing to produce shoddy research. Even good senior researchers sometimes just have a paper that... doesn't work out well, with a late found flaw in the methods that can't be fully mitigated or uninteresting findings, and then they'd rather still try to publish it than shelve months of work. That's not good, but almost unavoidable in the current academic environment. The accusations here are very different: Producing fake data is fraudulent, of course the papers need to be retracted, and people involved and in the know need to be punished. The quote from the article describes well something that seems to be a growing group of people within academia: > It's possible "these people don't care and never cared about the science," Michael Sanders, a professor of public policy at King's College London, said. "They talk about science, and they talk about experiments, and they sort of wrap themselves in the fabric of the scientific method — just as a way of selling stuff."
- gcoakes 3y ago
  Is there a market for publishing the retrospective of the process which resulted in a "late found flaw"? I would expect uninteresting data would still be published if only to prevent someone from going down that rabbit hole in the future.
light_hue_1 3y ago
No one has ever been unfairly accused. The bar for being accused is so incredibly high. People are given so many possible outs and are allowed to produce the most implausible explanations. And then, even having one or two faked papers basically has no impact. Here's an example. Ken Rogoff and Carmen Reinhart. Professors at Harvard. Despite having faked the paper that introduced austerity to the world, had an immense impact on the economy, and caused massive and needless suffering for hundreds of millions of people. That paper still continues to haunt countries today. To make their paper work they had to lie 3 times: to exclude some countries, exclude data from others selectively, and reweight all of the data. None of this is reported in their paper! Each of these took deliberate actions on their part, they cannot be mistakes. Every single "mistake" they made is carefully calculated to produce the result they want. Every one of them is needed otherwise their paper would be junk. What happened to them? Well, Reinhart has been promoted to the Chief Economist of the World Bank and is still a professor at Harvard. Rogoff is still a professor at Harvard too. Their faked paper literally resulted in deaths, toppled governments, and the rise of the alt right in places like Greece. But they got a promotion for it. The only people who are afraid are the people who are faking their data. No real scientist is getting accused of anything here.
- lukeasrodgers 3y ago
  What do you make of Reinhart and Rogoff’s rebuttal to this criticism? They acknowledge the excel error but dispute the rest. https://www.nytimes.com/2013/04/26/opinion/reinhart-and-rogoff-responding-to-our-critics.html https://www.nytimes.com/2013/04/26/opinion/reinhart-and-rogo...
  - light_hue_1 3y ago
    Look at it from the point of view a scientist. They published a paper that claimed they ran analysis X. Reviewers look at it, and believed them, because we usually don't assume that people are lying. And it's hard to read over every line of code and compare it to the paper to make sure they did run the analysis they claimed. But in reality, they did a totally different analysis. They hid data, they fudged data, they inserted weighting factors into the analysis. And they never mentioned in this in the paper. In the article, they claim this was justified. But hiding the true nature of the analysis they did was not. That hiding, of making the paper read one way but the code another, is plan and simple scientific fraud. If what they believed in reality is that the code is correct, and that's the better analysis, then they should have just reported in the paper. The reason they didn't, is because reviewers would have looked at it and said that this is unacceptable and doesn't meet basic scientific standards. So they lied.
Workaccount2 3y ago
I wonder if it's like a steroid doping situation: What kinds of steroids and how much of them can I get away with using so I'm in the perfect spot where I am ahead my fellow doping athletes, but still under the bar of detection...
- s1artibartfast 3y ago
  There's a pretty clear solution for that in athletics which is sometimes implemented. They take blood samples and store them for decades. New methods can come along and analyze the data and detect fraud long after the fact. The same can be done in science, it just more disclosure of data.
  - taeric 3y ago
    This doesn't really help, though? My understanding is that in the sports where doping was a thing, if you wanted to be competitive at top level, you basically had to find a way to dope. This isn't something 'just the winners' did. Near everyone was doing it. So, yeah, you can look for it with later testing, but all that will do is claw back a winner. It doesn't help anyone that wasn't doing it. Is why none of us have any clue who the best cyclist during the Armstrong era was. With the scary view that the cheating was so widespread that the current winners were still the best if everyone stopped cheating.
    - s1artibartfast 3y ago
      Of course you can't fix the past before they started retaining samples. The point is to prevent people from doping moving forward because they know they can be caught. If we had better data on cyclists from when Armstrong was racing, you could absolutely claw back the Awards and give them to the best cyclist that wasn't doping.
      - taeric 3y ago
        I think that misses just how entrenched doping was. As mind blowing as it is, it is fairly safe to say that everyone that was making the leaderboards was likely cheating.
next_xibalba 3y ago
> it's like blaming the Police for stopping a criminal because they took away the criminal's source of income Coming soon to a San Francisco near you!
- peterfirefly 3y ago
  It's obviously already there in parts of California/Oregon.
burkaman 3y ago
Everyone, please read the whole article, not just this comment. The second two quotes here appear very out of context in this comment - they are about anger and frustration towards Gino, not towards Data Colada. The first quote here is not a defense of Gino, it's a nuanced opinion of self-proclaimed vigilantes who are also human and might not always make perfect decisions. It is followed by this: > Discussing the allegations that Gino falsified data is seen almost unanimously as fair game, especially in light of Harvard putting her on leave. People expressed more mixed feelings toward disgraced researchers such as Cuddy, who failed to replicate her findings but did not fake her data. I have no special insight here, but this at least seems like a reasonable thing to discuss. > It feels, to me, that they're making Data Colada (the data vigilante group) the bad guys here. I strongly disagree. The light questioning of Data Colada was a very small part of the article, and even that section did not give me the impression that they are "bad guys".
- jonhohle 3y ago
  If the findings can’t be replicated, were they worth publishing to begin with? I know there is pressure to publish, but the effect on society is significantly greater than a single researcher’s personal job security. If nothing else, papers with expensive or one-off experiments should be clearly marked as observations that should not be relied on as foundational for other work until replicated. When they’re cited peer reviewers could make sure they are referenced as such without requiring the peer reviewer to intimately know the details of the reference.
  - the_snooze 3y ago
    I'm in a different field (computer security and privacy) and I want to point out that the notion of "replication" is trickier than what you might expect. Let's say I do a simple empircal measurement study, say on network traffic on some piece of internet infrastructure day-by-day in the year 2020. I find there's a huge sustained increase starting in mid-March, sustaining through the end of the year. I submit my results for publication. How can someone replicate those results, when the measurements were subject to conditions unique to that observation period? The best anyone can do is to replicate my statistics/analysis if I gave them my data set. That's different notion of replication. Would you consider that sufficient?
lo_zamoyski 3y ago
Guilt is a very effective way of keeping corrupt status quos going. Even if you change your ways, there is still dirt on you. The only way to break the cycle is if someone from the outside exposes the filth, or someone guilty on the inside decides to pick up his cross and accept the consequences of admission and exposure. On top of that, there are many who won't even entertain the possibility that the accused could be guilty even with evidence, because they have an emotional interest in preserving a false image of decency. Or they're terrified of the consequences of reporting the problem. This is what happens with sex abuse. When there's enough guilt to go around in the right places, the guilty form a kind of hellish camaraderie, a Mexican standoff of pointing fingers. Once such people assume positions of power, they also have a tendency to concentrate corruption in the institution by favoring the corrupt. Doing so entrenches the standoff and eliminates those who would oppose or expose them. The so-called "lavender mafia" is a well-known example, because the media hate nothing more than the RCC, but this occurs everywhere and at greater rates. What's important is what we distinguish between institutions and the people that temporarily occupy them. Good institutions should generally be preserved. It's the corruption that should be dealt with. Of course, that's not always an option. Sometimes, corruption destroys the institution, or perhaps the institution is beyond repair. But generally, there is a presumption in favor of cleaning house rather than abolition.

tgv 3y ago

Indeed. That's nasty. They throw in a detracting bit of racist accusations as well. > all related citations are lost What an extreme BS argument. It's really grasping at straws.

prepend 3y ago
The citations aren’t lost though, are they. They still exist in the paper, they just point to a retraction. You can still find the papers that cited the retracted paper. And I suspect you’ll still get new citations since it’s not unheard of that people cite a paper based on seeing it in some other paper, rather than reading the paper directly. I don’t think people should do that, but they do.
- tokai 3y ago
  The citations won't be counted by the big databases like scopus and web of science, which are often a prerequisite when being measured for grants etc.
  - casey2 3y ago
    Next to nothing related to scientific research is tracked in scopus and web of science anyway, so your point is moot.
    - tokai 3y ago
      I don't follow you. They track scientific publications and their citation networks. I feel like you are reading something in my post I didn't claim. My point was that the citations can be lost in the sense than they wont be counted by the bibliographic databases that are usually used by administrators to get citation numbers for evaluating researchers and institutions. Of course the citations still exist between the papers. But for a researcher their bibliometric indicators will come out lower after a redaction, in most cases when they are evaluated for grants or career progression.
      - prepend 3y ago
        But that measurement isn’t done through Scopus or Web of Science. It’s done by evaluating biosketches of grant applicants in their application. It’s not like the grant reviewer checks Scopus for the applicants citation count or something. I recently reviewed some grant applications and I reviewed the applicants bio sketch including their experience and publications. I didnt count the citations for each paper. But if I did, I certainly wouldn’t count citations for any papers that were retracted.
  - prepend 3y ago
    Of course they shouldn’t be measured. They are recalled. Of course, I don’t think people with recalled papers due to their fraud need to ever concern themselves with grants. (Kind of like felons can never be an officer of a publicly traded company)
nelgaard 3y ago
Of course the citations should be lost (not counted). But the paragraph is about collaboraters being angry and frustrated. And they seem to be angry at those who faked the data. Which is understandable. You would hope that this would also mean that collaborators getting their name on a paper would put a little more effort into scutinizing the data provided by co-authors. Maybe it is just a little to easy to work on paper and just accept the underlying data without asking any questions.

bakuninsbart 3y ago

That narrative is insane. It is one thing to produce shoddy research. Even good senior researchers sometimes just have a paper that... doesn't work out well, with a late found flaw in the methods that can't be fully mitigated or uninteresting findings, and then they'd rather still try to publish it than shelve months of work. That's not good, but almost unavoidable in the current academic environment. The accusations here are very different: Producing fake data is fraudulent, of course the papers need to be retracted, and people involved and in the know need to be punished. The quote from the article describes well something that seems to be a growing group of people within academia: > It's possible "these people don't care and never cared about the science," Michael Sanders, a professor of public policy at King's College London, said. "They talk about science, and they talk about experiments, and they sort of wrap themselves in the fabric of the scientific method — just as a way of selling stuff."

gcoakes 3y ago
Is there a market for publishing the retrospective of the process which resulted in a "late found flaw"? I would expect uninteresting data would still be published if only to prevent someone from going down that rabbit hole in the future.

light_hue_1 3y ago

No one has ever been unfairly accused. The bar for being accused is so incredibly high. People are given so many possible outs and are allowed to produce the most implausible explanations. And then, even having one or two faked papers basically has no impact. Here's an example. Ken Rogoff and Carmen Reinhart. Professors at Harvard. Despite having faked the paper that introduced austerity to the world, had an immense impact on the economy, and caused massive and needless suffering for hundreds of millions of people. That paper still continues to haunt countries today. To make their paper work they had to lie 3 times: to exclude some countries, exclude data from others selectively, and reweight all of the data. None of this is reported in their paper! Each of these took deliberate actions on their part, they cannot be mistakes. Every single "mistake" they made is carefully calculated to produce the result they want. Every one of them is needed otherwise their paper would be junk. What happened to them? Well, Reinhart has been promoted to the Chief Economist of the World Bank and is still a professor at Harvard. Rogoff is still a professor at Harvard too. Their faked paper literally resulted in deaths, toppled governments, and the rise of the alt right in places like Greece. But they got a promotion for it. The only people who are afraid are the people who are faking their data. No real scientist is getting accused of anything here.

lukeasrodgers 3y ago
What do you make of Reinhart and Rogoff’s rebuttal to this criticism? They acknowledge the excel error but dispute the rest. https://www.nytimes.com/2013/04/26/opinion/reinhart-and-rogoff-responding-to-our-critics.html https://www.nytimes.com/2013/04/26/opinion/reinhart-and-rogo...
- light_hue_1 3y ago
  Look at it from the point of view a scientist. They published a paper that claimed they ran analysis X. Reviewers look at it, and believed them, because we usually don't assume that people are lying. And it's hard to read over every line of code and compare it to the paper to make sure they did run the analysis they claimed. But in reality, they did a totally different analysis. They hid data, they fudged data, they inserted weighting factors into the analysis. And they never mentioned in this in the paper. In the article, they claim this was justified. But hiding the true nature of the analysis they did was not. That hiding, of making the paper read one way but the code another, is plan and simple scientific fraud. If what they believed in reality is that the code is correct, and that's the better analysis, then they should have just reported in the paper. The reason they didn't, is because reviewers would have looked at it and said that this is unacceptable and doesn't meet basic scientific standards. So they lied.

Workaccount2 3y ago

I wonder if it's like a steroid doping situation: What kinds of steroids and how much of them can I get away with using so I'm in the perfect spot where I am ahead my fellow doping athletes, but still under the bar of detection...

s1artibartfast 3y ago
There's a pretty clear solution for that in athletics which is sometimes implemented. They take blood samples and store them for decades. New methods can come along and analyze the data and detect fraud long after the fact. The same can be done in science, it just more disclosure of data.
- taeric 3y ago
  This doesn't really help, though? My understanding is that in the sports where doping was a thing, if you wanted to be competitive at top level, you basically had to find a way to dope. This isn't something 'just the winners' did. Near everyone was doing it. So, yeah, you can look for it with later testing, but all that will do is claw back a winner. It doesn't help anyone that wasn't doing it. Is why none of us have any clue who the best cyclist during the Armstrong era was. With the scary view that the cheating was so widespread that the current winners were still the best if everyone stopped cheating.
  - s1artibartfast 3y ago
    Of course you can't fix the past before they started retaining samples. The point is to prevent people from doping moving forward because they know they can be caught. If we had better data on cyclists from when Armstrong was racing, you could absolutely claw back the Awards and give them to the best cyclist that wasn't doping.
    - taeric 3y ago
      I think that misses just how entrenched doping was. As mind blowing as it is, it is fairly safe to say that everyone that was making the leaderboards was likely cheating.

next_xibalba 3y ago

> it's like blaming the Police for stopping a criminal because they took away the criminal's source of income Coming soon to a San Francisco near you!

peterfirefly 3y ago
It's obviously already there in parts of California/Oregon.

burkaman 3y ago

Everyone, please read the whole article, not just this comment. The second two quotes here appear very out of context in this comment - they are about anger and frustration towards Gino, not towards Data Colada. The first quote here is not a defense of Gino, it's a nuanced opinion of self-proclaimed vigilantes who are also human and might not always make perfect decisions. It is followed by this: > Discussing the allegations that Gino falsified data is seen almost unanimously as fair game, especially in light of Harvard putting her on leave. People expressed more mixed feelings toward disgraced researchers such as Cuddy, who failed to replicate her findings but did not fake her data. I have no special insight here, but this at least seems like a reasonable thing to discuss. > It feels, to me, that they're making Data Colada (the data vigilante group) the bad guys here. I strongly disagree. The light questioning of Data Colada was a very small part of the article, and even that section did not give me the impression that they are "bad guys".

jonhohle 3y ago
If the findings can’t be replicated, were they worth publishing to begin with? I know there is pressure to publish, but the effect on society is significantly greater than a single researcher’s personal job security. If nothing else, papers with expensive or one-off experiments should be clearly marked as observations that should not be relied on as foundational for other work until replicated. When they’re cited peer reviewers could make sure they are referenced as such without requiring the peer reviewer to intimately know the details of the reference.
- the_snooze 3y ago
  I'm in a different field (computer security and privacy) and I want to point out that the notion of "replication" is trickier than what you might expect. Let's say I do a simple empircal measurement study, say on network traffic on some piece of internet infrastructure day-by-day in the year 2020. I find there's a huge sustained increase starting in mid-March, sustaining through the end of the year. I submit my results for publication. How can someone replicate those results, when the measurements were subject to conditions unique to that observation period? The best anyone can do is to replicate my statistics/analysis if I gave them my data set. That's different notion of replication. Would you consider that sufficient?

lo_zamoyski 3y ago

Guilt is a very effective way of keeping corrupt status quos going. Even if you change your ways, there is still dirt on you. The only way to break the cycle is if someone from the outside exposes the filth, or someone guilty on the inside decides to pick up his cross and accept the consequences of admission and exposure. On top of that, there are many who won't even entertain the possibility that the accused could be guilty even with evidence, because they have an emotional interest in preserving a false image of decency. Or they're terrified of the consequences of reporting the problem. This is what happens with sex abuse. When there's enough guilt to go around in the right places, the guilty form a kind of hellish camaraderie, a Mexican standoff of pointing fingers. Once such people assume positions of power, they also have a tendency to concentrate corruption in the institution by favoring the corrupt. Doing so entrenches the standoff and eliminates those who would oppose or expose them. The so-called "lavender mafia" is a well-known example, because the media hate nothing more than the RCC, but this occurs everywhere and at greater rates. What's important is what we distinguish between institutions and the people that temporarily occupy them. Good institutions should generally be preserved. It's the corruption that should be dealt with. Of course, that's not always an option. Sometimes, corruption destroys the institution, or perhaps the institution is beyond repair. But generally, there is a presumption in favor of cleaning house rather than abolition.