5 ms·

Alternatively there is a baseline of fraudulent behavior in any human organization of 1-5% and since there are tens of thousands of high-profile researchers thi

Alternatively there is a baseline of fraudulent behavior in any human organization of 1-5% and since there are tens of thousands of high-profile researchers this sort of thing is inevitable. The question you should be asking is whether the field is able to correct and address its mistakes. Ironically cases like this one are the success stories: we don’t have enough data to know how many cases we’re missing.

panarky 3y ago
The very fact that the fraud is discovered, that reporters amplify it, and that it can bring down the president of the university, is evidence to me that the system still works.
- quickcheque 3y ago
  Sure, any system with a false negative and false positive rate 'works'.
- wpietri 3y ago
  Maybe? I'd want to see a clear model of flows and selection biases before I concluded that. Another way to look at it: perhaps Tessier-Lavigne only got this scrutiny because he was president of the university. And the fact that they didn't guarantee anonymity when "not guaranteeing anonymity in an investigation of this importance is an 'extremely unusual move'" might be a sign that the scrutiny was politically diminished. So it could be that most of the equally dubious researchers don't get caught because not enough attention is paid to patterns like this except when it's somebody especially prominent. Or it could be that this one was not as well covered up, perhaps because of the sheer number of issues. Or that the cross-institution issues made Stanford more willing to note the wrongdoing. Or that Stanford is less likely to sweep things under the rug because of its prominence. Or just that there was some ongoing tension between the trustees and the president and that this was an opportunity to win a political fight.
  - nequo 3y ago
    These are good points and hard to know. But the Retraction Watch is tracking stories of both mistakes and fraud in published research, across universities: https://retractionwatch.com/ https://retractionwatch.com/
- mike_hearn 3y ago
  A tenacious undergrad doing journalism as a hobby is not a system.
  - panarky 3y ago
    The fate of the world lies in the hands of the young and inexperienced. Grad students, Supreme Court clerks, 19-year-old soldiers.
karaterobot 3y ago
The problem is that we don't know what the baseline really is. We know that between a third and a half of results from peer reviewed papers in many domains cannot be replicated. Looking closer, we see what look like irregularities in some of them, but it's harder to say which of them are fraud, which are honest mistakes, and which of them just can't be replicated due to some other factors. But because so many of these studies just don't pay off for one reason or another, I would agree that it is getting really hard to rely on a process which is, if nothing else, supposed to result in reliable and trustworthy information.
- cycomanic 3y ago
  Where is that number of 1/3-1/2 coming from? And which fields? I find that very hard to believe (at least if we exclude the obvious fraudulent journals, where no actual research gets published)
  - zeroCalories 3y ago
    I think he's referencing the replication crisis that was a big deal a few years ago. Psychology was hit hard(unsurprising), but a few other fields in the biology area were also hit.
    - karaterobot 3y ago
      It's worst in Psychology and the Social Sciences, but it's not limited to them. Per Wikipedia: > A 2016 survey by Nature on 1,576 researchers who took a brief online questionnaire on reproducibility found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results (including 87% of chemists, 77% of biologists, 69% of physicists and engineers, 67% of medical researchers, 64% of earth and environmental scientists, and 62% of all others), and more than half have failed to reproduce their own experiments. But fewer than 20% had been contacted by another researcher unable to reproduce their work. The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists. Most researchers said they still trust the published literature Not sure if the results of that online study have (or can) themselves be reproduced, however. It's turtles all the way down.
hayd 3y ago
It's only a story because he's president, if he were only a researcher/professor this would not even be a story. This is NOT a success story, it shows that this fraudulent behavior is endemic and an effective strategy for climbing the academic ladder. A success story would be this is exposed at large... we work out some kind of effective peer-reproduced tests... and the hundreds/thousands of cheating professors are fired.
- moab 3y ago
  So just because one person is cheating, it means all academics are cheating? FWIW, most top-ranked CS conferences have an artifact evaluation track, and it doesn't look good if you submit an experimental paper and don't go through the artifact evaluation process. Things are certainly changing in CS, at least on the experimental side. It's also possible that theorems are incorrect, but subsequent work that figures this out will comment on it and fix it. The scientific record is self-correcting, and fraud / bullshit does get caught out.
  - hayd 3y ago
    It's not just "one person", there is wide-spread fraud across many disciplines of academia. The situation, of course, is vastly different across subjects/disciplines, e.g. math and CS are not really much affected and I would agree they're self-correcting. I might agree they're self-correcting in the (very) long-term, but we're seeing fictitious results fund entire careers. We don't know the damage that having 20+ years of incorrect results being built upon will have... And that's not to speak of those who were overlooked, and left academia, because their opportunities were taken by these cheaters (who knows what cost that has for society).
- tptacek 3y ago
  Endemic means "regularly occurring". How many examples of this kind of misconduct are you aware of? Ok, now, what's the denominator? How much research is actually conducted? I'm personally familiar with 3 fields (CS, bio, and geology) and what I've learned is that the number of labs --- let alone projects --- is mind-boggling. If your examples constitute 1% of all research conducted --- which would represent a cosmic-scale fuckload of research projects --- how much should I care about it?
whydoyoucare 3y ago
If a field takes two decades to "correct" its mistakes, then there are several things wrong with it. And if we have top positions held by unethical people, who have got away with it, and possibly climbed to the top because of it, then I do not know what to feel or say about this.
- FrustratedMonky 3y ago
  It's taken String Theory a few decades to correct itself.
vintermann 3y ago
I don't think the baseline is the same. The more competition, the more temptation to cheat. When the margins to win are small enough, cheaters are disproportionately rewarded. Think of Tour de France. Famously doping-riddled. There are a lot of clean cyclists, but they are much less likely to be able to compete in the tour. You can fight cheating with policing: doping controls, etc. But as the competition gets more extreme, the more resources you need to spend on policing. There's a breaking point, where what you need to spend on policing exceeds what you get from competition. This is why almost no municipalities have a free-for-all policy for taxis. There are too many people technically able to drive people for money. All that competition drives prices lower, sure, but asymptotically. You get less and less lower prices the more competition you pile on - but the incentives for taxi drivers to cheat (by evading taxes, doing money laundering as a side gig etc.) keep growing. London did an interesting thing - with their gruelling geography knowledge exam, they tried to use all that competitive energy to buy something other than marginally lower prices. Still incentive to cheat, of course, but catching cheaters on an exam is probably cheaper and easier than catching cheaters in the economy. (Municipalities that auction taxi permits get to keep most of the bad incentives, without the advantage of competition on price.)
jlawson 3y ago
Any human organization? I don't expect 1-5% fraud in airline pilots, bank tellers, grocery store clerks, judges, structural engineers, restaurant chefs, or even cops (they can be assholes but you don't have to bribe them in functional countries). I think academics can do better than 1-5% fraudulent.
- cycomanic 3y ago
  What? In all of the ones you mentioned there is a known significant amount of fraudulent behaviour. Store clerks, theft is about 1-2% of sales typically. It has been said for years that the majority of that theft is from employees. Airline pilots have been known to drink during their flights (or go away from there seat for other reasons that are not in the rules). Cops, I mean don't get me started, just the protection of a cop who has done something wrong by the other cops would count as fraudulent, but I don't see many cops going after their own black sheep. Judges, in Germany deals (i.e. the accused pleads guilty to lesser charges so the bigger ones get dropped) are only legal under very limited circumstances (almost never and need to be properly documented). Nevertheless, in studies >80% of lawyers reported that they had encountered these deals). I think you seriously underestimate the amount of fraudulent behaviour.
  - cycomanic 3y ago
    Also coming back to judges. The behaviour by Thomas and Alito regarding presents etc. Would count as serious scientific misconduct in academia. So there's a significant percentage just there already.
- P_I_Staker 3y ago
  I expect far higher levels of fraud in these professions.
michael1999 3y ago
No. This level of scrutiny and diligence is rare, and was selectively applied based on the targets profile. The "field" did nothing about this over 20 years. A computer science freshman did this as a hobby, not as a participant in neuroscience. Perhaps "nothing" is too harsh. Various people in the field raised concerns on several occasions. But the journals did nothing. The "field" still honoured him. And _Stanford_ did nothing (except enable him and pay him well) until public embarrassment revealed the ugliness.
- dmvdoug 3y ago
  This is the important and troubling point. Everyone trumpets science as a model of a rational, self-correcting social enterprise. But we see time and time again that it takes non-scientists to blow the whistle and call foul and gin up enough outside attention before something gets done to make the correction. That puts the lie to the notion of self-correction.
  - moab 3y ago
    This is an issue at the department politics level. For the scientific field, once someone starts retracting papers (and arguably, even before this), everybody knows that you should take person X's papers with a huge grain of salt. E.g., in math / theory, if someone has a history of making big blunders that invalidate their results, you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result. So yes, I do trumpet science as a model of a rational, self-correcting social enterprise, at least in CS. Other sciences like biology and psychology have some way to go.
    - VHRanger 3y ago
      The thing is that replication is inherently easy in CS. Especially now that people are expected to post code online. Forcing authors to share raw data and code in all papers would already be a start. I don't know why top impact factor papers don't do this already.

panarky 3y ago

The very fact that the fraud is discovered, that reporters amplify it, and that it can bring down the president of the university, is evidence to me that the system still works.

quickcheque 3y ago
Sure, any system with a false negative and false positive rate 'works'.
wpietri 3y ago
Maybe? I'd want to see a clear model of flows and selection biases before I concluded that. Another way to look at it: perhaps Tessier-Lavigne only got this scrutiny because he was president of the university. And the fact that they didn't guarantee anonymity when "not guaranteeing anonymity in an investigation of this importance is an 'extremely unusual move'" might be a sign that the scrutiny was politically diminished. So it could be that most of the equally dubious researchers don't get caught because not enough attention is paid to patterns like this except when it's somebody especially prominent. Or it could be that this one was not as well covered up, perhaps because of the sheer number of issues. Or that the cross-institution issues made Stanford more willing to note the wrongdoing. Or that Stanford is less likely to sweep things under the rug because of its prominence. Or just that there was some ongoing tension between the trustees and the president and that this was an opportunity to win a political fight.
- nequo 3y ago
  These are good points and hard to know. But the Retraction Watch is tracking stories of both mistakes and fraud in published research, across universities: https://retractionwatch.com/ https://retractionwatch.com/
mike_hearn 3y ago
A tenacious undergrad doing journalism as a hobby is not a system.
- panarky 3y ago
  The fate of the world lies in the hands of the young and inexperienced. Grad students, Supreme Court clerks, 19-year-old soldiers.

karaterobot 3y ago

The problem is that we don't know what the baseline really is. We know that between a third and a half of results from peer reviewed papers in many domains cannot be replicated. Looking closer, we see what look like irregularities in some of them, but it's harder to say which of them are fraud, which are honest mistakes, and which of them just can't be replicated due to some other factors. But because so many of these studies just don't pay off for one reason or another, I would agree that it is getting really hard to rely on a process which is, if nothing else, supposed to result in reliable and trustworthy information.

cycomanic 3y ago
Where is that number of 1/3-1/2 coming from? And which fields? I find that very hard to believe (at least if we exclude the obvious fraudulent journals, where no actual research gets published)
- zeroCalories 3y ago
  I think he's referencing the replication crisis that was a big deal a few years ago. Psychology was hit hard(unsurprising), but a few other fields in the biology area were also hit.
  - karaterobot 3y ago
    It's worst in Psychology and the Social Sciences, but it's not limited to them. Per Wikipedia: > A 2016 survey by Nature on 1,576 researchers who took a brief online questionnaire on reproducibility found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results (including 87% of chemists, 77% of biologists, 69% of physicists and engineers, 67% of medical researchers, 64% of earth and environmental scientists, and 62% of all others), and more than half have failed to reproduce their own experiments. But fewer than 20% had been contacted by another researcher unable to reproduce their work. The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists. Most researchers said they still trust the published literature Not sure if the results of that online study have (or can) themselves be reproduced, however. It's turtles all the way down.

hayd 3y ago

It's only a story because he's president, if he were only a researcher/professor this would not even be a story. This is NOT a success story, it shows that this fraudulent behavior is endemic and an effective strategy for climbing the academic ladder. A success story would be this is exposed at large... we work out some kind of effective peer-reproduced tests... and the hundreds/thousands of cheating professors are fired.

moab 3y ago
So just because one person is cheating, it means all academics are cheating? FWIW, most top-ranked CS conferences have an artifact evaluation track, and it doesn't look good if you submit an experimental paper and don't go through the artifact evaluation process. Things are certainly changing in CS, at least on the experimental side. It's also possible that theorems are incorrect, but subsequent work that figures this out will comment on it and fix it. The scientific record is self-correcting, and fraud / bullshit does get caught out.
- hayd 3y ago
  It's not just "one person", there is wide-spread fraud across many disciplines of academia. The situation, of course, is vastly different across subjects/disciplines, e.g. math and CS are not really much affected and I would agree they're self-correcting. I might agree they're self-correcting in the (very) long-term, but we're seeing fictitious results fund entire careers. We don't know the damage that having 20+ years of incorrect results being built upon will have... And that's not to speak of those who were overlooked, and left academia, because their opportunities were taken by these cheaters (who knows what cost that has for society).
tptacek 3y ago
Endemic means "regularly occurring". How many examples of this kind of misconduct are you aware of? Ok, now, what's the denominator? How much research is actually conducted? I'm personally familiar with 3 fields (CS, bio, and geology) and what I've learned is that the number of labs --- let alone projects --- is mind-boggling. If your examples constitute 1% of all research conducted --- which would represent a cosmic-scale fuckload of research projects --- how much should I care about it?

whydoyoucare 3y ago

If a field takes two decades to "correct" its mistakes, then there are several things wrong with it. And if we have top positions held by unethical people, who have got away with it, and possibly climbed to the top because of it, then I do not know what to feel or say about this.

FrustratedMonky 3y ago
It's taken String Theory a few decades to correct itself.

vintermann 3y ago

I don't think the baseline is the same. The more competition, the more temptation to cheat. When the margins to win are small enough, cheaters are disproportionately rewarded. Think of Tour de France. Famously doping-riddled. There are a lot of clean cyclists, but they are much less likely to be able to compete in the tour. You can fight cheating with policing: doping controls, etc. But as the competition gets more extreme, the more resources you need to spend on policing. There's a breaking point, where what you need to spend on policing exceeds what you get from competition. This is why almost no municipalities have a free-for-all policy for taxis. There are too many people technically able to drive people for money. All that competition drives prices lower, sure, but asymptotically. You get less and less lower prices the more competition you pile on - but the incentives for taxi drivers to cheat (by evading taxes, doing money laundering as a side gig etc.) keep growing. London did an interesting thing - with their gruelling geography knowledge exam, they tried to use all that competitive energy to buy something other than marginally lower prices. Still incentive to cheat, of course, but catching cheaters on an exam is probably cheaper and easier than catching cheaters in the economy. (Municipalities that auction taxi permits get to keep most of the bad incentives, without the advantage of competition on price.)

jlawson 3y ago

Any human organization? I don't expect 1-5% fraud in airline pilots, bank tellers, grocery store clerks, judges, structural engineers, restaurant chefs, or even cops (they can be assholes but you don't have to bribe them in functional countries). I think academics can do better than 1-5% fraudulent.

cycomanic 3y ago
What? In all of the ones you mentioned there is a known significant amount of fraudulent behaviour. Store clerks, theft is about 1-2% of sales typically. It has been said for years that the majority of that theft is from employees. Airline pilots have been known to drink during their flights (or go away from there seat for other reasons that are not in the rules). Cops, I mean don't get me started, just the protection of a cop who has done something wrong by the other cops would count as fraudulent, but I don't see many cops going after their own black sheep. Judges, in Germany deals (i.e. the accused pleads guilty to lesser charges so the bigger ones get dropped) are only legal under very limited circumstances (almost never and need to be properly documented). Nevertheless, in studies >80% of lawyers reported that they had encountered these deals). I think you seriously underestimate the amount of fraudulent behaviour.
- cycomanic 3y ago
  Also coming back to judges. The behaviour by Thomas and Alito regarding presents etc. Would count as serious scientific misconduct in academia. So there's a significant percentage just there already.
P_I_Staker 3y ago
I expect far higher levels of fraud in these professions.

michael1999 3y ago

No. This level of scrutiny and diligence is rare, and was selectively applied based on the targets profile. The "field" did nothing about this over 20 years. A computer science freshman did this as a hobby, not as a participant in neuroscience. Perhaps "nothing" is too harsh. Various people in the field raised concerns on several occasions. But the journals did nothing. The "field" still honoured him. And _Stanford_ did nothing (except enable him and pay him well) until public embarrassment revealed the ugliness.

dmvdoug 3y ago
This is the important and troubling point. Everyone trumpets science as a model of a rational, self-correcting social enterprise. But we see time and time again that it takes non-scientists to blow the whistle and call foul and gin up enough outside attention before something gets done to make the correction. That puts the lie to the notion of self-correction.
- moab 3y ago
  This is an issue at the department politics level. For the scientific field, once someone starts retracting papers (and arguably, even before this), everybody knows that you should take person X's papers with a huge grain of salt. E.g., in math / theory, if someone has a history of making big blunders that invalidate their results, you will be very hesitant to accept results from a new paper they put on arXiv until your community has vetted the result. So yes, I do trumpet science as a model of a rational, self-correcting social enterprise, at least in CS. Other sciences like biology and psychology have some way to go.
  - VHRanger 3y ago
    The thing is that replication is inherently easy in CS. Especially now that people are expected to post code online. Forcing authors to share raw data and code in all papers would already be a start. I don't know why top impact factor papers don't do this already.